bpf: Add gen_epilogue to bpf_verifier_ops
Subsystems (5)
Martin KaFai Lau
reviewerv4 cover bpf: Add gen_epilogue to bpf_verifier_ops
From: Martin KaFai Lau <[email protected]>
This set allows the subsystem to patch codes before BPF_EXIT. The verifier ops, .gen_epilogue, is added for this purpose. One of the use case will be in the bpf qdisc, the bpf qdisc subsystem can ensure the skb->dev is in the correct value. The bpf qdisc subsystem can either inline fixing it in the epilogue or call another kernel function to handle it (e.g. drop) in the epilogue. Another use case could be in bpf_tcp_ca.c to enforce snd_cwnd has valid value (e.g. positive value).
v4:
* Fixed a bug in the memcpy in patch 3
The size in the memcpy should be
epilogue_cnt * sizeof(*epilogue_buf)
v3:
* Moved epilogue_buf[16] to env.
Patch 1 is added to move the existing insn_buf[16] to env.
* Fixed a case that the bpf prog has a BPF_JMP that goes back to the first instruction of the main prog.
The jump back to 1st insn case also applies to the prologue. Patch 2 is added to handle it.
* If the bpf main prog has multiple BPF_EXIT, use a BPF_JA to goto the earlier patched epilogue.
Note that there are (BPF_JMP32 | BPF_JA) vs (BPF_JMP | BPF_JA) details in the patch 3 commit message.
* There are subtle changes in patch 3, so I reset the Reviewed-by.
* Added patch 8 and patch 9 to cover the changes in patch 2 and patch 3.
* Dropped the kfunc call from pro/epilogue and its selftests.
v2:
* Remove the RFC tag. Keep the ordering at where .gen_epilogue is called in the verifier relative to the check_max_stack_depth(). This will be consistent with the other extra stack_depth usage like optimize_bpf_loop().
* Use __xlated check provided by the test_loader to check the patched instructions after gen_pro/epilogue (Eduard).
* Added Patch 3 by Eduard (Thanks!).
Eduard Zingerman (1):
selftests/bpf: attach struct_ops maps before test prog runs
Martin KaFai Lau (8):
bpf: Move insn_buf[16] to bpf_verifier_env
bpf: Adjust BPF_JMP that jumps to the 1st insn of the prologue
bpf: Add gen_epilogue to bpf_verifier_ops
bpf: Export bpf_base_func_proto
selftests/bpf: Test gen_prologue and gen_epilogue selftests/bpf: Add tailcall epilogue test
selftests/bpf: A pro/epilogue test when the main prog jumps back to
the 1st insn
selftests/bpf: Test epilogue patching when the main prog has multiple
BPF_EXIT
include/linux/bpf.h | 2 + include/linux/bpf_verifier.h | 4 + kernel/bpf/helpers.c | 1 + kernel/bpf/verifier.c | 73 +++++-- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 190 ++++++++++++++++++ .../selftests/bpf/bpf_testmod/bpf_testmod.h | 11 + .../bpf/bpf_testmod/bpf_testmod_kfunc.h | 6 + .../selftests/bpf/prog_tests/pro_epilogue.c | 54 +++++ .../selftests/bpf/progs/epilogue_exit.c | 78 +++++++ .../selftests/bpf/progs/epilogue_tailcall.c | 58 ++++++ .../bpf/progs/pro_epilogue_goto_start.c | 149 ++++++++++++++ .../selftests/bpf/progs/pro_epilogue_kfunc.c | 156 ++++++++++++++ .../bpf/progs/pro_epilogue_subprog.c | 125 ++++++++++++ tools/testing/selftests/bpf/test_loader.c | 27 +++ 14 files changed, 922 insertions(+), 12 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/pro_epilogue.c create mode 100644 tools/testing/selftests/bpf/progs/epilogue_exit.c create mode 100644 tools/testing/selftests/bpf/progs/epilogue_tailcall.c create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_goto_start.c create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_subprog.c
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v4 1/9 bpf: Move insn_buf[16] to bpf_verifier_env
From: Martin KaFai Lau <[email protected]>
This patch moves the 'struct bpf_insn insn_buf[16]' stack usage to the bpf_verifier_env. A '#define INSN_BUF_SIZE 16' is also added to replace the ARRAY_SIZE(insn_buf) usages.
Both convert_ctx_accesses() and do_misc_fixup() are changed to use the env->insn_buf.
It is a prep work for adding the epilogue_buf[16] in a later patch.
--- include/linux/bpf_verifier.h | 3 +++ kernel/bpf/verifier.c | 15 ++++++++------- 2 files changed, 11 insertions(+), 7 deletions(-)
include/linux/bpf_verifier.h +3 -0
@@ -23,6 +23,8 @@* (in the "-8,-16,...,-512" form) */ #define TMP_STR_BUF_LEN 320 +/* Patch buffer size */ +#define INSN_BUF_SIZE 16/* Liveness marks, used for registers and spilled-regs (in stack slots). * Read marks propagate upwards until they find a write mark; they record that
@@ -780,6 +782,7 @@ struct bpf_verifier_env {* e.g., in reg_type_str() to generate reg_type string */ char tmp_str_buf[TMP_STR_BUF_LEN ]; +struct bpf_insn insn_buf[INSN_BUF_SIZE ]; };static inline struct bpf_func_info_aux *subprog_aux (struct bpf_verifier_env *env,int subprog)
kernel/bpf/verifier.c +8 -7
@@ -19678,7 +19678,8 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)const struct bpf_verifier_ops *ops = env->ops;int i, cnt, size, ctx_field_size, delta =0 ;const int insn_cnt = env->prog->len; -struct bpf_insn insn_buf[16 ], *insn; +struct bpf_insn *insn_buf = env->insn_buf; +struct bpf_insn *insn;u32 target_size, size_default, off;struct bpf_prog *new_prog;enum bpf_access_type type;
@@ -19691,7 +19692,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) } cnt = ops->gen_prologue (insn_buf, env->seen_direct_write, env->prog); -if (cnt >=ARRAY_SIZE (insn_buf)) { +if (cnt >=INSN_BUF_SIZE ) {verbose (env,"bpf verifier is misconfigured\n" );return -EINVAL ; }else if (cnt) {
@@ -19838,7 +19839,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) target_size =0 ; cnt =convert_ctx_access (type, insn, insn_buf, env->prog, &target_size); -if (cnt ==0 || cnt >=ARRAY_SIZE (insn_buf) || +if (cnt ==0 || cnt >=INSN_BUF_SIZE || (ctx_field_size && !target_size)) {verbose (env,"bpf verifier is misconfigured\n" );return -EINVAL ;
@@ -19847,7 +19848,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)if (is_narrower_load && size < target_size) {u8 shift =bpf_ctx_narrow_access_offset ( off, size, size_default) *8 ; -if (shift && cnt +1 >=ARRAY_SIZE (insn_buf)) { +if (shift && cnt +1 >=INSN_BUF_SIZE ) {verbose (env,"bpf verifier narrow ctx load misconfigured\n" );return -EINVAL ; }
@@ -20392,7 +20393,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env)const int insn_cnt = prog->len;const struct bpf_map_ops *ops;struct bpf_insn_aux_data *aux; -struct bpf_insn insn_buf[16 ]; +struct bpf_insn *insn_buf = env->insn_buf;struct bpf_prog *new_prog;struct bpf_map *map_ptr;int i, ret, cnt, delta =0 , cur_subprog =0 ;
@@ -20511,7 +20512,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env) (BPF_MODE(insn->code) == BPF_ABS || BPF_MODE(insn->code) == BPF_IND)) { cnt = env->ops->gen_ld_abs (insn, insn_buf); -if (cnt ==0 || cnt >=ARRAY_SIZE (insn_buf)) { +if (cnt ==0 || cnt >=INSN_BUF_SIZE ) {verbose (env,"bpf verifier is misconfigured\n" );return -EINVAL ; }
@@ -20804,7 +20805,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env) cnt = ops->map_gen_lookup (map_ptr, insn_buf);if (cnt == -EOPNOTSUPP ) goto patch_map_ops_generic; -if (cnt <=0 || cnt >=ARRAY_SIZE (insn_buf)) { +if (cnt <=0 || cnt >=INSN_BUF_SIZE ) {verbose (env,"bpf verifier is misconfigured\n" );return -EINVAL ; }
Signature · 1 line
2.43.5
Re: [PATCH v4 bpf-next 1/9] bpf: Move insn_buf[16] to bpf_verifier_env
Quoted 6 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch moves the 'struct
bpf_insninsn_buf[16]' stack usage to thebpf_verifier_env. A '#defineINSN_BUF_SIZE16' is also added to replace theARRAY_SIZE(insn_buf)usages.
Quoted 7 lines
Both
convert_ctx_accesses()anddo_misc_fixup()are changed to use theenv->insn_buf.It is a prep work for adding the epilogue_buf[16] in a later patch.
Signed-off-by Martin KaFai Lau <[email protected]>---
Not sure if this refactoring is worth it but code looks correct. Note that there is also inline_bpf_loop()
(it needs a slightly bigger buffer).
[...]
Re: [PATCH v4 bpf-next 1/9] bpf: Move insn_buf[16] to bpf_verifier_env
Quoted 11 lines · Eduard Zingerman
Quoted 9 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch moves the 'struct
bpf_insninsn_buf[16]' stack usage to thebpf_verifier_env. A '#defineINSN_BUF_SIZE16' is also added to replace theARRAY_SIZE(insn_buf)usages.Both
convert_ctx_accesses()anddo_misc_fixup()are changed to use theenv->insn_buf.
Quoted 8 lines
Quoted 4 lines
It is a prep work for adding the epilogue_buf[16] in a later patch.
Signed-off-by Martin KaFai Lau <[email protected]>---
Not sure if this refactoring is worth it but code looks correct. Note that there is also
inline_bpf_loop()
(it needs a slightly bigger buffer).
Probably worth it in the follow up, since people complain that this or that function in verifier.c reaches stack size limit when compiled with sanitizers.
These buffers on stack are the biggest consumers.
Martin KaFai Lau
reviewerRe: [PATCH v4 bpf-next 1/9] bpf: Move insn_buf[16] to bpf_verifier_env
Quoted 17 lines · Alexei Starovoitov
Quoted 16 lines · Eduard Zingerman
Quoted 13 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch moves the 'struct
bpf_insninsn_buf[16]' stack usage to thebpf_verifier_env. A '#defineINSN_BUF_SIZE16' is also added to replace theARRAY_SIZE(insn_buf)usages.Both
convert_ctx_accesses()anddo_misc_fixup()are changed to use theenv->insn_buf.It is a prep work for adding the epilogue_buf[16] in a later patch.
Signed-off-by Martin KaFai Lau <[email protected]>---
Quoted 8 lines
Quoted 3 lines
Not sure if this refactoring is worth it but code looks correct. Note that there is also
inline_bpf_loop()
(it needs a slightly bigger buffer).Probably worth it in the follow up, since people complain that this or that function in
verifier.creaches stack size limit when compiled with sanitizers.
These buffers on stack are the biggest consumers.
ok. I will drop this patch for now. Redo it again as a followup and will consider inline_bpf_loop() together at that time.
Regarding the stack size, I did notice the compilation warning difference on the stack size which I should have put in the commit message.
Before:./kernel/bpf/verifier.c:22133:5: warning: stack frame size (2584) exceeds limit (2048) in 'bpf_check' [-Wframe-larger-than]
After:./kernel/bpf/verifier.c:22184:5: warning: stack frame size (2264) exceeds limit (2048) in 'bpf_check' [-Wframe-larger-than]
Re: [PATCH v4 bpf-next 1/9] bpf: Move insn_buf[16] to bpf_verifier_env
Quoted 23 lines · Martin KaFai Lau
Quoted 21 lines · Alexei Starovoitov
Quoted 19 lines · Eduard Zingerman
Quoted 6 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch moves the 'struct
bpf_insninsn_buf[16]' stack usage to thebpf_verifier_env. A '#defineINSN_BUF_SIZE16' is also added to replace theARRAY_SIZE(insn_buf)usages.Quoted 7 lines
Both
convert_ctx_accesses()anddo_misc_fixup()are changed to use theenv->insn_buf.It is a prep work for adding the epilogue_buf[16] in a later patch.
Signed-off-by Martin KaFai Lau <[email protected]>---
Not sure if this refactoring is worth it but code looks correct. Note that there is also
inline_bpf_loop()
(it needs a slightly bigger buffer).
Quoted 7 lines
Quoted 4 lines
Probably worth it in the follow up, since people complain that this or that function in
verifier.creaches stack size limit when compiled with sanitizers.
These buffers on stack are the biggest consumers.ok. I will drop this patch for now. Redo it again as a followup and will consider
inline_bpf_loop()together at that time.
why? Keep it. It's an improvement already.
Quoted 10 lines
Regarding the stack size, I did notice the compilation warning difference on the stack size which I should have put in the commit message.
Before:
./kernel/bpf/verifier.c:22133:5: warning: stack frame size (2584) exceeds limit (2048) in 'bpf_check' [-Wframe-larger-than]After:
./kernel/bpf/verifier.c:22184:5: warning: stack frame size (2264) exceeds limit (2048) in 'bpf_check' [-Wframe-larger-than]
Exactly. It's a step forward.
Martin KaFai Lau
reviewerRe: [PATCH v4 bpf-next 1/9] bpf: Move insn_buf[16] to bpf_verifier_env
Quoted 17 lines · Alexei Starovoitov
Quoted 16 lines · Martin KaFai Lau
Quoted 14 lines · Alexei Starovoitov
Quoted 13 lines · Eduard Zingerman
Quoted 11 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch moves the 'struct
bpf_insninsn_buf[16]' stack usage to thebpf_verifier_env. A '#defineINSN_BUF_SIZE16' is also added to replace theARRAY_SIZE(insn_buf)usages.Both
convert_ctx_accesses()anddo_misc_fixup()are changed to use theenv->insn_buf.It is a prep work for adding the epilogue_buf[16] in a later patch.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Signed-off-by Martin KaFai Lau <[email protected]>---
Not sure if this refactoring is worth it but code looks correct. Note that there is also
inline_bpf_loop()
(it needs a slightly bigger buffer).Probably worth it in the follow up, since people complain that
I read it and Eduard's earlier comment together as the whole patch 1 as a follow up. :)
yep. I will keep this one and follow up with the insn_buf in the inline_bpf_loop().
I will update the commit message with the stack usage in the next respin.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
this or that function in
verifier.creaches stack size limit when compiled with sanitizers.
These buffers on stack are the biggest consumers.ok. I will drop this patch for now. Redo it again as a followup and will consider
inline_bpf_loop()together at that time.why? Keep it. It's an improvement already.
Martin KaFai Lau
reviewerPATCH v4 2/9 bpf: Adjust BPF_JMP that jumps to the 1st insn of the prologue
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
The next patch will add a ctx ptr saving instruction "(r1 = *(u64 *)(r10 -8)" at the beginning for the main prog when there is an epilogue patch (by the .gen_epilogue() verifier ops added in the next patch).
There is one corner case if the bpf prog has a BPF_JMP that jumps
to the 1st instruction. It needs an adjustment such that
those BPF_JMP instructions won't jump to the newly added
ctx saving instruction.
The commit 5337ac4c9b80 ("bpf: Fix the corner case with may_goto and jump to the 1st insn.") has the details on this case.
Note that the jump back to 1st instruction is not limited to the ctx ptr saving instruction. The same also applies to the prologue. A later test, pro_epilogue_goto_start.c, has a test for the prologue only case.
Thus, this patch does one adjustment after gen_prologue and the future ctx ptr saving. It is done by adjust_jmp_off(env->prog, 0, delta) where delta has the total number of instructions in the prologue and
the future ctx ptr saving instruction.
The adjust_jmp_off(env->prog, 0, delta) assumes that the prologue does not have a goto 1st instruction itself.
To accommodate the prologue might have a goto 1st insn itself, adjust_jmp_off() needs to skip the prologue instructions. This patch adds a skip_cnt argument to the adjust_jmp_off(). The skip_cnt is the number of instructions at the beginning that does not need adjustment. adjust_jmp_off(prog, 0, delta, delta) is used in this patch.
--- kernel/bpf/verifier.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
kernel/bpf/verifier.c +7 -4
@@ -19277,14 +19277,14 @@ static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of* For all jmp insns in a given 'prog' that point to 'tgt_idx' insn adjust the * jump offset by 'delta'. */ -static int adjust_jmp_off (struct bpf_prog *prog,u32 tgt_idx,u32 delta) +static int adjust_jmp_off (struct bpf_prog *prog,u32 tgt_idx,u32 delta,u32 skip_cnt) { -struct bpf_insn *insn = prog->insnsi; +struct bpf_insn *insn = prog->insnsi + skip_cnt;u32 insn_cnt = prog->len, i;s32 imm;s16 off; -for (i =0 ; i < insn_cnt; i++, insn++) { +for (i = skip_cnt; i < insn_cnt; i++, insn++) {u8 code = insn->code; if ((BPF_CLASS(code) != BPF_JMP && BPF_CLASS(code) != BPF_JMP32) ||
@@ -19705,6 +19705,9 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) } } +if (delta) +WARN_ON (adjust_jmp_off (env->prog,0 , delta, delta)); +if (bpf_prog_is_offloaded (env->prog->aux))return 0 ;
@@ -21136,7 +21139,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env)* to insn after BPF_ST that inits may_goto count. * Adjustment will succeed because bpf_patch_insn_data() didn't fail. */ -WARN_ON (adjust_jmp_off (env->prog, subprog_start,1 )); +WARN_ON (adjust_jmp_off (env->prog, subprog_start,1 ,0 )); }/* Since poke tab is now finalized, publish aux to tracker. */
Signature · 1 line
2.43.5
Re: [PATCH v4 bpf-next 2/9] bpf: Adjust BPF_JMP that jumps to the 1st insn of the prologue
Quoted 48 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
The next patch will add a ctx ptr saving instruction "(r1 = *(u64 *)(r10 -8)" at the beginning for the main prog when there is an epilogue patch (by the .gen_epilogue() verifier ops added in the next patch).
There is one corner case if the bpf prog has a
BPF_JMPthat jumps
to the 1st instruction. It needs an adjustment such that
thoseBPF_JMPinstructions won't jump to the newly added
ctx saving instruction.
The commit5337ac4c9b80("bpf: Fix the corner case withmay_gotoand jump to the 1st insn.") has the details on this case.Note that the jump back to 1st instruction is not limited to the ctx ptr saving instruction. The same also applies to the prologue. A later test,
pro_epilogue_goto_start.c, has a test for the prologue only case.Thus, this patch does one adjustment after
gen_prologueand the future ctx ptr saving. It is done byadjust_jmp_off(env->prog, 0, delta)where delta has the total number of instructions in the prologue and
the future ctx ptr saving instruction.The
adjust_jmp_off(env->prog, 0, delta)assumes that the prologue does not have a goto 1st instruction itself.
To accommodate the prologue might have a goto 1st insn itself,adjust_jmp_off()needs to skip the prologue instructions. This patch adds askip_cntargument to theadjust_jmp_off(). Theskip_cntis the number of instructions at the beginning that does not need adjustment.adjust_jmp_off(prog, 0, delta, delta)is used in this patch.Signed-off-by Martin KaFai Lau <[email protected]>--- kernel/bpf/verifier.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)kernel/bpf/verifier.c +0
-1@@ -19277,14 +19277,14 @@ static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of* For all jmp insns in a given 'prog' that point to 'tgt_idx' insn adjust the * jump offset by 'delta'. */ -static int adjust_jmp_off (struct bpf_prog *prog,u32 tgt_idx,u32 delta)
kernel/bpf/verifier.c static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of · line 19287
Quoted 10 lines
kernel/bpf/verifier.c +3
-2+static int adjust_jmp_off (struct bpf_prog *prog,u32 tgt_idx,u32 delta,u32 skip_cnt) { -struct bpf_insn *insn = prog->insnsi; +struct bpf_insn *insn = prog->insnsi + skip_cnt;u32 insn_cnt = prog->len, i;s32 imm;s16 off; -for (i =0 ; i < insn_cnt; i++, insn++) { +for (i = skip_cnt; i < insn_cnt; i++, insn++) {
Do we really need to add this argument?
kernel/bpf/verifier.c static int do_misc_fixups(struct bpf_verifier_env *env) · line 21142
Quoted 2 lines
kernel/bpf/verifier.c +1
-1-WARN_ON (adjust_jmp_off (env->prog, subprog_start,1 )); +WARN_ON (adjust_jmp_off (env->prog, subprog_start,1 ,0 ));
We can always do for (i = delta; ...
The above case of skip_cnt == 0 is lucky to work this way. It would be less surprising to skip all insns in the patch. Maybe I'm missing something.
Martin KaFai Lau
reviewerRe: [PATCH v4 bpf-next 2/9] bpf: Adjust BPF_JMP that jumps to the 1st insn of the prologue
Quoted 62 lines · Alexei Starovoitov
Quoted 48 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
The next patch will add a ctx ptr saving instruction "(r1 = *(u64 *)(r10 -8)" at the beginning for the main prog when there is an epilogue patch (by the .gen_epilogue() verifier ops added in the next patch).
There is one corner case if the bpf prog has a
BPF_JMPthat jumps
to the 1st instruction. It needs an adjustment such that
thoseBPF_JMPinstructions won't jump to the newly added
ctx saving instruction.
The commit5337ac4c9b80("bpf: Fix the corner case withmay_gotoand jump to the 1st insn.") has the details on this case.Note that the jump back to 1st instruction is not limited to the ctx ptr saving instruction. The same also applies to the prologue. A later test,
pro_epilogue_goto_start.c, has a test for the prologue only case.Thus, this patch does one adjustment after
gen_prologueand the future ctx ptr saving. It is done byadjust_jmp_off(env->prog, 0, delta)where delta has the total number of instructions in the prologue and
the future ctx ptr saving instruction.The
adjust_jmp_off(env->prog, 0, delta)assumes that the prologue does not have a goto 1st instruction itself.
To accommodate the prologue might have a goto 1st insn itself,adjust_jmp_off()needs to skip the prologue instructions. This patch adds askip_cntargument to theadjust_jmp_off(). Theskip_cntis the number of instructions at the beginning that does not need adjustment.adjust_jmp_off(prog, 0, delta, delta)is used in this patch.Signed-off-by Martin KaFai Lau <[email protected]>--- kernel/bpf/verifier.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)kernel/bpf/verifier.c +0
-1@@ -19277,14 +19277,14 @@ static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of* For all jmp insns in a given 'prog' that point to 'tgt_idx' insn adjust the * jump offset by 'delta'. */ -static int adjust_jmp_off (struct bpf_prog *prog,u32 tgt_idx,u32 delta)kernel/bpf/verifier.c static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of · line 19287
Quoted 10 lines
kernel/bpf/verifier.c +3
-2+static int adjust_jmp_off (struct bpf_prog *prog,u32 tgt_idx,u32 delta,u32 skip_cnt) { -struct bpf_insn *insn = prog->insnsi; +struct bpf_insn *insn = prog->insnsi + skip_cnt;u32 insn_cnt = prog->len, i;s32 imm;s16 off; -for (i =0 ; i < insn_cnt; i++, insn++) { +for (i = skip_cnt; i < insn_cnt; i++, insn++) {Do we really need to add this argument?
Quoted 8 lines
kernel/bpf/verifier.c static int do_misc_fixups(struct bpf_verifier_env *env) · line 21142
Quoted 2 lines
kernel/bpf/verifier.c +1
-1-WARN_ON (adjust_jmp_off (env->prog, subprog_start,1 )); +WARN_ON (adjust_jmp_off (env->prog, subprog_start,1 ,0 ));We can always do for (i = delta; ...
The above case of
skip_cnt== 0 is lucky to work this way. It would be less surprising to skip all insns in the patch. Maybe I'm missing something.
For subprog_start case, tgt_idx (where the patch started) may not be 0. How about this:
for (i = 0; i < insn_cnt; i++, insn++) {
if (tgt_idx <= i && i < tgt_idx + delta)
continue;
/* ... */
}Re: [PATCH v4 bpf-next 2/9] bpf: Adjust BPF_JMP that jumps to the 1st insn of the prologue
Quoted 15 lines · Martin KaFai Lau
Quoted 6 lines
kernel/bpf/verifier.c static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of · line 19287
Quoted 3 lines
kernel/bpf/verifier.c +1
-1-for (i =0 ; i < insn_cnt; i++, insn++) { +for (i = skip_cnt; i < insn_cnt; i++, insn++) {Do we really need to add this argument?
Quoted 8 lines
kernel/bpf/verifier.c static int do_misc_fixups(struct bpf_verifier_env *env) · line 21142
Quoted 2 lines
kernel/bpf/verifier.c +1
-1-WARN_ON (adjust_jmp_off (env->prog, subprog_start,1 )); +WARN_ON (adjust_jmp_off (env->prog, subprog_start,1 ,0 ));We can always do for (i = delta; ...
The above case of
skip_cnt== 0 is lucky to work this way. It would be less surprising to skip all insns in the patch. Maybe I'm missing something.
Quoted 6 lines
For
subprog_startcase,tgt_idx(where the patch started) may not be 0. How about this:for (i = 0; i < insn_cnt; i++, insn++) { if (tgt_idx <= i && i < tgt_idx + delta) continue;
Yeah. Right. Same idea, but certainly your way is more correct instead of my buggy proposal.
In that sense the "for (i = skip_cnt" approach
is also a bit buggy, if tgt_idx != 0.
Martin KaFai Lau
reviewerRe: [PATCH v4 bpf-next 2/9] bpf: Adjust BPF_JMP that jumps to the 1st insn of the prologue
Quoted 23 lines · Alexei Starovoitov
Quoted 15 lines · Martin KaFai Lau
Quoted 6 lines
kernel/bpf/verifier.c static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of · line 19287
Quoted 3 lines
kernel/bpf/verifier.c +1
-1-for (i =0 ; i < insn_cnt; i++, insn++) { +for (i = skip_cnt; i < insn_cnt; i++, insn++) {Do we really need to add this argument?
Quoted 8 lines
kernel/bpf/verifier.c static int do_misc_fixups(struct bpf_verifier_env *env) · line 21142
Quoted 2 lines
kernel/bpf/verifier.c +1
-1-WARN_ON (adjust_jmp_off (env->prog, subprog_start,1 )); +WARN_ON (adjust_jmp_off (env->prog, subprog_start,1 ,0 ));We can always do for (i = delta; ...
The above case of
skip_cnt== 0 is lucky to work this way. It would be less surprising to skip all insns in the patch. Maybe I'm missing something.Quoted 6 lines
For
subprog_startcase,tgt_idx(where the patch started) may not be 0. How about this:for (i = 0; i < insn_cnt; i++, insn++) { if (tgt_idx <= i && i < tgt_idx + delta) continue;
Quoted 5 lines
Yeah. Right. Same idea, but certainly your way is more correct instead of my buggy proposal.
In that sense the "for (i =
skip_cnt" approach
is also a bit buggy, iftgt_idx!= 0.
Yep. Adding skip_cnt like this patch 2 is not right. I didn't think hard enough what to do with the existing adjust_jmp_off().
I will remove skip_cnt in the next respin.
Thanks for the review!
Martin KaFai Lau
reviewerPATCH v4 3/9 bpf: Add gen_epilogue to bpf_verifier_ops
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
This patch adds a .gen_epilogue to the bpf_verifier_ops. It is similar to the existing .gen_prologue. Instead of allowing a subsystem to run code at the beginning of a bpf prog, it allows the subsystem to run code just before the bpf prog exit.
One of the use case is to allow the upcoming bpf qdisc to ensure that the skb->dev is the same as the qdisc->dev_queue->dev. The bpf qdisc struct_ops implementation could either fix it up or drop the skb. Another use case could be in bpf_tcp_ca.c to enforce snd_cwnd has sane value (e.g. non zero).
The epilogue can do the useful thing (like checking skb->dev) if it can access the bpf prog's ctx. Unlike prologue, r1 may not hold the ctx pointer. This patch saves the r1 in the stack if the .gen_epilogue has returned some instructions in the "epilogue_buf".
The existing .gen_prologue is done in convert_ctx_accesses(). The new .gen_epilogue is done in the convert_ctx_accesses() also. When it sees the (BPF_JMP | BPF_EXIT) instruction, it will be patched with the earlier generated "epilogue_buf". The epilogue patching is only done for the main prog.
Only one epilogue will be patched to the main program. When the bpf prog has multiple BPF_EXIT instructions, a BPF_JA is used to goto the earlier patched epilogue. Majority of the archs support (BPF_JMP32 | BPF_JA): x86, arm, s390, risv64, loongarch, powerpc and arc. This patch keeps it simple and always use (BPF_JMP32 | BPF_JA).
--- include/linux/bpf.h | 2 ++ include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 47 +++++++++++++++++++++++++++++++++++- 3 files changed, 49 insertions(+), 1 deletion(-)
include/linux/bpf.h +2 -0
@@ -974,6 +974,8 @@ struct bpf_verifier_ops { struct bpf_insn_access_aux *info);int (*gen_prologue)(struct bpf_insn *insn,bool direct_write,const struct bpf_prog *prog); +int (*gen_epilogue)(struct bpf_insn *insn,const struct bpf_prog *prog, +s16 ctx_stack_off);int (*gen_ld_abs)(const struct bpf_insn *orig,struct bpf_insn *insn_buf); u32 (*convert_ctx_access)(enum bpf_access_type type,
include/linux/bpf_verifier.h +1 -0
@@ -783,6 +783,7 @@ struct bpf_verifier_env {*/ char tmp_str_buf[TMP_STR_BUF_LEN ];struct bpf_insn insn_buf[INSN_BUF_SIZE ]; +struct bpf_insn epilogue_buf[INSN_BUF_SIZE ]; };static inline struct bpf_func_info_aux *subprog_aux (struct bpf_verifier_env *env,int subprog)
kernel/bpf/verifier.c +46 -1
73 lines not on this page · raw message
Signature · 1 line
2.43.5
Quoted 31 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch adds a .gen_epilogue to the
bpf_verifier_ops. It is similar to the existing .gen_prologue. Instead of allowing a subsystem to run code at the beginning of a bpf prog, it allows the subsystem to run code just before the bpf prog exit.One of the use case is to allow the upcoming bpf qdisc to ensure that the
skb->devis the same as theqdisc->dev_queue->dev. The bpf qdiscstruct_opsimplementation could either fix it up or drop the skb. Another use case could be inbpf_tcp_ca.cto enforcesnd_cwndhas sane value (e.g. non zero).The epilogue can do the useful thing (like checking
skb->dev) if it can access the bpf prog's ctx. Unlike prologue, r1 may not hold the ctx pointer. This patch saves the r1 in the stack if the .gen_epilogue has returned some instructions in the "epilogue_buf".The existing .gen_prologue is done in
convert_ctx_accesses(). The new .gen_epilogue is done in theconvert_ctx_accesses()also. When it sees the (BPF_JMP|BPF_EXIT) instruction, it will be patched with the earlier generated "epilogue_buf". The epilogue patching is only done for the main prog.Only one epilogue will be patched to the main program. When the bpf prog has multiple
BPF_EXITinstructions, aBPF_JAis used to goto the earlier patched epilogue. Majority of the archs support (BPF_JMP32|BPF_JA): x86, arm, s390, risv64, loongarch, powerpc and arc. This patch keeps it simple and always use (BPF_JMP32|BPF_JA).
Quoted 2 lines
Signed-off-by Martin KaFai Lau <[email protected]>---
[...]
kernel/bpf/verifier.c static int convert_ctx_accesses(struct bpf_verifier_env *env) · line 19774
Quoted 12 lines
kernel/bpf/verifier.c +8
-0@@ -19740,6 +19764,26 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) insn->code =BPF_STX |BPF_PROBE_ATOMIC |BPF_SIZE (insn->code); env->prog->aux->num_exentries++;continue ; + }else if (insn->code == (BPF_JMP |BPF_EXIT ) && + epilogue_cnt && + i + delta < subprogs[1 ].start) { +/* Generate epilogue for the main prog */ +if (epilogue_idx) { +/* jump back to the earlier generated epilogue */ + insn_buf[0 ] =BPF_JMP32_IMM (BPF_JA ,0 , + epilogue_idx - i - delta -1 ,0 );
Nit: maybe add BPF_GOTOL macro or mention that this is a 'gotol' instruction in the comment?
(this is how it is called in llvm).
Quoted 15 lines
kernel/bpf/verifier.c +12
-0+ cnt =1 ; + }else { +memcpy (insn_buf, epilogue_buf, + epilogue_cnt *sizeof (*epilogue_buf)); + cnt = epilogue_cnt; +/* epilogue_idx cannot be 0. It must have at +* least one ctx ptr saving insn before the +* epilogue. +*/ + epilogue_idx = i + delta; + } + goto patch_insn_buf; }else {continue ; }
[...]
Martin KaFai Lau
reviewerQuoted 39 lines · Eduard Zingerman
Quoted 31 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch adds a .gen_epilogue to the
bpf_verifier_ops. It is similar to the existing .gen_prologue. Instead of allowing a subsystem to run code at the beginning of a bpf prog, it allows the subsystem to run code just before the bpf prog exit.One of the use case is to allow the upcoming bpf qdisc to ensure that the
skb->devis the same as theqdisc->dev_queue->dev. The bpf qdiscstruct_opsimplementation could either fix it up or drop the skb. Another use case could be inbpf_tcp_ca.cto enforcesnd_cwndhas sane value (e.g. non zero).The epilogue can do the useful thing (like checking
skb->dev) if it can access the bpf prog's ctx. Unlike prologue, r1 may not hold the ctx pointer. This patch saves the r1 in the stack if the .gen_epilogue has returned some instructions in the "epilogue_buf".The existing .gen_prologue is done in
convert_ctx_accesses(). The new .gen_epilogue is done in theconvert_ctx_accesses()also. When it sees the (BPF_JMP|BPF_EXIT) instruction, it will be patched with the earlier generated "epilogue_buf". The epilogue patching is only done for the main prog.Only one epilogue will be patched to the main program. When the bpf prog has multiple
BPF_EXITinstructions, aBPF_JAis used to goto the earlier patched epilogue. Majority of the archs support (BPF_JMP32|BPF_JA): x86, arm, s390, risv64, loongarch, powerpc and arc. This patch keeps it simple and always use (BPF_JMP32|BPF_JA).Quoted 2 lines
Signed-off-by Martin KaFai Lau <[email protected]>---
Acked-by Eduard Zingerman <[email protected]>[...]
Quoted 15 lines
kernel/bpf/verifier.c static int convert_ctx_accesses(struct bpf_verifier_env *env) · line 19774
Quoted 12 lines
kernel/bpf/verifier.c +8
-0@@ -19740,6 +19764,26 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) insn->code =BPF_STX |BPF_PROBE_ATOMIC |BPF_SIZE (insn->code); env->prog->aux->num_exentries++;continue ; + }else if (insn->code == (BPF_JMP |BPF_EXIT ) && + epilogue_cnt && + i + delta < subprogs[1 ].start) { +/* Generate epilogue for the main prog */ +if (epilogue_idx) { +/* jump back to the earlier generated epilogue */ + insn_buf[0 ] =BPF_JMP32_IMM (BPF_JA ,0 , + epilogue_idx - i - delta -1 ,0 );Nit: maybe add
BPF_GOTOLmacro or mention that this is a 'gotol' instruction in the comment?
(this is how it is called in llvm).
sgtm.
There is a BPF_JMP_A(OFF). BPF_JMP32_A(IMM) probably will be more consistent with the other existing macros.
Martin KaFai Lau
reviewerPATCH v4 4/9 bpf: Export bpf_base_func_proto
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
The bpf_testmod needs to use the bpf_tail_call helper in a later selftest patch. This patch is to EXPORT_GPL_SYMBOL the bpf_base_func_proto.
--- kernel/bpf/helpers.c | 1 + 1 file changed, 1 insertion(+)
kernel/bpf/helpers.c +1 -0
@@ -2034,6 +2034,7 @@ bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return NULL ; } } +EXPORT_SYMBOL_GPL (bpf_base_func_proto);void bpf_list_head_free (const struct btf_field *field,void *list_head,struct bpf_spin_lock *spin_lock)
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v4 5/9 selftests/bpf: attach struct_ops maps before test prog runs
In reply to Martin KaFai LauFrom: Eduard Zingerman <[email protected]>
In test_loader based tests to bpf_map__attach_struct_ops() before call to bpf_prog_test_run_opts() in order to trigger bpf_struct_ops->reg() callbacks on kernel side. This allows to use __retval macro for struct_ops tests.
--- tools/testing/selftests/bpf/test_loader.c | 27 +++++++++++++++++++++++ 1 file changed, 27 insertions(+)
tools/testing/selftests/bpf/test_loader.c +27 -0
@@ -890,11 +890,13 @@ void run_subtest(struct test_loader *tester, {struct test_subspec *subspec = unpriv ? &spec->unpriv : &spec->priv;struct bpf_program *tprog =NULL , *tprog_iter; +struct bpf_link *link, *links[32 ] = {};struct test_spec *spec_iter;struct cap_state caps = {};struct bpf_object *tobj;struct bpf_map *map;int retval, err, i; +int links_cnt =0 ;bool should_load;if (!test__start_subtest (subspec->name))
@@ -999,6 +1001,26 @@ void run_subtest(struct test_loader *tester,if (restore_capabilities (&caps)) goto tobj_cleanup; +/* Do bpf_map__attach_struct_ops() for each struct_ops map. +* This should trigger bpf_struct_ops->reg callback on kernel side. +*/ +bpf_object__for_each_map (map, tobj) { +if (!bpf_map__autocreate (map) || +bpf_map__type (map) !=BPF_MAP_TYPE_STRUCT_OPS ) +continue ; +if (links_cnt >=ARRAY_SIZE (links)) { +PRINT_FAIL ("too many struct_ops maps" ); + goto tobj_cleanup; + } + link =bpf_map__attach_struct_ops (map); +if (!link) { +PRINT_FAIL ("bpf_map__attach_struct_ops failed for map %s: err=%d\n" , +bpf_map__name (map), err); + goto tobj_cleanup; + } + links[links_cnt++] = link; + } +if (tester->pre_execution_cb) { err = tester->pre_execution_cb (tobj);if (err) {
@@ -1013,9 +1035,14 @@ void run_subtest(struct test_loader *tester,PRINT_FAIL ("Unexpected retval: %d != %d\n" , retval, subspec->retval); goto tobj_cleanup; } +/* redo bpf_map__attach_struct_ops for each test */ +while (links_cnt >0 ) +bpf_link__destroy (links[--links_cnt]); } tobj_cleanup: +while (links_cnt >0 ) +bpf_link__destroy (links[--links_cnt]);bpf_object__close (tobj); subtest_cleanup:test__end_subtest ();
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v4 6/9 selftests/bpf: Test gen_prologue and gen_epilogue
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
This test adds a new struct_ops "bpf_testmod_st_ops" in bpf_testmod. The ops of the bpf_testmod_st_ops is triggered by new kfunc calls "bpf_kfunc_st_ops_test_*logue". These new kfunc calls are primarily used by the SEC("syscall") program. The test triggering sequence is like:
SEC("syscall")
syscall_prologue_subprog(struct st_ops_args *args)
bpf_kfunc_st_op_test_prologue(args)
st_ops->test_prologue(args).gen_prologue adds 1000 to args->a .gen_epilogue adds 10000 to args->a .gen_epilogue will also set the r0 to 2 * args->a.
The .gen_prologue and .gen_epilogue of the bpf_testmod_st_ops will test the prog->aux->attach_func_name to decide if it needs to generate codes.
The main programs of the pro_epilogue_subprog.c will call a subprog() which does "args->a += 1".
The main programs of the pro_epilogue_kfunc.c will call a new kfunc bpf_kfunc_st_ops_inc10 which does "args->a += 10".
This patch uses the test_loader infra to check the __xlated instructions patched after gen_prologue and/or gen_epilogue. The __xlated check is based on Eduard's example (Thanks!) in v1.
args->a is returned by the struct_ops prog (either the main prog or the epilogue). Thus, the __retval of the SEC("syscall") prog is checked. For example, when triggering the ops in the 'SEC("struct_ops/test_epilogue_subprog") int test_epilogue_subprog' The expected args->a is +1 (subprog call) + 10000 (.gen_epilogue) = 10001. The expected return value is 2 * 10001 (.gen_epilogue).
--- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 190 ++++++++++++++++++ .../selftests/bpf/bpf_testmod/bpf_testmod.h | 11 + .../bpf/bpf_testmod/bpf_testmod_kfunc.h | 6 + .../selftests/bpf/prog_tests/pro_epilogue.c | 12 ++ .../selftests/bpf/progs/pro_epilogue_kfunc.c | 156 ++++++++++++++ .../bpf/progs/pro_epilogue_subprog.c | 125 ++++++++++++ 6 files changed, 500 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/pro_epilogue.c create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_subprog.c
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +190 -0
@@ -17,6 +17,7 @@#include <linux/in.h> #include <linux/in6.h> #include <linux/un.h> +#include <linux/filter.h> #include <net/sock.h> #include <linux/namei.h> #include "bpf_testmod.h"
@@ -928,6 +929,51 @@ __bpf_kfunc int bpf_kfunc_call_kernel_getpeername(struct addr_args *args)return err; } +static DEFINE_MUTEX (st_ops_mutex); +static struct bpf_testmod_st_ops *st_ops; + +__bpf_kfunc int bpf_kfunc_st_ops_test_prologue(struct st_ops_args *args) +{ +int ret =-1 ; + +mutex_lock (&st_ops_mutex); +if (st_ops && st_ops->test_prologue) + ret = st_ops->test_prologue (args); +mutex_unlock (&st_ops_mutex); + +return ret; +} + +__bpf_kfunc int bpf_kfunc_st_ops_test_epilogue(struct st_ops_args *args) +{ +int ret =-1 ; + +mutex_lock (&st_ops_mutex); +if (st_ops && st_ops->test_epilogue) + ret = st_ops->test_epilogue (args); +mutex_unlock (&st_ops_mutex); + +return ret; +} + +__bpf_kfunc int bpf_kfunc_st_ops_test_pro_epilogue(struct st_ops_args *args) +{ +int ret =-1 ; + +mutex_lock (&st_ops_mutex); +if (st_ops && st_ops->test_pro_epilogue) + ret = st_ops->test_pro_epilogue (args); +mutex_unlock (&st_ops_mutex); + +return ret; +} + +__bpf_kfunc int bpf_kfunc_st_ops_inc10(struct st_ops_args *args) +{ + args->a +=10 ; +return args->a; +} + BTF_KFUNCS_START(bpf_testmod_check_kfunc_ids) BTF_ID_FLAGS(func, bpf_testmod_test_mod_kfunc) BTF_ID_FLAGS(func, bpf_kfunc_call_test1)
@@ -964,6 +1010,10 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_kernel_sendmsg, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_kfunc_call_sock_sendmsg, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_kfunc_call_kernel_getsockname, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_kfunc_call_kernel_getpeername, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_kfunc_st_ops_test_prologue, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_kfunc_st_ops_test_epilogue, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_kfunc_st_ops_test_pro_epilogue, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_kfunc_st_ops_inc10, KF_TRUSTED_ARGS)BTF_KFUNCS_END (bpf_testmod_check_kfunc_ids) static int bpf_testmod_ops_init(struct btf *btf)
@@ -1083,6 +1133,144 @@ struct bpf_struct_ops bpf_testmod_ops2 = { .owner = THIS_MODULE, }; +static int bpf_test_mod_st_ops__test_prologue (struct st_ops_args *args) +{ +return 0 ; +} + +static int bpf_test_mod_st_ops__test_epilogue (struct st_ops_args *args) +{ +return 0 ; +} + +static int bpf_test_mod_st_ops__test_pro_epilogue (struct st_ops_args *args) +{ +return 0 ; +} + +static int st_ops_gen_prologue (struct bpf_insn *insn_buf,bool direct_write, +const struct bpf_prog *prog) +{ +struct bpf_insn *insn = insn_buf; + +if (strcmp (prog->aux->attach_func_name,"test_prologue" ) && +strcmp (prog->aux->attach_func_name,"test_pro_epilogue" )) +return 0 ; + +/* r6 = r1[0]; // r6 will be "struct st_ops *args". r1 is "u64 *ctx". +* r7 = r6->a; +* r7 += 1000; +* r6->a = r7; +*/ + *insn++ =BPF_LDX_MEM (BPF_DW ,BPF_REG_6 ,BPF_REG_1 ,0 ); + *insn++ =BPF_LDX_MEM (BPF_W ,BPF_REG_7 ,BPF_REG_6 , offsetof(struct st_ops_args , a)); + *insn++ =BPF_ALU32_IMM (BPF_ADD ,BPF_REG_7 ,1000 ); + *insn++ =BPF_STX_MEM (BPF_W ,BPF_REG_6 ,BPF_REG_7 , offsetof(struct st_ops_args , a)); + *insn++ = prog->insnsi[0 ]; + +return insn - insn_buf; +} + +static int st_ops_gen_epilogue (struct bpf_insn *insn_buf,const struct bpf_prog *prog, +s16 ctx_stack_off) +{ +struct bpf_insn *insn = insn_buf; + +if (strcmp (prog->aux->attach_func_name,"test_epilogue" ) && +strcmp (prog->aux->attach_func_name,"test_pro_epilogue" )) +return 0 ; + +/* r1 = stack[ctx_stack_off]; // r1 will be "u64 *ctx" +* r1 = r1[0]; // r1 will be "struct st_ops *args" +* r6 = r1->a; +* r6 += 10000; +* r1->a = r6; +* r0 = r6; +* r0 *= 2; +* BPF_EXIT; +*/ + *insn++ =BPF_LDX_MEM (BPF_DW ,BPF_REG_1 ,BPF_REG_FP , ctx_stack_off); + *insn++ =BPF_LDX_MEM (BPF_DW ,BPF_REG_1 ,BPF_REG_1 ,0 ); + *insn++ =BPF_LDX_MEM (BPF_W ,BPF_REG_6 ,BPF_REG_1 , offsetof(struct st_ops_args , a)); + *insn++ =BPF_ALU32_IMM (BPF_ADD ,BPF_REG_6 ,10000 ); + *insn++ =BPF_STX_MEM (BPF_W ,BPF_REG_1 ,BPF_REG_6 , offsetof(struct st_ops_args , a)); + *insn++ =BPF_MOV32_REG (BPF_REG_0 ,BPF_REG_6 ); + *insn++ =BPF_ALU32_IMM (BPF_MUL ,BPF_REG_0 ,2 ); + *insn++ =BPF_EXIT_INSN (); + +return insn - insn_buf; +} + +static int st_ops_btf_struct_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size) +{ +if (off <0 || off + size >sizeof (struct st_ops_args )) +return -EACCES ; +return 0 ; +} + +static const struct bpf_verifier_ops st_ops_verifier_ops = { + .is_valid_access = bpf_testmod_ops_is_valid_access, + .btf_struct_access = st_ops_btf_struct_access, + .gen_prologue = st_ops_gen_prologue, + .gen_epilogue = st_ops_gen_epilogue, + .get_func_proto = bpf_base_func_proto, +}; + +static struct bpf_testmod_st_ops st_ops_cfi_stubs = { + .test_prologue = bpf_test_mod_st_ops__test_prologue, + .test_epilogue = bpf_test_mod_st_ops__test_epilogue, + .test_pro_epilogue = bpf_test_mod_st_ops__test_pro_epilogue, +}; + +static int st_ops_reg (void *kdata,struct bpf_link *link) +{ +int err =0 ; + +mutex_lock (&st_ops_mutex); +if (st_ops) { +pr_err ("st_ops has already been registered\n" ); + err = -EEXIST ; + goto unlock; + } + st_ops = kdata; + +unlock: +mutex_unlock (&st_ops_mutex); +return err; +} + +static void st_ops_unreg (void *kdata,struct bpf_link *link) +{ +mutex_lock (&st_ops_mutex); + st_ops =NULL ; +mutex_unlock (&st_ops_mutex); +} + +static int st_ops_init (struct btf *btf) +{ +return 0 ; +} + +static int st_ops_init_member (const struct btf_type *t, +const struct btf_member *member, +void *kdata,const void *udata) +{ +return 0 ; +} + +static struct bpf_struct_ops testmod_st_ops = { + .verifier_ops = &st_ops_verifier_ops, + .init = st_ops_init, + .init_member = st_ops_init_member, + .reg = st_ops_reg, + .unreg = st_ops_unreg, + .cfi_stubs = &st_ops_cfi_stubs, + .name ="bpf_testmod_st_ops" , + .owner =THIS_MODULE , +}; +extern int bpf_fentry_test1 (int a);static int bpf_testmod_init (void )
@@ -1100,8 +1288,10 @@ static int bpf_testmod_init(void) ret = ret ?:register_btf_kfunc_id_set (BPF_PROG_TYPE_SCHED_CLS , &bpf_testmod_kfunc_set); ret = ret ?:register_btf_kfunc_id_set (BPF_PROG_TYPE_TRACING , &bpf_testmod_kfunc_set); ret = ret ?:register_btf_kfunc_id_set (BPF_PROG_TYPE_SYSCALL , &bpf_testmod_kfunc_set); + ret = ret ?:register_btf_kfunc_id_set (BPF_PROG_TYPE_STRUCT_OPS , &bpf_testmod_kfunc_set); ret = ret ?:register_bpf_struct_ops (&bpf_bpf_testmod_ops, bpf_testmod_ops); ret = ret ?:register_bpf_struct_ops (&bpf_testmod_ops2, bpf_testmod_ops2); + ret = ret ?:register_bpf_struct_ops (&testmod_st_ops, bpf_testmod_st_ops); ret = ret ?:register_btf_id_dtor_kfuncs (bpf_testmod_dtors,ARRAY_SIZE (bpf_testmod_dtors),THIS_MODULE );
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h +11 -0
@@ -94,4 +94,15 @@ struct bpf_testmod_ops2 { int (*test_1)(void); }; +struct st_ops_args { +int a; +}; + +struct bpf_testmod_st_ops { +int (*test_prologue)(struct st_ops_args *args); +int (*test_epilogue)(struct st_ops_args *args); +int (*test_pro_epilogue)(struct st_ops_args *args); +struct module *owner; +}; +#endif /* _BPF_TESTMOD_H */
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h +6 -0
@@ -144,4 +144,10 @@ void bpf_kfunc_dynptr_test(struct bpf_dynptr *ptr, struct bpf_dynptr *ptr__nullastruct bpf_testmod_ctx *bpf_testmod_ctx_create (int *err) __ksym; void bpf_testmod_ctx_release(struct bpf_testmod_ctx *ctx) __ksym; +struct st_ops_args ; +int bpf_kfunc_st_ops_test_prologue(struct st_ops_args *args) __ksym; +int bpf_kfunc_st_ops_test_epilogue(struct st_ops_args *args) __ksym; +int bpf_kfunc_st_ops_test_pro_epilogue(struct st_ops_args *args) __ksym; +int bpf_kfunc_st_ops_inc10(struct st_ops_args *args) __ksym; +#endif /* _BPF_TESTMOD_KFUNC_H */
tools/testing/selftests/bpf/prog_tests/pro_epilogue.c new file +12 -0
@@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include "pro_epilogue_subprog.skel.h" +#include "pro_epilogue_kfunc.skel.h" + +void test_pro_epilogue (void ) +{ +RUN_TESTS (pro_epilogue_subprog); +RUN_TESTS (pro_epilogue_kfunc); +}
tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c new file +156 -0
156 lines not on this page · raw message
tools/testing/selftests/bpf/progs/pro_epilogue_subprog.c new file +125 -0
125 lines not on this page · raw message
Signature · 1 line
2.43.5
Re: [PATCH v4 bpf-next 6/9] selftests/bpf: Test gen_prologue and gen_epilogue
Quoted 37 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This test adds a new
struct_ops"bpf_testmod_st_ops" inbpf_testmod. The ops of thebpf_testmod_st_opsis triggered by new kfunc calls "bpf_kfunc_st_ops_test_*logue". These new kfunc calls are primarily used by theSEC("syscall")program. The test triggering sequence is like:SEC("syscall") syscall_prologue_subprog(struct st_ops_args *args) bpf_kfunc_st_op_test_prologue(args) st_ops->test_prologue(args).gen_prologue adds 1000 to
args->a.gen_epilogue adds 10000 toargs->a.gen_epilogue will also set the r0 to 2 *args->a.The .gen_prologue and .gen_epilogue of the
bpf_testmod_st_opswill test theprog->aux->attach_func_nameto decide if it needs to generate codes.The main programs of the
pro_epilogue_subprog.cwill call asubprog()which does "args->a+= 1".The main programs of the
pro_epilogue_kfunc.cwill call a new kfuncbpf_kfunc_st_ops_inc10which does "args->a+= 10".This patch uses the
test_loaderinfra to check the__xlatedinstructions patched aftergen_prologueand/orgen_epilogue. The__xlatedcheck is based on Eduard's example (Thanks!) in v1.
args->ais returned by thestruct_opsprog (either the main prog or the epilogue). Thus, the__retvalof theSEC("syscall")prog is checked. For example, when triggering the ops in the 'SEC("struct_ops/test_epilogue_subprog")inttest_epilogue_subprog' The expectedargs->ais +1 (subprog call) + 10000 (.gen_epilogue) = 10001. The expected return value is 2 * 10001 (.gen_epilogue).
Quoted 3 lines
Suggested-by Eduard Zingerman <[email protected]>Signed-off-by Martin KaFai Lau <[email protected]>---
[...]
Quoted 12 lines
tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c new file +6
-0@@ -0,0 +1,156 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include "bpf_misc.h"
tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c line 16
Quoted 10 lines
tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c new file +10
-0+#include "../bpf_testmod/bpf_testmod.h" +#include "../bpf_testmod/bpf_testmod_kfunc.h" + +char _license[] SEC("license") = "GPL"; + +void __kfunc_btf_root (void ) +{ +struct st_ops_args args = {}; + +bpf_kfunc_st_ops_inc10 (&args);
Nit: 'bpf_kfunc_st_ops_inc10(0);' would also work.
tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c line 17
Quoted 1 line
tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c new file +1
-0+}
As a side note, I think that kfunc and subprog sets of tests could be combined in order to have less code. Probably does not matter.
[...]
Martin KaFai Lau
reviewerRe: [PATCH v4 bpf-next 6/9] selftests/bpf: Test gen_prologue and gen_epilogue
Quoted 62 lines · Eduard Zingerman
Quoted 37 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This test adds a new
struct_ops"bpf_testmod_st_ops" inbpf_testmod. The ops of thebpf_testmod_st_opsis triggered by new kfunc calls "bpf_kfunc_st_ops_test_*logue". These new kfunc calls are primarily used by theSEC("syscall")program. The test triggering sequence is like:SEC("syscall") syscall_prologue_subprog(struct st_ops_args *args) bpf_kfunc_st_op_test_prologue(args) st_ops->test_prologue(args).gen_prologue adds 1000 to
args->a.gen_epilogue adds 10000 toargs->a.gen_epilogue will also set the r0 to 2 *args->a.The .gen_prologue and .gen_epilogue of the
bpf_testmod_st_opswill test theprog->aux->attach_func_nameto decide if it needs to generate codes.The main programs of the
pro_epilogue_subprog.cwill call asubprog()which does "args->a+= 1".The main programs of the
pro_epilogue_kfunc.cwill call a new kfuncbpf_kfunc_st_ops_inc10which does "args->a+= 10".This patch uses the
test_loaderinfra to check the__xlatedinstructions patched aftergen_prologueand/orgen_epilogue. The__xlatedcheck is based on Eduard's example (Thanks!) in v1.
args->ais returned by thestruct_opsprog (either the main prog or the epilogue). Thus, the__retvalof theSEC("syscall")prog is checked. For example, when triggering the ops in the 'SEC("struct_ops/test_epilogue_subprog")inttest_epilogue_subprog' The expectedargs->ais +1 (subprog call) + 10000 (.gen_epilogue) = 10001. The expected return value is 2 * 10001 (.gen_epilogue).Quoted 3 lines
Suggested-by Eduard Zingerman <[email protected]>Signed-off-by Martin KaFai Lau <[email protected]>---
Acked-by Eduard Zingerman <[email protected]>[...]
Quoted 16 lines
tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c new file +10
-0@@ -0,0 +1,156 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include "bpf_misc.h" +#include "../bpf_testmod/bpf_testmod.h" +#include "../bpf_testmod/bpf_testmod_kfunc.h" + +char _license[] SEC("license") = "GPL";
Quoted 8 lines
tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c line 16
Quoted 6 lines
tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c new file +6
-0+ +void __kfunc_btf_root (void ) +{ +struct st_ops_args args = {}; + +bpf_kfunc_st_ops_inc10 (&args);Nit: '
bpf_kfunc_st_ops_inc10(0);' would also work.
sgtm. I think it will make it obvious that it won't be executed also.
Quoted 5 lines
tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c line 17
Quoted 2 lines
tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c new file +1
-0+}As a side note, I think that kfunc and subprog sets of tests could be combined in order to have less code. Probably does not matter.
ok. I will drop the _subprog.c and only keep the _kfunc.c.
The _kfunc.c calls a subprog and a kfunc which should have already covered the _subprog.c cases.
Martin KaFai Lau
reviewerPATCH v4 7/9 selftests/bpf: Add tailcall epilogue test
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
This patch adds a gen_epilogue test to test a main prog using a bpf_tail_call.
A non test_loader test is used. The tailcall target program, "test_epilogue_subprog", needs to be used in a struct_ops map before it can be loaded. Another struct_ops map is also needed to host the actual "test_epilogue_tailcall" struct_ops program that does the bpf_tail_call. The earlier test_loader patch will attach all struct_ops maps but the bpf_testmod.c does not support >1 attached struct_ops.
The earlier patch used the test_loader which has already covered checking for the patched pro/epilogue instructions. This is done by the __xlated tag.
This patch goes for the regular skel load and syscall test to do the tailcall test that can also allow to directly pass the the "struct st_ops_args *args" as ctx_in to the SEC("syscall") program.
--- .../selftests/bpf/prog_tests/pro_epilogue.c | 38 ++++++++++++ .../selftests/bpf/progs/epilogue_tailcall.c | 58 +++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/epilogue_tailcall.c
tools/testing/selftests/bpf/prog_tests/pro_epilogue.c +38 -0
@@ -4,9 +4,47 @@#include <test_progs.h> #include "pro_epilogue_subprog.skel.h" #include "pro_epilogue_kfunc.skel.h" +#include "epilogue_tailcall.skel.h" + +struct st_ops_args { +int a; +}; + +static void test_tailcall (void ) +{ +LIBBPF_OPTS (bpf_test_run_opts, topts); +struct epilogue_tailcall *skel; +struct st_ops_args args; +int err, prog_fd; + + skel =epilogue_tailcall__open_and_load (); +if (!ASSERT_OK_PTR (skel,"epilogue_tailcall__open_and_load" )) +return ; + + topts.ctx_in = &args; + topts.ctx_size_in =sizeof (args); + + skel->links.epilogue_tailcall = +bpf_map__attach_struct_ops (skel->maps.epilogue_tailcall); +if (!ASSERT_OK_PTR (skel->links.epilogue_tailcall,"attach_struct_ops" )) + goto done; + +/* tailcall prog + gen_epilogue */ +memset (&args,0 ,sizeof (args)); + prog_fd =bpf_program__fd (skel->progs.syscall_epilogue_tailcall); + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"bpf_prog_test_run_opts" ); +ASSERT_EQ (args.a,10001 ,"args.a" ); +ASSERT_EQ (topts.retval,10001 *2 ,"topts.retval" ); + +done: +epilogue_tailcall__destroy (skel); +}void test_pro_epilogue (void ) {RUN_TESTS (pro_epilogue_subprog);RUN_TESTS (pro_epilogue_kfunc); +if (test__start_subtest ("tailcall" )) +test_tailcall (); }
tools/testing/selftests/bpf/progs/epilogue_tailcall.c new file +58 -0
58 lines not on this page · raw message
Signature · 1 line
2.43.5
Re: [PATCH v4 bpf-next 7/9] selftests/bpf: Add tailcall epilogue test
Quoted 17 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch adds a
gen_epiloguetest to test a main prog using abpf_tail_call.A non
test_loadertest is used. The tailcall target program, "test_epilogue_subprog", needs to be used in astruct_opsmap before it can be loaded. Anotherstruct_opsmap is also needed to host the actual "test_epilogue_tailcall"struct_opsprogram that does thebpf_tail_call. The earliertest_loaderpatch will attach allstruct_opsmaps but thebpf_testmod.cdoes not support >1 attachedstruct_ops.The earlier patch used the
test_loaderwhich has already covered checking for the patched pro/epilogue instructions. This is done by the__xlatedtag.
Quoted 7 lines
This patch goes for the regular skel load and syscall test to do the tailcall test that can also allow to directly pass the the "struct
st_ops_args*args" asctx_into theSEC("syscall")program.Signed-off-by Martin KaFai Lau <[email protected]>---
[...]
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/pro_epilogue.c +9
-0+static void test_tailcall (void ) +{ +LIBBPF_OPTS (bpf_test_run_opts, topts); +struct epilogue_tailcall *skel; +struct st_ops_args args; +int err, prog_fd; + + skel =epilogue_tailcall__open_and_load (); +if (!ASSERT_OK_PTR (skel,"epilogue_tailcall__open_and_load" ))
tools/testing/selftests/bpf/prog_tests/pro_epilogue.c line 31
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/pro_epilogue.c +10
-0+return ; + + topts.ctx_in = &args; + topts.ctx_size_in =sizeof (args); + + skel->links.epilogue_tailcall = +bpf_map__attach_struct_ops (skel->maps.epilogue_tailcall); +if (!ASSERT_OK_PTR (skel->links.epilogue_tailcall,"attach_struct_ops" )) + goto done; +
Nitpick:
Both test_epilogue_tailcall and test_epilogue_subprog would be augmented with epilogue, and we know that tail call run as expected because only test_epilogue_subprog does +1, right?
If above is true, could you please update the comment a bit, e.g.:
/* Both test_epilogue_tailcall and test_epilogue_subprog are
* augmented with epilogue. When syscall_epilogue_tailcall()
* is run test_epilogue_tailcall() is triggered,
* it executes a tail call and control is transferred to
* test_epilogue_subprog(). Only test_epilogue_subprog()
* does args->a += 1, thus final args.a value of 10001
* guarantees that tail call was executed as expected. */
(For some reason it took me a while to understand what happens in this test)
Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/pro_epilogue.c +11
-0+/* tailcall prog + gen_epilogue */ +memset (&args,0 ,sizeof (args)); + prog_fd =bpf_program__fd (skel->progs.syscall_epilogue_tailcall); + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"bpf_prog_test_run_opts" ); +ASSERT_EQ (args.a,10001 ,"args.a" ); +ASSERT_EQ (topts.retval,10001 *2 ,"topts.retval" ); + +done: +epilogue_tailcall__destroy (skel); +}
[...]
Martin KaFai Lau
reviewerRe: [PATCH v4 bpf-next 7/9] selftests/bpf: Add tailcall epilogue test
Quoted 46 lines · Eduard Zingerman
Quoted 17 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch adds a
gen_epiloguetest to test a main prog using abpf_tail_call.A non
test_loadertest is used. The tailcall target program, "test_epilogue_subprog", needs to be used in astruct_opsmap before it can be loaded. Anotherstruct_opsmap is also needed to host the actual "test_epilogue_tailcall"struct_opsprogram that does thebpf_tail_call. The earliertest_loaderpatch will attach allstruct_opsmaps but thebpf_testmod.cdoes not support >1 attachedstruct_ops.The earlier patch used the
test_loaderwhich has already covered checking for the patched pro/epilogue instructions. This is done by the__xlatedtag.Quoted 7 lines
This patch goes for the regular skel load and syscall test to do the tailcall test that can also allow to directly pass the the "struct
st_ops_args*args" asctx_into theSEC("syscall")program.Signed-off-by Martin KaFai Lau <[email protected]>---
Acked-by Eduard Zingerman <[email protected]>[...]
Quoted 16 lines
tools/testing/selftests/bpf/prog_tests/pro_epilogue.c +16
-0+static void test_tailcall (void ) +{ +LIBBPF_OPTS (bpf_test_run_opts, topts); +struct epilogue_tailcall *skel; +struct st_ops_args args; +int err, prog_fd; + + skel =epilogue_tailcall__open_and_load (); +if (!ASSERT_OK_PTR (skel,"epilogue_tailcall__open_and_load" )) +return ; + + topts.ctx_in = &args; + topts.ctx_size_in =sizeof (args); + + skel->links.epilogue_tailcall = +bpf_map__attach_struct_ops (skel->maps.epilogue_tailcall);
Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/pro_epilogue.c line 31
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/pro_epilogue.c +3
-0+if (!ASSERT_OK_PTR (skel->links.epilogue_tailcall,"attach_struct_ops" )) + goto done; +Nitpick:
Bothtest_epilogue_tailcallandtest_epilogue_subprogwould be augmented with epilogue, and we know that tail call run as expected because onlytest_epilogue_subprogdoes +1, right?
Yes. and also the epilogue of the test_epilogue_subprog is executed.
Quoted 11 lines
If above is true, could you please update the comment a bit, e.g.:
/* Both
test_epilogue_tailcallandtest_epilogue_subprogare
* augmented with epilogue. Whensyscall_epilogue_tailcall()
* is runtest_epilogue_tailcall()is triggered,
* it executes a tail call and control is transferred to
*test_epilogue_subprog(). Onlytest_epilogue_subprog()
* doesargs->a+= 1, thus final args.a value of 10001
* guarantees that tail call was executed as expected. */
Added. I massaged the wordings a little.
Martin KaFai Lau
reviewerPATCH v4 8/9 selftests/bpf: A pro/epilogue test when the main prog jumps back to the 1st insn
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
This patch adds a pro/epilogue test when the main prog has a goto insn that goes back to the very first instruction of the prog. It is to test the correctness of the adjust_jmp_off(prog, 0, delta) after the verifier has applied the prologue and/or epilogue patch.
--- .../selftests/bpf/prog_tests/pro_epilogue.c | 2 + .../bpf/progs/pro_epilogue_goto_start.c | 149 ++++++++++++++++++ 2 files changed, 151 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_goto_start.c
tools/testing/selftests/bpf/prog_tests/pro_epilogue.c +2 -0
@@ -5,6 +5,7 @@#include "pro_epilogue_subprog.skel.h" #include "pro_epilogue_kfunc.skel.h" #include "epilogue_tailcall.skel.h" +#include "pro_epilogue_goto_start.skel.h" struct st_ops_args {int a;
@@ -45,6 +46,7 @@ void test_pro_epilogue(void) {RUN_TESTS (pro_epilogue_subprog);RUN_TESTS (pro_epilogue_kfunc); +RUN_TESTS (pro_epilogue_goto_start);if (test__start_subtest ("tailcall" ))test_tailcall (); }
tools/testing/selftests/bpf/progs/pro_epilogue_goto_start.c new file +149 -0
149 lines not on this page · raw message
Signature · 1 line
2.43.5
Re: [PATCH v4 bpf-next 8/9] selftests/bpf: A pro/epilogue test when the main prog jumps back to the 1st insn
Quoted 9 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch adds a pro/epilogue test when the main prog has a goto insn that goes back to the very first instruction of the prog. It is to test the correctness of the
adjust_jmp_off(prog, 0, delta)after the verifier has applied the prologue and/or epilogue patch.Signed-off-by Martin KaFai Lau <[email protected]>---
[...]
Martin KaFai Lau
reviewerPATCH v4 9/9 selftests/bpf: Test epilogue patching when the main prog has multiple BPF_EXIT
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
This patch tests the epilogue patching when the main prog has multiple BPF_EXIT. The verifier should have patched the 2nd (and later) BPF_EXIT with a BPF_JA that goes back to the earlier patched epilogue instructions.
--- .../selftests/bpf/prog_tests/pro_epilogue.c | 2 + .../selftests/bpf/progs/epilogue_exit.c | 78 +++++++++++++++++++ 2 files changed, 80 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/epilogue_exit.c
tools/testing/selftests/bpf/prog_tests/pro_epilogue.c +2 -0
@@ -6,6 +6,7 @@#include "pro_epilogue_kfunc.skel.h" #include "epilogue_tailcall.skel.h" #include "pro_epilogue_goto_start.skel.h" +#include "epilogue_exit.skel.h" struct st_ops_args {int a;
@@ -47,6 +48,7 @@ void test_pro_epilogue(void)RUN_TESTS (pro_epilogue_subprog);RUN_TESTS (pro_epilogue_kfunc);RUN_TESTS (pro_epilogue_goto_start); +RUN_TESTS (epilogue_exit);if (test__start_subtest ("tailcall" ))test_tailcall (); }
tools/testing/selftests/bpf/progs/epilogue_exit.c new file +78 -0
78 lines not on this page · raw message
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerRe: [PATCH v4 bpf-next 9/9] selftests/bpf: Test epilogue patching when the main prog has multiple BPF_EXIT
Quoted 78 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch tests the epilogue patching when the main prog has multiple
BPF_EXIT. The verifier should have patched the 2nd (and later)BPF_EXITwith aBPF_JAthat goes back to the earlier patched epilogue instructions.Signed-off-by Martin KaFai Lau <[email protected]>--- .../selftests/bpf/prog_tests/pro_epilogue.c | 2 + .../selftests/bpf/progs/epilogue_exit.c | 78 +++++++++++++++++++ 2 files changed, 80 insertions(+)create mode 100644
tools/testing/selftests/bpf/progs/epilogue_exit.ctools/testing/selftests/bpf/prog_tests/pro_epilogue.c +2
-0@@ -6,6 +6,7 @@#include "pro_epilogue_kfunc.skel.h" #include "epilogue_tailcall.skel.h" #include "pro_epilogue_goto_start.skel.h" +#include "epilogue_exit.skel.h" struct st_ops_args {int a;@@ -47,6 +48,7 @@ void test_pro_epilogue(void)RUN_TESTS (pro_epilogue_subprog);RUN_TESTS (pro_epilogue_kfunc);RUN_TESTS (pro_epilogue_goto_start); +RUN_TESTS (epilogue_exit);if (test__start_subtest ("tailcall" ))test_tailcall (); }tools/testing/selftests/bpf/progs/epilogue_exit.c new file +38
-0@@ -0,0 +1,78 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include "bpf_misc.h" +#include "../bpf_testmod/bpf_testmod.h" +#include "../bpf_testmod/bpf_testmod_kfunc.h" + +char _license[] SEC("license") = "GPL"; + +__success +/* save __u64 *ctx to stack */ +__xlated("0: *(u64 *)(r10 -8) = r1") +/* main prog */ +__xlated("1: r1 = *(u64 *)(r1 +0)") +__xlated("2: r2 = *(u32 *)(r1 +0)") +__xlated("3: if r2 == 0x0 goto pc+10") +__xlated("4: r0 = 0") +__xlated ("5: *(u32 *)(r1 +0) = 0" ) +/* epilogue */ +__xlated("6: r1 = *(u64 *)(r10 -8)") +__xlated("7: r1 = *(u64 *)(r1 +0)") +__xlated("8: r6 = *(u32 *)(r1 +0)") +__xlated("9: w6 += 10000") +__xlated("10: *(u32 *)(r1 +0) = r6") +__xlated("11: w0 = w6") +__xlated("12: w0 *= 2") +__xlated ("13: exit" ) +/* 2nd part of the main prog after the first exit */ +__xlated("14: *(u32 *)(r1 +0) = 1") +__xlated ("15: r0 = 1" ) +/* Clear the r1 to ensure it does not have +* off-by-1 error and ensure it jumps back to the +* beginning of epilogue which initializes +* the r1 with the ctx ptr. + */ +__xlated ("16: r1 = 0" )
tools/testing/selftests/bpf/progs/epilogue_exit.c line 48
Quoted 10 lines
tools/testing/selftests/bpf/progs/epilogue_exit.c new file +10
-0+__xlated("17: gotol pc-12") +SEC("struct_ops/test_epilogue_exit") +__naked int test_epilogue_exit (void ) +{ + asmvolatile ( +"r1 = *(u64 *)(r1 +0);" +"r2 = *(u32 *)(r1 +0);" +"if r2 == 0 goto +3;" +"r0 = 0;" + "*(u32 *)(r1 + 0) = 0;"
llvm17 cannot take "*(u32 *)(r1 +0) = 0".
Instead:
r3 = 0;
*(u32 *)(r1 + 0) = r3;
The above solved the llvm17 error:
https://github.com/kernel-patches/bpf/actions/runs/10586206183/job/29334690461
However, there is still a zext with s390 that added extra insn and failed the __xlated check. will try an adjustment in the tests to avoid the zext.
pw-bot: cr
Quoted 30 lines
tools/testing/selftests/bpf/progs/epilogue_exit.c new file +30
-0+"exit;" +"*(u32 *)(r1 + 0) = 1;" +"r0 = 1;" +"r1 = 0;" +"exit;" + ::: __clobber_all); +} + +SEC(".struct_ops.link") +struct bpf_testmod_st_ops epilogue_exit = { + .test_epilogue = (void *)test_epilogue_exit, +}; + +SEC("syscall") +__retval(20000) +int syscall_epilogue_exit0 (void *ctx) +{ +struct st_ops_args args = { .a =1 }; + +return bpf_kfunc_st_ops_test_epilogue (&args); +} + +SEC("syscall") +__retval(20002) +int syscall_epilogue_exit1 (void *ctx) +{ +struct st_ops_args args = {}; + +return bpf_kfunc_st_ops_test_epilogue (&args); +}
Eduard Zingerman
maintainerRe: [PATCH v4 bpf-next 9/9] selftests/bpf: Test epilogue patching when the main prog has multiple BPF_EXIT
On Tue, 2024-08-27 at 17:58 -0700, Martin KaFai Lau wrote:
[...]
Quoted 14 lines
tools/testing/selftests/bpf/progs/epilogue_exit.c line 48
Quoted 9 lines
tools/testing/selftests/bpf/progs/epilogue_exit.c new file +9
-0+SEC("struct_ops/test_epilogue_exit") +__naked int test_epilogue_exit (void ) +{ + asmvolatile ( +"r1 = *(u64 *)(r1 +0);" +"r2 = *(u32 *)(r1 +0);" +"if r2 == 0 goto +3;" +"r0 = 0;" + "*(u32 *)(r1 + 0) = 0;"llvm17 cannot take "*(u32 *)(r1 +0) = 0".
Instead:
Quoted 8 lines
r3 = 0;
*(u32 *)(r1 + 0) = r3;The above solved the llvm17 error:
https://github.com/kernel-patches/bpf/actions/runs/10586206183/job/29334690461However, there is still a zext with s390 that added extra insn and failed the
__xlatedcheck. will try an adjustment in the tests to avoid the zext.
Another option would be to limit archs for the test, e.g. use __arch_x86_64 and __arch_arm64.
Martin KaFai Lau
reviewerRe: [PATCH v4 bpf-next 9/9] selftests/bpf: Test epilogue patching when the main prog has multiple BPF_EXIT
Quoted 21 lines · Eduard Zingerman
On Tue, 2024-08-27 at 17:58 -0700, Martin KaFai Lau wrote:
[...]
Quoted 17 lines
tools/testing/selftests/bpf/progs/epilogue_exit.c line 48
Quoted 9 lines
tools/testing/selftests/bpf/progs/epilogue_exit.c new file +9
-0+SEC("struct_ops/test_epilogue_exit") +__naked int test_epilogue_exit (void ) +{ + asmvolatile ( +"r1 = *(u64 *)(r1 +0);" +"r2 = *(u32 *)(r1 +0);" +"if r2 == 0 goto +3;" +"r0 = 0;" + "*(u32 *)(r1 + 0) = 0;"llvm17 cannot take "*(u32 *)(r1 +0) = 0".
Instead:
r3 = 0;
*(u32 *)(r1 + 0) = r3;
Quoted 8 lines
Quoted 5 lines
The above solved the llvm17 error:
https://github.com/kernel-patches/bpf/actions/runs/10586206183/job/29334690461However, there is still a zext with s390 that added extra insn and failed the
__xlatedcheck. will try an adjustment in the tests to avoid the zext.Another option would be to limit archs for the test, e.g. use
__arch_x86_64and__arch_arm64.
Ah. good to know.
I have used all 64bits ops to solve it. It passed the earlier bpf CI run:
https://github.com/kernel-patches/bpf/actions/runs/10590714532
I will respin with the 64bits ops solution.
Re: [PATCH v4 bpf-next 9/9] selftests/bpf: Test epilogue patching when the main prog has multiple BPF_EXIT
In reply to Martin KaFai LauQuoted 9 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch tests the epilogue patching when the main prog has multiple
BPF_EXIT. The verifier should have patched the 2nd (and later)BPF_EXITwith aBPF_JAthat goes back to the earlier patched epilogue instructions.Signed-off-by Martin KaFai Lau <[email protected]>---
[...]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Eduard Zingerman
maintainerRe: [PATCH v4 bpf-next 2/9] bpf: Adjust BPF_JMP that jumps to the 1st insn of the prologue
On Tue, 2024-08-27 at 12:48 -0700, Martin KaFai Lau wrote:
[...]
Review static int convert_ctx_accesses(struct bpf_verifier_env *env) · line 19710
Quoted 7 lines
(quoted diff) +3
-0@@ -19705,6 +19705,9 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) } } + if (delta) + WARN_ON(adjust_jmp_off(env->prog, 0, delta, delta)); +
I double checked my old calculations and agree that this adjustment is necessary. Alexei's suggestion to use delta == skip_cnt sounds reasonable.
Quoted 3 lines
(quoted diff) +0
-0if (bpf_prog_is_offloaded(env->prog->aux)) return 0;
[...]