bpf: Search and add kfuncs in struct_ops prologue and epilogue
Subsystems (5)
PATCH v2 1/2 bpf: Search and add kfuncs in struct_ops prologue and epilogue
From: Amery Hung <[email protected]>
Currently, add_kfunc_call() is only invoked once before the main verification loop. Therefore, the verifier could not find the bpf_kfunc_btf_tab of a new kfunc call which is not seen in user defined struct_ops operators but introduced in gen_prologue or gen_epilogue during do_misc_fixup(). Fix this by searching kfuncs in the patching instruction buffer and add them to prog->aux->kfunc_tab.
--- kernel/bpf/verifier.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-)
kernel/bpf/verifier.c +24 -1
@@ -3215,6 +3215,21 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog,return res ? &res->func_model :NULL ; } +static int add_kfunc_in_insns (struct bpf_verifier_env *env, +struct bpf_insn *insn,int cnt) +{ +int i, ret; + +for (i =0 ; i < cnt; i++, insn++) { +if (bpf_pseudo_kfunc_call (insn)) { + ret =add_kfunc_call (env, insn->imm, insn->off); +if (ret <0 ) +return ret; + } + } +return 0 ; +} +static int add_subprog_and_kfunc (struct bpf_verifier_env *env) {struct bpf_subprog_info *subprog = env->subprog_info;
@@ -20368,7 +20383,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) {struct bpf_subprog_info *subprogs = env->subprog_info;const struct bpf_verifier_ops *ops = env->ops; -int i, cnt, size, ctx_field_size, delta =0 , epilogue_cnt =0 ; +int i, cnt, size, ctx_field_size, ret, delta =0 , epilogue_cnt =0 ;const int insn_cnt = env->prog->len;struct bpf_insn *epilogue_buf = env->epilogue_buf;struct bpf_insn *insn_buf = env->insn_buf;
@@ -20397,6 +20412,10 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)return -ENOMEM ; env->prog = new_prog; delta += cnt -1 ; + + ret =add_kfunc_in_insns (env, epilogue_buf, epilogue_cnt -1 ); +if (ret <0 ) +return ret; } }
@@ -20417,6 +20436,10 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) env->prog = new_prog; delta += cnt -1 ; + + ret =add_kfunc_in_insns (env, insn_buf, cnt -1 ); +if (ret <0 ) +return ret; } }
Signature · 1 line
2.47.1
PATCH v2 2/2 selftests/bpf: Test gen_pro/epilogue that generate kfuncs
Test gen_prologue and gen_epilogue that generate kfuncs that have not been seen in the main program.
The main bpf program and return value checks are identical to pro_epilogue.c introduced in commit 47e69431b57a ("selftests/bpf: Test gen_prologue and gen_epilogue"). However, now when bpf_testmod_st_ops detects a program name with prefix "test_kfunc_", it generates slightly different prologue and epilogue: They still add 1000 to args->a in prologue, add 10000 to args->a and set r0 to 2 * args->a in epilogue, but involve kfuncs.
At high level, the alternative version of prologue and epilogue look like this:
cgrp = bpf_cgroup_from_id(0);
if (cgrp)
bpf_cgroup_release(cgrp);
else
/* Perform what original bpf_testmod_st_ops prologue or
* epilogue does
*/Since 0 is never a valid cgroup id, the original prologue or epilogue logic will be performed. As a result, the __retval check should expect the exact same return value.
--- include/linux/filter.h | 10 + kernel/bpf/btf.c | 1 + .../selftests/bpf/prog_tests/pro_epilogue.c | 2 + .../bpf/progs/pro_epilogue_with_kfunc.c | 182 ++++++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 90 +++++++++ 5 files changed, 285 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_with_kfunc.c
include/linux/filter.h +10 -0
@@ -469,6 +469,16 @@ static inline bool insn_is_cast_user(const struct bpf_insn *insn) .off = 0, \ .imm = BPF_CALL_IMM(FUNC) }) +/* Kfunc call */ + +#define BPF_CALL_KFUNC (OFF ,IMM ) \ + ((struct bpf_insn) { \ + .code = BPF_JMP | BPF_CALL, \ + .dst_reg = 0, \ + .src_reg = BPF_PSEUDO_KFUNC_CALL, \ + .off = OFF, \ + .imm = IMM }) +/* Raw code statement block */ #define BPF_RAW_INSN (CODE ,DST ,SRC ,OFF ,IMM ) \
kernel/bpf/btf.c +1 -0
@@ -606,6 +606,7 @@ s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)spin_unlock_bh (&btf_idr_lock);return ret; } +EXPORT_SYMBOL_GPL (bpf_find_btf_id);const struct btf_type *btf_type_skip_modifiers (const struct btf *btf,u32 id,u32 *res_id)
tools/testing/selftests/bpf/prog_tests/pro_epilogue.c +2 -0
@@ -6,6 +6,7 @@#include "epilogue_tailcall.skel.h" #include "pro_epilogue_goto_start.skel.h" #include "epilogue_exit.skel.h" +#include "pro_epilogue_with_kfunc.skel.h" struct st_ops_args {__u64 a;
@@ -55,6 +56,7 @@ void test_pro_epilogue(void)RUN_TESTS (pro_epilogue);RUN_TESTS (pro_epilogue_goto_start);RUN_TESTS (epilogue_exit); +RUN_TESTS (pro_epilogue_with_kfunc);if (test__start_subtest ("tailcall" ))test_tailcall (); }
tools/testing/selftests/bpf/progs/pro_epilogue_with_kfunc.c new file +182 -0
182 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +90 -0
114 lines not on this page · raw message
Signature · 1 line
2.47.1
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 2/2] selftests/bpf: Test gen_pro/epilogue that generate kfuncs
On Thu, 2025-02-20 at 13:25 -0800, Amery Hung wrote:
[...]
Given that prologue and epilogue generation is already tested, it appears that it would be sufficient to add only two tests:
'test_kfunc_pro_epilogue' / 'syscall_pro_epilogue'.
Not sure if testing prologue and epilogue generation separately adds much value in this context, wdyt?
[...]
kernel/bpf/btf.c s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p) · line 609
Quoted 9 lines
kernel/bpf/btf.c +1
-0@@ -606,6 +606,7 @@ s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)spin_unlock_bh (&btf_idr_lock);return ret; } +EXPORT_SYMBOL_GPL (bpf_find_btf_id);
I think this is not necessary, see below.
[...]
kernel/bpf/btf.c static void st_ops_unreg(void *kdata, struct bpf_link *link) · line 1499
Quoted 8 lines
kernel/bpf/btf.c +4
-0@@ -1410,6 +1493,13 @@ static void st_ops_unreg(void *kdata, struct bpf_link *link)static int st_ops_init (struct btf *btf) { +struct btf *kfunc_btf; + + bpf_cgroup_from_id_id =bpf_find_btf_id ("bpf_cgroup_from_id" ,BTF_KIND_FUNC , &kfunc_btf); + bpf_cgroup_release_id =bpf_find_btf_id ("bpf_cgroup_release" ,BTF_KIND_FUNC , &kfunc_btf);
Maybe use BTF_ID_LIST for this?
E.g. BTF_ID_LIST(bpf_testmod_dtor_ids) in this file, or
BTF_ID_LIST(special_kfunc_list) in verifier.c?
(Just in case, sorry if you know this already,
BTF_ID_LIST declares are set of symbols with special suffix/prefix, at build time tools/bpf/resolve_btfids looks for such symbols and patches their values to correspond to BTF ids of specified functions and structures).
Quoted 6 lines
kernel/bpf/btf.c +3
-0+if (!bpf_cgroup_from_id_id || !bpf_cgroup_release_id) +return -EINVAL ; +return 0 ; }
Re: [PATCH bpf-next v2 2/2] selftests/bpf: Test gen_pro/epilogue that generate kfuncs
Quoted 11 lines · Eduard Zingerman
On Thu, 2025-02-20 at 13:25 -0800, Amery Hung wrote:
[...]
Given that prologue and epilogue generation is already tested, it appears that it would be sufficient to add only two tests:
'test_kfunc_pro_epilogue' / 'syscall_pro_epilogue'.
Not sure if testing prologue and epilogue generation separately adds much value in this context, wdyt?
Agree. I will only keep the syscall_pro_epilogue test.
Quoted 29 lines
[...]
kernel/bpf/btf.c s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p) · line 609
Quoted 9 lines
kernel/bpf/btf.c +1
-0@@ -606,6 +606,7 @@ s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)spin_unlock_bh (&btf_idr_lock);return ret; } +EXPORT_SYMBOL_GPL (bpf_find_btf_id);I think this is not necessary, see below.
[...]
kernel/bpf/btf.c static void st_ops_unreg(void *kdata, struct bpf_link *link) · line 1499
Quoted 8 lines
kernel/bpf/btf.c +4
-0@@ -1410,6 +1493,13 @@ static void st_ops_unreg(void *kdata, struct bpf_link *link)static int st_ops_init (struct btf *btf) { +struct btf *kfunc_btf; + + bpf_cgroup_from_id_id =bpf_find_btf_id ("bpf_cgroup_from_id" ,BTF_KIND_FUNC , &kfunc_btf); + bpf_cgroup_release_id =bpf_find_btf_id ("bpf_cgroup_release" ,BTF_KIND_FUNC , &kfunc_btf);Maybe use
BTF_ID_LISTfor this?
E.g.BTF_ID_LIST(bpf_testmod_dtor_ids)in this file, or
BTF_ID_LIST(special_kfunc_list)inverifier.c?
Quoted 5 lines
(Just in case, sorry if you know this already,
BTF_ID_LISTdeclares are set of symbols with special suffix/prefix, at build timetools/bpf/resolve_btfidslooks for such symbols and patches their values to correspond to BTF ids of specified functions and structures).
Ah yes. It is an artifact when I was testing a patch for generating kfunc in module btf. But since there is no use case, I removed that part. I will change it to BTF_ID_LIST. Thanks for catching this.
Thanks,
Amery
Quoted 8 lines
Quoted 6 lines
kernel/bpf/btf.c +3
-0+if (!bpf_cgroup_from_id_id || !bpf_cgroup_release_id) +return -EINVAL ; +return 0 ; }
Re: [PATCH bpf-next v2 2/2] selftests/bpf: Test gen_pro/epilogue that generate kfuncs
Quoted 51 lines · Amery Hung
Quoted 11 lines · Eduard Zingerman
On Thu, 2025-02-20 at 13:25 -0800, Amery Hung wrote:
[...]
Given that prologue and epilogue generation is already tested, it appears that it would be sufficient to add only two tests:
'test_kfunc_pro_epilogue' / 'syscall_pro_epilogue'.
Not sure if testing prologue and epilogue generation separately adds much value in this context, wdyt?Agree. I will only keep the
syscall_pro_epiloguetest.Quoted 29 lines
[...]
kernel/bpf/btf.c s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p) · line 609
Quoted 9 lines
kernel/bpf/btf.c +1
-0@@ -606,6 +606,7 @@ s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)spin_unlock_bh (&btf_idr_lock);return ret; } +EXPORT_SYMBOL_GPL (bpf_find_btf_id);I think this is not necessary, see below.
[...]
kernel/bpf/btf.c static void st_ops_unreg(void *kdata, struct bpf_link *link) · line 1499
Quoted 8 lines
kernel/bpf/btf.c +4
-0@@ -1410,6 +1493,13 @@ static void st_ops_unreg(void *kdata, struct bpf_link *link)static int st_ops_init (struct btf *btf) { +struct btf *kfunc_btf; + + bpf_cgroup_from_id_id =bpf_find_btf_id ("bpf_cgroup_from_id" ,BTF_KIND_FUNC , &kfunc_btf); + bpf_cgroup_release_id =bpf_find_btf_id ("bpf_cgroup_release" ,BTF_KIND_FUNC , &kfunc_btf);Maybe use
BTF_ID_LISTfor this?
E.g.BTF_ID_LIST(bpf_testmod_dtor_ids)in this file, or
BTF_ID_LIST(special_kfunc_list)inverifier.c?Quoted 5 lines
(Just in case, sorry if you know this already,
BTF_ID_LISTdeclares are set of symbols with special suffix/prefix, at build timetools/bpf/resolve_btfidslooks for such symbols and patches their values to correspond to BTF ids of specified functions and structures).
Quoted 4 lines
Ah yes. It is an artifact when I was testing a patch for generating kfunc in module btf. But since there is no use case, I removed that part. I will change it to
BTF_ID_LIST. Thanks for catching this.
Actually when I use BTF_ID_LIST with a kernel kfunc, I got the warning below. Since it was not able to resolve the btf id, the test program failed to load as the generated byte code will contain invalid kfunc id.
BTF [M] bpf_testmod.ko
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
MOD bpf_testmod.ko
I am not familiar with how resolve_btfids work, specifically when building a kernel module. Do you have any suggestions?
Thanks,
Amery
Quoted 11 lines
Thanks,
AmeryQuoted 8 lines
Quoted 6 lines
kernel/bpf/btf.c +3
-0+if (!bpf_cgroup_from_id_id || !bpf_cgroup_release_id) +return -EINVAL ; +return 0 ; }
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 2/2] selftests/bpf: Test gen_pro/epilogue that generate kfuncs
Quoted 62 lines · Amery Hung
Quoted 51 lines · Amery Hung
Quoted 11 lines · Eduard Zingerman
On Thu, 2025-02-20 at 13:25 -0800, Amery Hung wrote:
[...]
Given that prologue and epilogue generation is already tested, it appears that it would be sufficient to add only two tests:
'test_kfunc_pro_epilogue' / 'syscall_pro_epilogue'.
Not sure if testing prologue and epilogue generation separately adds much value in this context, wdyt?Agree. I will only keep the
syscall_pro_epiloguetest.Quoted 29 lines
[...]
kernel/bpf/btf.c s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p) · line 609
Quoted 9 lines
kernel/bpf/btf.c +1
-0@@ -606,6 +606,7 @@ s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)spin_unlock_bh (&btf_idr_lock);return ret; } +EXPORT_SYMBOL_GPL (bpf_find_btf_id);I think this is not necessary, see below.
[...]
kernel/bpf/btf.c static void st_ops_unreg(void *kdata, struct bpf_link *link) · line 1499
Quoted 8 lines
kernel/bpf/btf.c +4
-0@@ -1410,6 +1493,13 @@ static void st_ops_unreg(void *kdata, struct bpf_link *link)static int st_ops_init (struct btf *btf) { +struct btf *kfunc_btf; + + bpf_cgroup_from_id_id =bpf_find_btf_id ("bpf_cgroup_from_id" ,BTF_KIND_FUNC , &kfunc_btf); + bpf_cgroup_release_id =bpf_find_btf_id ("bpf_cgroup_release" ,BTF_KIND_FUNC , &kfunc_btf);Maybe use
BTF_ID_LISTfor this?
E.g.BTF_ID_LIST(bpf_testmod_dtor_ids)in this file, or
BTF_ID_LIST(special_kfunc_list)inverifier.c?Quoted 5 lines
(Just in case, sorry if you know this already,
BTF_ID_LISTdeclares are set of symbols with special suffix/prefix, at build timetools/bpf/resolve_btfidslooks for such symbols and patches their values to correspond to BTF ids of specified functions and structures).Quoted 4 lines
Ah yes. It is an artifact when I was testing a patch for generating kfunc in module btf. But since there is no use case, I removed that part. I will change it to
BTF_ID_LIST. Thanks for catching this.Actually when I use
BTF_ID_LISTwith a kernel kfunc, I got the warning below. Since it was not able to resolve the btf id, the test program failed to load as the generated byte code will contain invalid kfunc id.
Quoted 7 lines
BTF [M] bpf_testmod.ko
WARN:resolve_btfids: unresolved symbolbpf_cgroup_release
WARN:resolve_btfids: unresolved symbolbpf_cgroup_from_id
MOD bpf_testmod.koI am not familiar with how
resolve_btfidswork, specifically when building a kernel module. Do you have any suggestions?
Looks like there is no way.resolve_btfids is called for module as follows:
resolve_btfids -b <path-to>/vmlinux bpf_testmod.ko
Where -b specifies base BTF.
However, the bpf_testmod.ko has .BTF.base section, where distilled BTF is stored. In such case tools/bpf/resolve_btfids/main.c:elf_collect() overrides base passed from command line, and uses .BTF.base instead. However, bpf_cgroup_release is not referenced in bpf_testmod.c, thus it does not get into distilled BTF. Hence, its id remains unresolved. And it cannot be referenced in bpf_testmod.c, because it is not an exported symbol.
So, for this particular case there are only two options: resolve id dynamically, as you did, or use a kfunc internal to this module instead (which should simplify the test, imo).
The broader question if want a capability to use BTF_ID_LIST referring to kernel functions from modules remains open.
[...]
Re: [PATCH bpf-next v2 2/2] selftests/bpf: Test gen_pro/epilogue that generate kfuncs
Quoted 77 lines · Eduard Zingerman
Quoted 62 lines · Amery Hung
Quoted 51 lines · Amery Hung
Quoted 11 lines · Eduard Zingerman
On Thu, 2025-02-20 at 13:25 -0800, Amery Hung wrote:
[...]
Given that prologue and epilogue generation is already tested, it appears that it would be sufficient to add only two tests:
'test_kfunc_pro_epilogue' / 'syscall_pro_epilogue'.
Not sure if testing prologue and epilogue generation separately adds much value in this context, wdyt?Agree. I will only keep the
syscall_pro_epiloguetest.Quoted 29 lines
[...]
kernel/bpf/btf.c s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p) · line 609
Quoted 9 lines
kernel/bpf/btf.c +1
-0@@ -606,6 +606,7 @@ s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)spin_unlock_bh (&btf_idr_lock);return ret; } +EXPORT_SYMBOL_GPL (bpf_find_btf_id);I think this is not necessary, see below.
[...]
kernel/bpf/btf.c static void st_ops_unreg(void *kdata, struct bpf_link *link) · line 1499
Quoted 8 lines
kernel/bpf/btf.c +4
-0@@ -1410,6 +1493,13 @@ static void st_ops_unreg(void *kdata, struct bpf_link *link)static int st_ops_init (struct btf *btf) { +struct btf *kfunc_btf; + + bpf_cgroup_from_id_id =bpf_find_btf_id ("bpf_cgroup_from_id" ,BTF_KIND_FUNC , &kfunc_btf); + bpf_cgroup_release_id =bpf_find_btf_id ("bpf_cgroup_release" ,BTF_KIND_FUNC , &kfunc_btf);Maybe use
BTF_ID_LISTfor this?
E.g.BTF_ID_LIST(bpf_testmod_dtor_ids)in this file, or
BTF_ID_LIST(special_kfunc_list)inverifier.c?Quoted 5 lines
(Just in case, sorry if you know this already,
BTF_ID_LISTdeclares are set of symbols with special suffix/prefix, at build timetools/bpf/resolve_btfidslooks for such symbols and patches their values to correspond to BTF ids of specified functions and structures).Quoted 4 lines
Ah yes. It is an artifact when I was testing a patch for generating kfunc in module btf. But since there is no use case, I removed that part. I will change it to
BTF_ID_LIST. Thanks for catching this.Actually when I use
BTF_ID_LISTwith a kernel kfunc, I got the warning below. Since it was not able to resolve the btf id, the test program failed to load as the generated byte code will contain invalid kfunc id.Quoted 7 lines
BTF [M] bpf_testmod.ko
WARN:resolve_btfids: unresolved symbolbpf_cgroup_release
WARN:resolve_btfids: unresolved symbolbpf_cgroup_from_id
MOD bpf_testmod.koI am not familiar with how
resolve_btfidswork, specifically when building a kernel module. Do you have any suggestions?Looks like there is no way.
resolve_btfidsis called for module as follows:
resolve_btfids-b <path-to>/vmlinux bpf_testmod.ko
Quoted 8 lines
Where -b specifies base BTF.
However, the bpf_testmod.ko has .BTF.base section, where distilled BTF is stored. In such case tools/bpf/resolve_btfids/main.c:elf_collect() overrides base passed from command line, and uses .BTF.base instead. However,bpf_cgroup_releaseis not referenced inbpf_testmod.c, thus it does not get into distilled BTF. Hence, its id remains unresolved. And it cannot be referenced inbpf_testmod.c, because it is not an exported symbol.
I see. Thanks for the explanation!
Quoted 5 lines
So, for this particular case there are only two options: resolve id dynamically, as you did, or use a kfunc internal to this module instead (which should simplify the test, imo).
Currently, generating module kfunc in pro/epilogue is not supported due to the complication in resolving insn->off of the kfunc call. I have one patch that test how it can work. But after some discussion with Martin, we think we can revisit it when there is a struct_ops module that wants to do so. I will keep the current dynamic btf id resolution in the next respin over using module kfunc unless people think we should support it now.
Quoted 5 lines
The broader question if want a capability to use
BTF_ID_LISTreferring to kernel functions from modules remains open.[...]
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 2/2] selftests/bpf: Test gen_pro/epilogue that generate kfuncs
Quoted 1 line · Amery Hung
On Thu, Feb 20, 2025 at 7:02 PM Eduard Zingerman <[email protected]> wrote:
[...]
Quoted 6 lines
Quoted 5 lines
So, for this particular case there are only two options: resolve id dynamically, as you did, or use a kfunc internal to this module instead (which should simplify the test, imo).
Quoted 7 lines
Currently, generating module kfunc in pro/epilogue is not supported due to the complication in resolving
insn->offof the kfunc call. I have one patch that test how it can work. But after some discussion with Martin, we think we can revisit it when there is astruct_opsmodule that wants to do so. I will keep the current dynamic btf id resolution in the next respin over using module kfunc unless people think we should support it now.
Fair enough.
Sorry for the false trail with BTF_ID_LIST, this resolve_btfids corner case was a surprise for me.