Support kernel module function calls from eBPF
Subsystems (13)
Kumar Kartikeya Dwivedi
maintainerv2 cover Support kernel module function calls from eBPF
This set enables kernel module function calls, and also modifies verifier logic to permit invalid kernel function calls as long as they are pruned as part of dead code elimination. This is done to provide better runtime portability for BPF objects, which can conditionally disable parts of code that are pruned later by the verifier (e.g. const volatile vars, kconfig options). libbpf modifications are made along with kernel changes to support module function calls. The set includes gen_loader support for emitting kfunc relocations.
It also converts TCP congestion control objects to use the module kfunc support instead of relying on IS_BUILTIN ifdef.
Changelog:
----------
RFC v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Address comments from Alexei
* Reuse fd_array instead of introducing kfunc_btf_fds array
* Take btf and module reference as needed, instead of preloading
* Add BTF_KIND_FUNC relocation support to gen_loader infrastructure
* Address comments from Andrii
* Drop hashmap in libbpf for finding index of existing BTF in fd_array
* Preserve invalid kfunc calls only when the symbol is weak
* Adjust verifier selftests
Kumar Kartikeya Dwivedi (10):
bpf: Introduce BPF support for kernel module function calls
bpf: Be conservative while processing invalid kfunc calls
bpf: btf: Introduce helpers for dynamic BTF set registration
tools: Allow specifying base BTF file in resolve_btfids
bpf: Enable TCP congestion control kfunc from modules
bpf: Bump MAX_BPF_STACK size to 768 bytes
libbpf: Support kernel module function calls
libbpf: Resolve invalid weak kfunc calls with imm = 0, off = 0
libbpf: Update gen_loader to emit BTF_KIND_FUNC relocations bpf, selftests: Add basic test for module kfunc call
include/linux/bpf.h | 8 +- include/linux/bpf_verifier.h | 2 + include/linux/bpfptr.h | 1 + include/linux/btf.h | 38 ++++ include/linux/filter.h | 4 +- kernel/bpf/btf.c | 56 +++++ kernel/bpf/core.c | 2 + kernel/bpf/verifier.c | 200 ++++++++++++++++-- kernel/trace/bpf_trace.c | 1 + net/bpf/test_run.c | 2 +- net/ipv4/bpf_tcp_ca.c | 36 +--- net/ipv4/tcp_bbr.c | 28 ++- net/ipv4/tcp_cubic.c | 26 ++- net/ipv4/tcp_dctcp.c | 26 ++- scripts/Makefile.modfinal | 1 + tools/bpf/resolve_btfids/main.c | 19 +- tools/lib/bpf/bpf.c | 1 + tools/lib/bpf/bpf_gen_internal.h | 12 +- tools/lib/bpf/gen_loader.c | 93 +++++++- tools/lib/bpf/libbpf.c | 81 +++++-- tools/lib/bpf/libbpf_internal.h | 1 + tools/testing/selftests/bpf/Makefile | 1 + .../selftests/bpf/bpf_testmod/bpf_testmod.c | 23 +- .../selftests/bpf/prog_tests/ksyms_module.c | 13 +- .../bpf/prog_tests/ksyms_module_libbpf.c | 18 ++ .../selftests/bpf/progs/test_ksyms_module.c | 9 + .../bpf/progs/test_ksyms_module_libbpf.c | 35 +++ tools/testing/selftests/bpf/verifier/calls.c | 22 +- .../selftests/bpf/verifier/raw_stack.c | 4 +- .../selftests/bpf/verifier/stack_ptr.c | 6 +- .../testing/selftests/bpf/verifier/var_off.c | 4 +- 31 files changed, 661 insertions(+), 112 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/ksyms_module_libbpf.c create mode 100644 tools/testing/selftests/bpf/progs/test_ksyms_module_libbpf.c
Signature · 1 line
2.33.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 1/10 bpf: Introduce BPF support for kernel module function calls
This change adds support on the kernel side to allow for BPF programs to call kernel module functions. Userspace will prepare an array of module BTF fds that is passed in during BPF_PROG_LOAD using fd_array parameter. In the kernel, the module BTFs are placed in the auxilliary struct for bpf_prog, and loaded as needed.
The verifier then uses insn->off to index into the fd_array. insn->off is used by subtracting one from it, as userspace has to set the index of array in insn->off incremented by 1. This lets us denote vmlinux btf by insn->off == 0, and the module kfunc using insn->off > 0. They are sorted based on offset in an array, and each offset corresponds to one descriptor, with a max limit up to 256 such module BTFs.
Another change is to check_kfunc_call callback, which now include a struct module * pointer, this is to be used in later patch such that the kfunc_id and module pointer are matched for dynamically registered BTF sets from loadable modules, so that same kfunc_id in two modules doesn't lead to check_kfunc_call succeeding. For the duration of the check_kfunc_call, the reference to struct module exists, as it returns the pointer stored in kfunc_btf_tab.
--- include/linux/bpf.h | 8 +- include/linux/bpf_verifier.h | 2 + kernel/bpf/core.c | 2 + kernel/bpf/verifier.c | 182 +++++++++++++++++++++++++++++++---- net/bpf/test_run.c | 2 +- net/ipv4/bpf_tcp_ca.c | 2 +- 6 files changed, 173 insertions(+), 25 deletions(-)
include/linux/bpf.h +5 -3
@@ -511,7 +511,7 @@ struct bpf_verifier_ops { const struct btf_type *t, int off, int size, enum bpf_access_type atype, u32 *next_btf_id); -bool (*check_kfunc_call)(u32 kfunc_btf_id); +bool (*check_kfunc_call)(u32 kfunc_btf_id,struct module *owner); };struct bpf_prog_offload_ops {
@@ -874,6 +874,7 @@ struct bpf_prog_aux {void *jit_data;/* JIT specific data. arch dependent */ struct bpf_jit_poke_descriptor *poke_tab;struct bpf_kfunc_desc_tab *kfunc_tab; +struct bpf_kfunc_btf_tab *kfunc_btf_tab;u32 size_poke_tab;struct bpf_ksym ksym;const struct bpf_prog_ops *ops;
@@ -1635,7 +1636,7 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog,int bpf_prog_test_run_sk_lookup (struct bpf_prog *prog,const union bpf_attr *kattr,union bpf_attr __user *uattr); -bool bpf_prog_test_check_kfunc_call (u32 kfunc_id); +bool bpf_prog_test_check_kfunc_call (u32 kfunc_id,struct module *owner);bool btf_ctx_access (int off,int size,enum bpf_access_type type,const struct bpf_prog *prog,struct bpf_insn_access_aux *info);
@@ -1856,7 +1857,8 @@ static inline int bpf_prog_test_run_sk_lookup(struct bpf_prog *prog,return -ENOTSUPP ; } -static inline bool bpf_prog_test_check_kfunc_call (u32 kfunc_id) +static inline bool bpf_prog_test_check_kfunc_call (u32 kfunc_id, +struct module *owner) {return false; }
include/linux/bpf_verifier.h +2 -0
@@ -527,5 +527,7 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,const struct bpf_prog *tgt_prog, u32 btf_id, struct bpf_attach_target_info *tgt_info); +void bpf_free_kfunc_btf_tab (struct bpf_kfunc_btf_tab *tab); +#endif /* _LINUX_BPF_VERIFIER_H */
kernel/bpf/core.c +2 -0
@@ -32,6 +32,7 @@#include <linux/perf_event.h> #include <linux/extable.h> #include <linux/log2.h> +#include <linux/bpf_verifier.h> #include <asm/barrier.h> #include <asm/unaligned.h>
@@ -2255,6 +2256,7 @@ static void bpf_prog_free_deferred(struct work_struct *work)int i; aux = container_of(work, struct bpf_prog_aux, work); +bpf_free_kfunc_btf_tab (aux->kfunc_btf_tab);bpf_free_used_maps (aux);bpf_free_used_btfs (aux);if (bpf_prog_is_dev_bound (aux))
kernel/bpf/verifier.c +162 -20
321 lines not on this page · raw message
net/bpf/test_run.c +1 -1
@@ -241,7 +241,7 @@ BTF_ID(func, bpf_kfunc_call_test2) BTF_ID(func, bpf_kfunc_call_test3)BTF_SET_END (test_sk_kfunc_ids) -bool bpf_prog_test_check_kfunc_call(u32 kfunc_id) +bool bpf_prog_test_check_kfunc_call(u32 kfunc_id, struct module *owner) {return btf_id_set_contains (&test_sk_kfunc_ids, kfunc_id); }
net/ipv4/bpf_tcp_ca.c +1 -1
@@ -255,7 +255,7 @@ BTF_ID(func, bbr_set_state)#endif /* CONFIG_X86 */ BTF_SET_END (bpf_tcp_ca_kfunc_ids) -static bool bpf_tcp_ca_check_kfunc_call(u32 kfunc_btf_id) +static bool bpf_tcp_ca_check_kfunc_call(u32 kfunc_btf_id, struct module *owner) {return btf_id_set_contains (&bpf_tcp_ca_kfunc_ids, kfunc_btf_id); }
Signature · 1 line
2.33.0
kernel test robot
build botRe: [PATCH bpf-next v2 01/10] bpf: Introduce BPF support for kernel module function calls
Thank you for the patch! Yet something to improve:
Hi Kumar,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on bpf-next/master]
url: https://github.com/0day-ci/linux/commits/Kumar-Kartikeya-Dwivedi/Support-kernel-module-function-calls-from-eBPF/20210914-203919 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master config: nios2-defconfig (attached as .config) compiler: nios2-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/09b59d21436f8510c9d662bcf8b7815af98f3c1f
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Kumar-Kartikeya-Dwivedi/Support-kernel-module-function-calls-from-eBPF/20210914-203919 git checkout 09b59d21436f8510c9d662bcf8b7815af98f3c1f
# save the attached .config to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=nios2 SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
nios2-linux-ld: kernel/bpf/core.o: in function `bpf_prog_free_deferred':
>> core.c:(.text+0x294): undefined reference to `bpf_free_kfunc_btf_tab'
core.c:(.text+0x294): relocation truncated to fit: R_NIOS2_CALL26 against `bpf_free_kfunc_btf_tab'
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [PATCH bpf-next v2 01/10] bpf: Introduce BPF support for kernel module function calls
In reply to Kumar Kartikeya DwivediThank you for the patch! Yet something to improve:
Hi Kumar,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on bpf-next/master]
url: https://github.com/0day-ci/linux/commits/Kumar-Kartikeya-Dwivedi/Support-kernel-module-function-calls-from-eBPF/20210914-203919 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master config: nds32-defconfig (attached as .config) compiler: nds32le-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/09b59d21436f8510c9d662bcf8b7815af98f3c1f
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Kumar-Kartikeya-Dwivedi/Support-kernel-module-function-calls-from-eBPF/20210914-203919 git checkout 09b59d21436f8510c9d662bcf8b7815af98f3c1f
# save the attached .config to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=nds32 SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
nds32le-linux-ld: kernel/bpf/core.o: in function `bpf_prog_free_deferred':
core.c:(.text+0x2c8): undefined reference to `bpf_free_kfunc_btf_tab'
>> nds32le-linux-ld: core.c:(.text+0x2cc): undefined reference to `bpf_free_kfunc_btf_tab'
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Kumar Kartikeya Dwivedi
maintainerPATCH v2 2/10 bpf: Be conservative while processing invalid kfunc calls
In reply to Kumar Kartikeya DwivediThis patch also modifies the BPF verifier to only return error for invalid kfunc calls specially marked by userspace (with insn->imm == 0, insn->off == 0) after the verifier has eliminated dead instructions. This can be handled in the fixup stage, and skip processing during add and check stages.
If such an invalid call is dropped, the fixup stage will not encounter insn->imm as 0, otherwise it bails out and returns an error.
This will be exposed as weak ksym support in libbpf in subsequent patch.
--- kernel/bpf/verifier.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
kernel/bpf/verifier.c +18 -0
@@ -1815,6 +1815,15 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset) prog_aux->kfunc_tab = tab; } +/* btf idr allocates IDs from 1, so func_id == 0 is always invalid, but +* instead of returning an error, be conservative and wait until the +* code elimination pass before returning error, so that invalid calls +* that get pruned out can be in BPF programs loaded from userspace. +* It is also required that offset be untouched (0) for such calls. +*/ +if (!func_id && !offset) +return 0 ; +if (!btf_tab && offset) { btf_tab =kzalloc (sizeof (*btf_tab),GFP_KERNEL );if (!btf_tab)
@@ -6624,6 +6633,10 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn)struct btf *desc_btf;int err; +/* skip for now, but return error when we find this in fixup_kfunc_call */ +if (!insn->imm) +return 0 ; + desc_btf =find_kfunc_desc_btf (env, insn->imm, insn->off, &btf_mod);if (IS_ERR (desc_btf))return PTR_ERR (desc_btf);
@@ -12758,6 +12771,11 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, {const struct bpf_kfunc_desc *desc; +if (!insn->imm) { +verbose (env,"invalid kernel function call not eliminated in verifier pass\n" ); +return -EINVAL ; + } +/* insn->imm has the btf func_id. Replace it with * an address (relative to __bpf_base_call). */
Signature · 1 line
2.33.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 3/10 bpf: btf: Introduce helpers for dynamic BTF set registration
In reply to Kumar Kartikeya DwivediThis adds helpers for registering btf_id_set from modules and the check_kfunc_call callback that can be used to look them up.
With in kernel sets, the way this is supposed to work is, in kernel callback looks up within the in-kernel kfunc whitelist, and then defers to the dynamic BTF set lookup if it doesn't find the BTF id. If there is no in-kernel BTF id set, this callback can be used directly.
Also fix includes for btf.h and bpfptr.h so that they can included in isolation. This is in preparation for their usage in tcp_bbr, tcp_cubic and tcp_dctcp modules in the next patch.
--- include/linux/bpfptr.h | 1 + include/linux/btf.h | 32 ++++++++++++++++++++++++++ kernel/bpf/btf.c | 51 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 84 insertions(+)
include/linux/bpfptr.h +1 -0
@@ -3,6 +3,7 @@#ifndef _LINUX_BPFPTR_H#define _LINUX_BPFPTR_H +#include <linux/mm.h> #include <linux/sockptr.h> typedef sockptr_t bpfptr_t ;
include/linux/btf.h +32 -0
@@ -5,8 +5,10 @@#define _LINUX_BTF_H 1#include <linux/types.h> +#include <linux/bpfptr.h> #include <uapi/linux/btf.h> #include <uapi/linux/bpf.h> +#include <linux/mutex.h> #define BTF_TYPE_EMIT (type) ((void)(type *)0)#define BTF_TYPE_EMIT_ENUM (enum_val) ((void)enum_val)
@@ -238,4 +240,34 @@ static inline const char *btf_name_by_offset(const struct btf *btf, }#endif +struct kfunc_btf_id_set { +struct list_head list; +struct btf_id_set *set; +struct module *owner; +}; + +struct kfunc_btf_id_list ; + +#ifdef CONFIG_DEBUG_INFO_BTF_MODULES +void register_kfunc_btf_id_set (struct kfunc_btf_id_list *l, +struct kfunc_btf_id_set *s); +void unregister_kfunc_btf_id_set (struct kfunc_btf_id_list *l, +struct kfunc_btf_id_set *s); +#else +static inline void register_kfunc_btf_id_set (struct kfunc_btf_id_list *l, +struct kfunc_btf_id_set *s) +{ +} +static inline void unregister_kfunc_btf_id_set (struct kfunc_btf_id_list *l, +struct kfunc_btf_id_set *s) +{ +} +#endif + +#define DECLARE_CHECK_KFUNC_CALLBACK (type) \ + bool __bpf_check_##type##_kfunc_call(u32 kfunc_id, struct module *owner) +#define DEFINE_KFUNC_BTF_ID_SET (set, name) \ + struct kfunc_btf_id_set name = { LIST_HEAD_INIT(name.list), (set), \ + THIS_MODULE } +#endif
kernel/bpf/btf.c +51 -0
54 lines not on this page · raw message
Signature · 1 line
2.33.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 4/10 tools: Allow specifying base BTF file in resolve_btfids
In reply to Kumar Kartikeya DwivediThis commits allows specifying the base BTF for resolving btf id lists/sets during link time in the resolve_btfids tool. The base BTF is set to NULL if no path is passed. This allows resolving BTF ids for module kernel objects.
--- tools/bpf/resolve_btfids/main.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
tools/bpf/resolve_btfids/main.c +17 -2
@@ -89,6 +89,7 @@ struct btf_id {struct object {const char *path;const char *btf; +const char *base_btf_path;struct {int fd;
@@ -477,16 +478,27 @@ static int symbols_resolve(struct object *obj)int nr_structs = obj->nr_structs;int nr_unions = obj->nr_unions;int nr_funcs = obj->nr_funcs; +struct btf *base_btf =NULL ;int err, type_id;struct btf *btf;__u32 nr_types; - btf =btf__parse (obj->btf ?: obj->path,NULL ); +if (obj->base_btf_path) { + base_btf =btf__parse (obj->base_btf_path,NULL ); + err =libbpf_get_error (base_btf); +if (err) { +pr_err ("FAILED: load base BTF from %s: %s\n" , + obj->base_btf_path,strerror (-err)); +return -1 ; + } + } + + btf =btf__parse_split (obj->btf ?: obj->path, base_btf); err =libbpf_get_error (btf);if (err) {pr_err ("FAILED: load BTF from %s: %s\n" , obj->btf ?: obj->path,strerror (-err)); -return -1 ; + goto out; } err =-1 ;
@@ -545,6 +557,7 @@ static int symbols_resolve(struct object *obj) err =0 ; out: +btf__free (base_btf);btf__free (btf);return err; }
@@ -697,6 +710,8 @@ int main(int argc, const char **argv) "BTF data"),OPT_BOOLEAN (0 ,"no-fail" , &no_fail,"do not fail if " BTF_IDS_SECTION " section is not found" ), +OPT_STRING ('s' ,"base-btf" , &obj.base_btf_path,"file" , +"path of file providing base BTF data" ), OPT_END() };int err =-1 ;
Signature · 1 line
2.33.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 5/10 bpf: Enable TCP congestion control kfunc from modules
In reply to Kumar Kartikeya DwivediThis commit moves BTF ID lookup into the newly added registration helper, in a way that the bbr, cubic, and dctcp implementation set up their sets in the bpf_tcp_ca kfunc_btf_set list, while the ones not dependent on modules are looked up from the wrapper function.
This lifts the restriction for them to be compiled as built in objects, and can be loaded as modules if required. Also modify Makefile.modfinal to resolve_btfids in TCP congestion control modules if the config option is set, using the base BTF support added in the previous commit.
See following commits for background on use of:
CONFIG_X86 ifdef:
569c484f9995 (bpf: Limit static tcp-cc functions in the .BTF_ids list to x86)
CONFIG_DYNAMIC_FTRACE ifdef:
7aae231ac93b (bpf: tcp: Limit calling some tcp cc functions to CONFIG_DYNAMIC_FTRACE)
[ resolve_btfids uses --no-fail because some crypto kernel modules
under arch/x86/crypto generated from ASM do not have the .BTF sections ]
--- include/linux/btf.h | 4 ++++ kernel/bpf/btf.c | 3 +++ net/ipv4/bpf_tcp_ca.c | 34 +++------------------------------- net/ipv4/tcp_bbr.c | 28 +++++++++++++++++++++++++++- net/ipv4/tcp_cubic.c | 26 +++++++++++++++++++++++++- net/ipv4/tcp_dctcp.c | 26 +++++++++++++++++++++++++- scripts/Makefile.modfinal | 1 + 7 files changed, 88 insertions(+), 34 deletions(-)
include/linux/btf.h +4 -0
@@ -270,4 +270,8 @@ static inline void unregister_kfunc_btf_id_set(struct kfunc_btf_id_list *l,struct kfunc_btf_id_set name = {LIST_HEAD_INIT (name.list), (set), \ THIS_MODULE } +extern struct kfunc_btf_id_list bpf_tcp_ca_kfunc_list; + +DECLARE_CHECK_KFUNC_CALLBACK (bpf_tcp_ca); +#endif
kernel/bpf/btf.c +3 -0
@@ -6266,3 +6266,6 @@ EXPORT_SYMBOL_GPL(unregister_kfunc_btf_id_set);mutex_unlock (&list_name.mutex); \return false; \ } + +DEFINE_KFUNC_BTF_ID_LIST (bpf_tcp_ca_kfunc_list); +DEFINE_CHECK_KFUNC_CALLBACK (bpf_tcp_ca, bpf_tcp_ca_kfunc_list);
net/ipv4/bpf_tcp_ca.c +3 -31
@@ -223,41 +223,13 @@ BTF_ID(func, tcp_reno_cong_avoid) BTF_ID(func, tcp_reno_undo_cwnd) BTF_ID(func, tcp_slow_start) BTF_ID(func, tcp_cong_avoid_ai) -#ifdef CONFIG_X86 -#ifdef CONFIG_DYNAMIC_FTRACE -#if IS_BUILTIN (CONFIG_TCP_CONG_CUBIC ) -BTF_ID(func, cubictcp_init) -BTF_ID(func, cubictcp_recalc_ssthresh) -BTF_ID(func, cubictcp_cong_avoid) -BTF_ID(func, cubictcp_state) -BTF_ID(func, cubictcp_cwnd_event) -BTF_ID(func, cubictcp_acked) -#endif -#if IS_BUILTIN (CONFIG_TCP_CONG_DCTCP ) -BTF_ID(func, dctcp_init) -BTF_ID(func, dctcp_update_alpha) -BTF_ID(func, dctcp_cwnd_event) -BTF_ID(func, dctcp_ssthresh) -BTF_ID(func, dctcp_cwnd_undo) -BTF_ID(func, dctcp_state) -#endif -#if IS_BUILTIN (CONFIG_TCP_CONG_BBR ) -BTF_ID(func, bbr_init) -BTF_ID(func, bbr_main) -BTF_ID(func, bbr_sndbuf_expand) -BTF_ID(func, bbr_undo_cwnd) -BTF_ID(func, bbr_cwnd_event) -BTF_ID(func, bbr_ssthresh) -BTF_ID(func, bbr_min_tso_segs) -BTF_ID(func, bbr_set_state) -#endif -#endif /* CONFIG_DYNAMIC_FTRACE */ -#endif /* CONFIG_X86 */ BTF_SET_END (bpf_tcp_ca_kfunc_ids) static bool bpf_tcp_ca_check_kfunc_call(u32 kfunc_btf_id, struct module *owner) { -return btf_id_set_contains (&bpf_tcp_ca_kfunc_ids, kfunc_btf_id); +if (btf_id_set_contains (&bpf_tcp_ca_kfunc_ids, kfunc_btf_id)) +return true; +return __bpf_check_bpf_tcp_ca_kfunc_call (kfunc_btf_id, owner); }static const struct bpf_verifier_ops bpf_tcp_ca_verifier_ops = {
net/ipv4/tcp_bbr.c +27 -1
@@ -56,6 +56,8 @@* otherwise TCP stack falls back to an internal pacing using one high * resolution timer per TCP socket and may use more resources. */ +#include <linux/btf.h> +#include <linux/btf_ids.h> #include <linux/module.h> #include <net/tcp.h> #include <linux/inet_diag.h>
@@ -1152,14 +1154,38 @@ static struct tcp_congestion_ops tcp_bbr_cong_ops __read_mostly = { .set_state = bbr_set_state, }; +BTF_SET_START(tcp_bbr_kfunc_ids) +#ifdef CONFIG_X86 +#ifdef CONFIG_DYNAMIC_FTRACE +BTF_ID(func, bbr_init) +BTF_ID(func, bbr_main) +BTF_ID(func, bbr_sndbuf_expand) +BTF_ID(func, bbr_undo_cwnd) +BTF_ID(func, bbr_cwnd_event) +BTF_ID(func, bbr_ssthresh) +BTF_ID(func, bbr_min_tso_segs) +BTF_ID(func, bbr_set_state) +#endif +#endif +BTF_SET_END (tcp_bbr_kfunc_ids) + +static DEFINE_KFUNC_BTF_ID_SET(&tcp_bbr_kfunc_ids, tcp_bbr_kfunc_btf_set); +static int __init bbr_register (void ) { +int ret; +BUILD_BUG_ON (sizeof (struct bbr ) >ICSK_CA_PRIV_SIZE ); -return tcp_register_congestion_control (&tcp_bbr_cong_ops); + ret =tcp_register_congestion_control (&tcp_bbr_cong_ops); +if (ret) +return ret; +register_kfunc_btf_id_set (&bpf_tcp_ca_kfunc_list, &tcp_bbr_kfunc_btf_set); +return 0 ; } static void __exit bbr_unregister(void) { +unregister_kfunc_btf_id_set (&bpf_tcp_ca_kfunc_list, &tcp_bbr_kfunc_btf_set);tcp_unregister_congestion_control (&tcp_bbr_cong_ops); }
net/ipv4/tcp_cubic.c +25 -1
@@ -25,6 +25,8 @@*/ #include <linux/mm.h> +#include <linux/btf.h> +#include <linux/btf_ids.h> #include <linux/module.h> #include <linux/math64.h> #include <net/tcp.h>
@@ -482,8 +484,25 @@ static struct tcp_congestion_ops cubictcp __read_mostly = { .name = "cubic", }; +BTF_SET_START(tcp_cubic_kfunc_ids) +#ifdef CONFIG_X86 +#ifdef CONFIG_DYNAMIC_FTRACE +BTF_ID(func, cubictcp_init) +BTF_ID(func, cubictcp_recalc_ssthresh) +BTF_ID(func, cubictcp_cong_avoid) +BTF_ID(func, cubictcp_state) +BTF_ID(func, cubictcp_cwnd_event) +BTF_ID(func, cubictcp_acked) +#endif +#endif +BTF_SET_END (tcp_cubic_kfunc_ids) + +static DEFINE_KFUNC_BTF_ID_SET(&tcp_cubic_kfunc_ids, tcp_cubic_kfunc_btf_set); +static int __init cubictcp_register (void ) { +int ret; +BUILD_BUG_ON (sizeof (struct bictcp ) >ICSK_CA_PRIV_SIZE );/* Precompute a bunch of the scaling factors that are used per-packet
@@ -514,11 +533,16 @@ static int __init cubictcp_register(void)/* divide by bic_scale and by constant Srtt (100ms) */ do_div (cube_factor, bic_scale *10 ); -return tcp_register_congestion_control (&cubictcp); + ret =tcp_register_congestion_control (&cubictcp); +if (ret) +return ret; +register_kfunc_btf_id_set (&bpf_tcp_ca_kfunc_list, &tcp_cubic_kfunc_btf_set); +return 0 ; } static void __exit cubictcp_unregister(void) { +unregister_kfunc_btf_id_set (&bpf_tcp_ca_kfunc_list, &tcp_cubic_kfunc_btf_set);tcp_unregister_congestion_control (&cubictcp); }
net/ipv4/tcp_dctcp.c +25 -1
@@ -36,6 +36,8 @@* Glenn Judd <[email protected]> */ +#include <linux/btf.h> +#include <linux/btf_ids.h> #include <linux/module.h> #include <linux/mm.h> #include <net/tcp.h>
@@ -236,14 +238,36 @@ static struct tcp_congestion_ops dctcp_reno __read_mostly = { .name = "dctcp-reno", }; +BTF_SET_START(tcp_dctcp_kfunc_ids) +#ifdef CONFIG_X86 +#ifdef CONFIG_DYNAMIC_FTRACE +BTF_ID(func, dctcp_init) +BTF_ID(func, dctcp_update_alpha) +BTF_ID(func, dctcp_cwnd_event) +BTF_ID(func, dctcp_ssthresh) +BTF_ID(func, dctcp_cwnd_undo) +BTF_ID(func, dctcp_state) +#endif +#endif +BTF_SET_END (tcp_dctcp_kfunc_ids) + +static DEFINE_KFUNC_BTF_ID_SET(&tcp_dctcp_kfunc_ids, tcp_dctcp_kfunc_btf_set); +static int __init dctcp_register (void ) { +int ret; +BUILD_BUG_ON (sizeof (struct dctcp ) >ICSK_CA_PRIV_SIZE ); -return tcp_register_congestion_control (&dctcp); + ret =tcp_register_congestion_control (&dctcp); +if (ret) +return ret; +register_kfunc_btf_id_set (&bpf_tcp_ca_kfunc_list, &tcp_dctcp_kfunc_btf_set); +return 0 ; } static void __exit dctcp_unregister(void) { +unregister_kfunc_btf_id_set (&bpf_tcp_ca_kfunc_list, &tcp_dctcp_kfunc_btf_set);tcp_unregister_congestion_control (&dctcp); }
scripts/Makefile.modfinal +1 -0
@@ -41,6 +41,7 @@ quiet_cmd_btf_ko = BTF [M] $@ cmd_btf_ko = \ if [ -f vmlinux ]; then \ LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J --btf_base vmlinux $@; \ + $(RESOLVE_BTFIDS) --no-fail -s vmlinux $@; \ else \ printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \ fi;
Signature · 1 line
2.33.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 6/10 bpf: Bump MAX_BPF_STACK size to 768 bytes
In reply to Kumar Kartikeya DwivediIncrease the maximum stack size accessible to BPF program to 768 bytes. This is done so that gen_loader can use 94 additional fds for kfunc BTFs that it passes in to fd_array from the remaining space available for the loader_stack struct to expand.
--- include/linux/filter.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/filter.h +2 -2
@@ -82,8 +82,8 @@ struct ctl_table_header;*/ #define BPF_SYM_ELF_TYPE 't' -/* BPF program can access up to 512 bytes of stack space. */ -#define MAX_BPF_STACK 512 +/* BPF program can access up to 768 bytes of stack space. */ +#define MAX_BPF_STACK 768/* Helper macros for filter block array initializers. */
Signature · 1 line
2.33.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 7/10 libbpf: Support kernel module function calls
In reply to Kumar Kartikeya DwivediThis patch adds libbpf support for kernel module function call support. The fd_array parameter is used during BPF program load is used to pass module BTFs referenced by the program. insn->off is set to index into this array + 1, because insn->off as 0 is reserved for btf_vmlinux. The kernel subtracts 1 from insn->off when indexing into env->fd_array.
We try to use existing insn->off for a module, since the kernel limits the maximum distinct module BTFs for kfuncs to 256, and also because index must never exceed the maximum allowed value that can fit in insn->off (INT16_MAX). In the future, if kernel interprets signed offset as unsigned for kfunc calls, this limit can be increased to UINT16_MAX.
--- tools/lib/bpf/bpf.c | 1 + tools/lib/bpf/libbpf.c | 56 +++++++++++++++++++++++++++++++-- tools/lib/bpf/libbpf_internal.h | 1 + 3 files changed, 55 insertions(+), 3 deletions(-)
tools/lib/bpf/bpf.c +1 -0
@@ -264,6 +264,7 @@ int libbpf__bpf_prog_load(const struct bpf_prog_load_params *load_attr) attr.line_info_rec_size = load_attr->line_info_rec_size; attr.line_info_cnt = load_attr->line_info_cnt; attr.line_info =ptr_to_u64 (load_attr->line_info); + attr.fd_array =ptr_to_u64 (load_attr->fd_array);if (load_attr->name) memcpy(attr.prog_name, load_attr->name,
tools/lib/bpf/libbpf.c +53 -3
92 lines not on this page · raw message
tools/lib/bpf/libbpf_internal.h +1 -0
@@ -289,6 +289,7 @@ struct bpf_prog_load_params {__u32 log_level;char *log_buf;size_t log_buf_sz; +int *fd_array; };int libbpf__bpf_prog_load (const struct bpf_prog_load_params *load_attr);
Signature · 1 line
2.33.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 8/10 libbpf: Resolve invalid weak kfunc calls with imm = 0, off = 0
In reply to Kumar Kartikeya DwivediPreserve these calls as it allows verifier to succeed in loading the program if they are determined to be unreachable after dead code elimination during program load. If not, the verifier will fail at runtime. This is done for ext->is_weak symbols similar to the case for variable ksyms.
--- tools/lib/bpf/libbpf.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)
tools/lib/bpf/libbpf.c +10 -9
@@ -3409,11 +3409,6 @@ static int bpf_object__collect_externs(struct bpf_object *obj)return -ENOTSUP ; } }else if (strcmp (sec_name,KSYMS_SEC ) ==0 ) { -if (btf_is_func (t) && ext->is_weak) { -pr_warn ("extern weak function %s is unsupported\n" , - ext->name); -return -ENOTSUP ; - } ksym_sec = sec; ext->type =EXT_KSYM ; skip_mods_and_typedefs(obj->btf, t->type,
@@ -5342,8 +5337,12 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)case RELO_EXTERN_FUNC : ext = &obj->externs[relo->sym_off]; insn[0 ].src_reg =BPF_PSEUDO_KFUNC_CALL ; - insn[0 ].imm = ext->ksym.kernel_btf_id; - insn[0 ].off = ext->ksym.offset; +if (ext->is_set) { + insn[0 ].imm = ext->ksym.kernel_btf_id; + insn[0 ].off = ext->ksym.offset; + }else {/* unresolved weak kfunc */ + insn[0 ].imm = insn[0 ].off =0 ; + }break ;case RELO_SUBPROG_ADDR :if (insn[0 ].src_reg !=BPF_PSEUDO_FUNC ) {
@@ -6734,8 +6733,10 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj, kfunc_id =find_ksym_btf_id (obj, ext->name,BTF_KIND_FUNC , &kern_btf, &kern_btf_fd); -if (kfunc_id <0 ) { -pr_warn ("extern (func ksym) '%s': not found in kernel BTF\n" , +if (kfunc_id == -ESRCH && ext->is_weak) { +return 0 ; + }else if (kfunc_id <0 ) { +pr_warn ("extern (func ksym) '%s': not found in kernel or module BTFs\n" , ext->name);return kfunc_id; }
Signature · 1 line
2.33.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 9/10 libbpf: Update gen_loader to emit BTF_KIND_FUNC relocations
In reply to Kumar Kartikeya DwivediThis change updates the BPF syscall loader to relocate BTF_KIND_FUNC relocations, with support for weak kfunc relocations.
One of the disadvantages of gen_loader is that due to stack size limitation, BTF fd array size is clamped to a smaller limit than what the kernel allows. Also, finding an existing BTF fd's slot is not trivial, because that would require to open all module BTFs and match on the open fds (like we do for libbpf), so we do the next best thing:
deduplicate slots for the same symbol.
--- tools/lib/bpf/bpf_gen_internal.h | 12 ++++- tools/lib/bpf/gen_loader.c | 93 ++++++++++++++++++++++++++++---- tools/lib/bpf/libbpf.c | 8 +-- 3 files changed, 99 insertions(+), 14 deletions(-)
tools/lib/bpf/bpf_gen_internal.h +11 -1
@@ -7,8 +7,15 @@ struct ksym_relo_desc {const char *name;int kind;int insn_idx; +bool is_weak; }; +struct kfunc_desc { +const char *name; +int index; +}; + +#define MAX_KFUNC_DESCS 94struct bpf_gen {struct gen_loader_opts *opts;void *data_start;
@@ -24,6 +31,8 @@ struct bpf_gen {int relo_cnt;char attach_target[128 ];int attach_kind; +struct kfunc_desc kfunc_descs[MAX_KFUNC_DESCS ]; +__u32 nr_kfuncs; };void bpf_gen__init (struct bpf_gen *gen,int log_level);
@@ -36,6 +45,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, struct bpf_prog_load_params *load_avoid bpf_gen__map_update_elem (struct bpf_gen *gen,int map_idx,void *value,__u32 value_size);void bpf_gen__map_freeze (struct bpf_gen *gen,int map_idx);void bpf_gen__record_attach_target (struct bpf_gen *gen,const char *name,enum bpf_attach_type type); -void bpf_gen__record_extern (struct bpf_gen *gen,const char *name,int kind,int insn_idx); +void bpf_gen__record_extern (struct bpf_gen *gen,const char *name,bool is_weak,int kind, +int insn_idx);#endif
tools/lib/bpf/gen_loader.c +84 -9
142 lines not on this page · raw message
tools/lib/bpf/libbpf.c +4 -4
@@ -6240,12 +6240,12 @@ static int bpf_program__record_externs(struct bpf_program *prog) ext->name);return -ENOTSUP ; } -bpf_gen__record_extern (obj->gen_loader, ext->name,BTF_KIND_VAR , - relo->insn_idx); +bpf_gen__record_extern (obj->gen_loader, ext->name, ext->is_weak, +BTF_KIND_VAR , relo->insn_idx);break ;case RELO_EXTERN_FUNC : -bpf_gen__record_extern (obj->gen_loader, ext->name,BTF_KIND_FUNC , - relo->insn_idx); +bpf_gen__record_extern (obj->gen_loader, ext->name, ext->is_weak, +BTF_KIND_FUNC , relo->insn_idx);break ;default :continue ;
Signature · 1 line
2.33.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 10/10 bpf, selftests: Add basic test for module kfunc call
In reply to Kumar Kartikeya DwivediThis also tests support for invalid kfunc calls we added in prior changes, such that verifier handles invalid call as long as it is removed by code elimination pass (before fixup_kfunc_call). A separate test for libbpf is added, which tests failure in loading.
Also adjust verifier selftests which assume 512 byte stack to now assume 768 byte stack.
--- include/linux/btf.h | 2 ++ kernel/bpf/btf.c | 2 ++ kernel/trace/bpf_trace.c | 1 + tools/testing/selftests/bpf/Makefile | 1 + .../selftests/bpf/bpf_testmod/bpf_testmod.c | 23 +++++++++++- .../selftests/bpf/prog_tests/ksyms_module.c | 13 ++++--- .../bpf/prog_tests/ksyms_module_libbpf.c | 18 ++++++++++ .../selftests/bpf/progs/test_ksyms_module.c | 9 +++++ .../bpf/progs/test_ksyms_module_libbpf.c | 35 +++++++++++++++++++ tools/testing/selftests/bpf/verifier/calls.c | 22 ++++++------ .../selftests/bpf/verifier/raw_stack.c | 4 +-- .../selftests/bpf/verifier/stack_ptr.c | 6 ++-- .../testing/selftests/bpf/verifier/var_off.c | 4 +-- 13 files changed, 116 insertions(+), 24 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/ksyms_module_libbpf.c create mode 100644 tools/testing/selftests/bpf/progs/test_ksyms_module_libbpf.c
include/linux/btf.h +2 -0
@@ -271,7 +271,9 @@ static inline void unregister_kfunc_btf_id_set(struct kfunc_btf_id_list *l, THIS_MODULE } extern struct kfunc_btf_id_list bpf_tcp_ca_kfunc_list; +extern struct kfunc_btf_id_list raw_tp_kfunc_list;DECLARE_CHECK_KFUNC_CALLBACK (bpf_tcp_ca); +DECLARE_CHECK_KFUNC_CALLBACK (raw_tp);#endif
kernel/bpf/btf.c +2 -0
@@ -6269,3 +6269,5 @@ EXPORT_SYMBOL_GPL(unregister_kfunc_btf_id_set);DEFINE_KFUNC_BTF_ID_LIST (bpf_tcp_ca_kfunc_list);DEFINE_CHECK_KFUNC_CALLBACK (bpf_tcp_ca, bpf_tcp_ca_kfunc_list); +DEFINE_KFUNC_BTF_ID_LIST (raw_tp_kfunc_list); +DEFINE_CHECK_KFUNC_CALLBACK (raw_tp, raw_tp_kfunc_list);
kernel/trace/bpf_trace.c +1 -0
@@ -1629,6 +1629,7 @@ int __weak bpf_prog_test_run_tracing(struct bpf_prog *prog,const struct bpf_verifier_ops raw_tracepoint_verifier_ops = { .get_func_proto = raw_tp_prog_func_proto, .is_valid_access = raw_tp_prog_is_valid_access, + .check_kfunc_call = __bpf_check_raw_tp_kfunc_call, };const struct bpf_prog_ops raw_tracepoint_prog_ops = {
tools/testing/selftests/bpf/Makefile +1 -0
@@ -174,6 +174,7 @@ $(OUTPUT)/bpf_testmod.ko: $(VMLINUX_BTF) $(wildcard bpf_testmod/Makefile bpf_tes $(Q)$(RM) bpf_testmod/bpf_testmod.ko # force re-compilation $(Q)$(MAKE) $(submake_extras) -C bpf_testmod $(Q)cp bpf_testmod/bpf_testmod.ko $@ + $(Q)$(RESOLVE_BTFIDS) -s ../../../../vmlinux bpf_testmod.ko $(OUTPUT)/test_stub.o: test_stub.c $(BPFOBJ) $(call msg,CC,,$@)
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +22 -1
@@ -1,6 +1,8 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2020 Facebook */ #include <linux/error-injection.h> +#include <linux/btf.h> +#include <linux/btf_ids.h> #include <linux/init.h> #include <linux/module.h> #include <linux/percpu-defs.h>
@@ -13,6 +15,12 @@DEFINE_PER_CPU (int, bpf_testmod_ksym_percpu) =123 ; +noinline void +bpf_testmod_test_mod_kfunc (int i) +{ +pr_info ("mod kfunc i=%d\n" , i); +} + noinline int bpf_testmod_loop_test(int n) {int i, sum =0 ;
@@ -71,13 +79,26 @@ static struct bin_attribute bin_attr_bpf_testmod_file __ro_after_init = { .write = bpf_testmod_test_write, }; +BTF_SET_START(bpf_testmod_kfunc_ids) +BTF_ID(func, bpf_testmod_test_mod_kfunc) +BTF_SET_END(bpf_testmod_kfunc_ids) + +static DEFINE_KFUNC_BTF_ID_SET(&bpf_testmod_kfunc_ids, bpf_testmod_kfunc_btf_set); +static int bpf_testmod_init (void ) { -return sysfs_create_bin_file (kernel_kobj, &bin_attr_bpf_testmod_file); +int ret; + + ret =sysfs_create_bin_file (kernel_kobj, &bin_attr_bpf_testmod_file); +if (ret) +return ret; +register_kfunc_btf_id_set (&raw_tp_kfunc_list, &bpf_testmod_kfunc_btf_set); +return 0 ; }static void bpf_testmod_exit (void ) { +unregister_kfunc_btf_id_set (&raw_tp_kfunc_list, &bpf_testmod_kfunc_btf_set);return sysfs_remove_bin_file (kernel_kobj, &bin_attr_bpf_testmod_file); }
tools/testing/selftests/bpf/prog_tests/ksyms_module.c +8 -5
@@ -6,19 +6,22 @@#include <bpf/btf.h> #include "test_ksyms_module.lskel.h" -static int duration; -void test_ksyms_module (void ) { -struct test_ksyms_module * skel; +struct test_ksyms_module *skel;int err; +if (!env.has_testmod) { +test__skip (); +return ; + } + skel =test_ksyms_module__open_and_load (); -if (CHECK (!skel,"skel_open" ,"failed to open skeleton\n" )) +if (!ASSERT_OK_PTR (skel,"test_ksyms_module__open_and_load" ))return ; err =test_ksyms_module__attach (skel); -if (CHECK (err,"skel_attach" ,"skeleton attach failed: %d\n" , err)) +if (!ASSERT_OK (err,"test_ksyms_module__attach" )) goto cleanup;usleep (1 );
tools/testing/selftests/bpf/prog_tests/ksyms_module_libbpf.c new file +18 -0
@@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <test_progs.h> +#include "test_ksyms_module_libbpf.skel.h" + +void test_ksyms_module_libbpf (void ) +{ +struct test_ksyms_module_libbpf *skel; + +if (!env.has_testmod) { +test__skip (); +return ; + } + + skel =test_ksyms_module_libbpf__open_and_load (); +if (!ASSERT_EQ (skel,NULL ,"test_ksyms_module__open_and_load" )) +test_ksyms_module_libbpf__destroy (skel); +}
tools/testing/selftests/bpf/progs/test_ksyms_module.c +9 -0
@@ -6,8 +6,11 @@#include <bpf/bpf_helpers.h> extern const int bpf_testmod_ksym_percpu __ksym; +extern void bpf_testmod_test_mod_kfunc(int i) __ksym; +extern void bpf_testmod_invalid_mod_kfunc(void) __ksym __weak;int out_mod_ksym_global =0 ; +const volatile int x =0 ;bool triggered = false;SEC ("raw_tp/sys_enter" )
@@ -16,6 +19,12 @@ int handler(const void *ctx)int *val;__u32 cpu; +/* This should be preserved by clang, but DCE'd by verifier, and still +* allow loading the raw_tp prog +*/ +if (x) +bpf_testmod_invalid_mod_kfunc (); +bpf_testmod_test_mod_kfunc (42 ); val = (int *)bpf_this_cpu_ptr (&bpf_testmod_ksym_percpu); out_mod_ksym_global = *val; triggered = true;
tools/testing/selftests/bpf/progs/test_ksyms_module_libbpf.c new file +35 -0
@@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "vmlinux.h" + +#include <bpf/bpf_helpers.h> + +extern void bpf_testmod_test_mod_kfunc(int i) __ksym; +extern void bpf_testmod_invalid_mod_kfunc(void) __ksym __weak; + +const volatile int x =0 ; + +SEC("raw_tp/sys_enter") +int handler_pass (const void *ctx) +{ +/* This should be preserved by clang, but DCE'd by verifier, and still +* allow loading the raw_tp prog +*/ +if (x) +bpf_testmod_invalid_mod_kfunc (); +bpf_testmod_test_mod_kfunc (42 ); +return 0 ; +} + +SEC("raw_tp/sys_enter") +int handler_fail (const void *ctx) +{ +/* This call should be preserved by clang, but fail verification. +*/ +if (!x) +bpf_testmod_invalid_mod_kfunc (); +bpf_testmod_test_mod_kfunc (42 ); +return 0 ; +} + +char LICENSE[] SEC("license") = "GPL";
tools/testing/selftests/bpf/verifier/calls.c +11 -11
@@ -750,12 +750,12 @@ "calls: stack overflow using two frames (pre-call access)", .insns = {/* prog 1 */ -BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-300 ,0 ), +BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-400 ,0 ),BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ),BPF_EXIT_INSN (),/* prog 2 */ -BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-300 ,0 ), +BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-400 ,0 ),BPF_MOV64_IMM (BPF_REG_0 ,0 ), BPF_EXIT_INSN(), },
@@ -768,11 +768,11 @@ .insns = {/* prog 1 */ BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), -BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-300 ,0 ), +BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-400 ,0 ),BPF_EXIT_INSN (),/* prog 2 */ -BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-300 ,0 ), +BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-400 ,0 ),BPF_MOV64_IMM (BPF_REG_0 ,0 ), BPF_EXIT_INSN(), },
@@ -846,12 +846,12 @@/* B */ BPF_JMP_IMM (BPF_JGT ,BPF_REG_1 ,2 ,1 ),BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,-6 ),/* call A */ -BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-256 ,0 ), +BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-512 ,0 ), BPF_EXIT_INSN(), }, .prog_type =BPF_PROG_TYPE_XDP , -/* stack_main=64, stack_A=224, stack_B=256 -* and max(main+A, main+A+B) > 512 +/* stack_main=64, stack_A=224, stack_B=512 +* and max(main+A, main+A+B) > 768 */ .errstr = "combined stack", .result = REJECT,
@@ -865,14 +865,14 @@* } * void func1(int alloc_or_recurse) { * if (alloc_or_recurse) { -* frame_pointer[-300] = 1; +* frame_pointer[-400] = 1; * } else { * func2(alloc_or_recurse); * } * } * void func2(int alloc_or_recurse) { * if (alloc_or_recurse) { -* frame_pointer[-300] = 1; +* frame_pointer[-400] = 1; * } * } */
@@ -888,13 +888,13 @@BPF_EXIT_INSN (),/* A */ BPF_JMP_IMM (BPF_JEQ ,BPF_REG_1 ,0 ,2 ), -BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-300 ,0 ), +BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-400 ,0 ),BPF_EXIT_INSN (),BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ),/* call B */ BPF_EXIT_INSN (),/* B */ BPF_JMP_IMM (BPF_JEQ ,BPF_REG_1 ,0 ,1 ), -BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-300 ,0 ), +BPF_ST_MEM (BPF_B ,BPF_REG_10 ,-400 ,0 ), BPF_EXIT_INSN(), }, .prog_type = BPF_PROG_TYPE_XDP,
tools/testing/selftests/bpf/verifier/raw_stack.c +2 -2
@@ -197,7 +197,7 @@ .insns = {BPF_MOV64_IMM (BPF_REG_2 ,4 ),BPF_ALU64_REG (BPF_MOV ,BPF_REG_6 ,BPF_REG_10 ), -BPF_ALU64_IMM (BPF_ADD ,BPF_REG_6 ,-513 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_6 ,-769 ),BPF_MOV64_REG (BPF_REG_3 ,BPF_REG_6 ),BPF_MOV64_IMM (BPF_REG_4 ,8 ), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_skb_load_bytes),
@@ -205,7 +205,7 @@ BPF_EXIT_INSN(), }, .result = REJECT, - .errstr = "invalid indirect access to stack R3 off=-513 size=8", + .errstr = "invalid indirect access to stack R3 off=-769 size=8", .prog_type = BPF_PROG_TYPE_SCHED_CLS, }, {
tools/testing/selftests/bpf/verifier/stack_ptr.c +3 -3
@@ -165,7 +165,7 @@ "PTR_TO_STACK check low 2", .insns = {BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), -BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-513 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-769 ),BPF_ST_MEM (BPF_B ,BPF_REG_1 ,1 ,42 ),BPF_LDX_MEM (BPF_B ,BPF_REG_0 ,BPF_REG_1 ,1 ), BPF_EXIT_INSN(),
@@ -179,13 +179,13 @@ "PTR_TO_STACK check low 3", .insns = {BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), -BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-513 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-769 ),BPF_ST_MEM (BPF_B ,BPF_REG_1 ,0 ,42 ),BPF_LDX_MEM (BPF_B ,BPF_REG_0 ,BPF_REG_1 ,0 ), BPF_EXIT_INSN(), }, .errstr_unpriv ="R1 stack pointer arithmetic goes out of range" , - .errstr = "invalid write to stack R1 off=-513 size=1", + .errstr = "invalid write to stack R1 off=-769 size=1", .result = REJECT, }, {
tools/testing/selftests/bpf/verifier/var_off.c +2 -2
@@ -196,8 +196,8 @@BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 ,0 ),/* Make it small and 4-byte aligned */ BPF_ALU64_IMM (BPF_AND ,BPF_REG_2 ,4 ), -BPF_ALU64_IMM (BPF_SUB ,BPF_REG_2 ,516 ), -/* add it to fp. We now have either fp-516 or fp-512, but +BPF_ALU64_IMM (BPF_SUB ,BPF_REG_2 ,772 ), +/* add it to fp. We now have either fp-772 or fp-768, but * we don't know which */ BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_10),
Signature · 1 line
2.33.0