Support BTF-powered BPF tracing programs for kernel modules
Subsystems (8)
Andrii Nakryiko
maintainerv2 cover Support BTF-powered BPF tracing programs for kernel modules
Building on top of two previous patch sets ([0] and not yet landed [1]), this patch sets extends kernel and libbpf with support for attaching BTF-powered raw tracepoint (tp_btf) and tracing (fentry/fexit/fmod_ret/lsm) BPF programs to BPF hooks defined in kernel modules.
Kernel UAPI for BPF_PROG_LOAD is extended with extra parameter (attach_btf_obj_id) which allows to specify kernel module BTF in which the BTF type is identifed by attach_btf_id.
From end user perspective there are no extra actions that need to happen. Libbpf will continue searching across all kernel module BTFs, if desired attach BTF type is not found in vmlinux. That way it doesn't matter if BPF hook that user is trying to attach to is built-in into vmlinux image or is loaded in kernel module.
Currently pahole doesn't generate BTF_KIND_FUNC info for ftrace-able static functions in kernel modules, so expose traced function in bpf_sidecar.ko. Once pahole is enhanced, we can go back to static function.
[0] https://patchwork.kernel.org/project/netdevbpf/list/?series=380759&state=*
[1] https://patchwork.kernel.org/project/netdevbpf/list/?series=393677&state=*
v1->v2:
- avoid increasing bpf_reg_state by reordering fields (Alexei);
- preserve btf_data_size in struct module;
- rebase on top of v3 of patch [1].
Andrii Nakryiko (8):
bpf: keep module's btf_data_size intact after load
bpf: remove hard-coded btf_vmlinux assumption from BPF verifier
bpf: allow to specify kernel module BTFs when attaching BPF programs
libbpf: factor out low-level BPF program loading helper
libbpf: support attachment of BPF tracing programs to kernel modules selftests/bpf: add tp_btf CO-RE reloc test for modules selftests/bpf: make bpf_testmod's traceable function global selftests/bpf: add fentry/fexit/fmod_ret selftest for kernel module
include/linux/bpf.h | 13 +- include/linux/bpf_verifier.h | 28 +++- include/linux/btf.h | 7 +- include/uapi/linux/bpf.h | 1 + kernel/bpf/btf.c | 90 +++++++---- kernel/bpf/syscall.c | 44 +++++- kernel/bpf/verifier.c | 77 ++++++---- kernel/module.c | 1 - net/ipv4/bpf_tcp_ca.c | 3 +- tools/include/uapi/linux/bpf.h | 1 + tools/lib/bpf/bpf.c | 101 ++++++++---- tools/lib/bpf/libbpf.c | 145 +++++++++++++----- tools/lib/bpf/libbpf_internal.h | 30 ++++ .../selftests/bpf/bpf_testmod/bpf_testmod.c | 3 +- .../selftests/bpf/prog_tests/core_reloc.c | 3 +- .../selftests/bpf/prog_tests/module_attach.c | 53 +++++++ .../bpf/progs/test_core_reloc_module.c | 32 +++- .../selftests/bpf/progs/test_module_attach.c | 66 ++++++++ 18 files changed, 546 insertions(+), 152 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/module_attach.c create mode 100644 tools/testing/selftests/bpf/progs/test_module_attach.c
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH v2 1/8 bpf: keep module's btf_data_size intact after load
Having real btf_data_size stored in struct module is benefitial to quickly determine which kernel modules have associated BTF object and which don't. There is no harm in keeping this info, as opposed to keeping invalid pointer.
--- kernel/module.c | 1 - 1 file changed, 1 deletion(-)
kernel/module.c +0 -1
@@ -3712,7 +3712,6 @@ static noinline int do_init_module(struct module *mod)#ifdef CONFIG_DEBUG_INFO_BTF_MODULES /* .BTF is not SHF_ALLOC and will get removed, so sanitize pointer */ mod->btf_data =NULL ; - mod->btf_data_size =0 ;#endif /* * We want to free module_init, but be aware that kallsyms may be
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH v2 2/8 bpf: remove hard-coded btf_vmlinux assumption from BPF verifier
In reply to Andrii NakryikoRemove a permeating assumption thoughout BPF verifier of vmlinux BTF. Instead, wherever BTF type IDs are involved, also track the instance of struct btf that goes along with the type ID. This allows to gradually add support for kernel module BTFs and using/tracking module types across BPF helper calls and registers.
This patch also renames btf_id() function to btf_obj_id() to minimize naming clash with using btf_id to denote BTF *type* ID, rather than BTF *object*'s ID.
Also, altough btf_vmlinux can't get destructed and thus doesn't need refcounting, module BTFs need that, so apply BTF refcounting universally when BPF program is using BTF-powered attachment (tp_btf, fentry/fexit, etc). This makes for simpler clean up code.
Now that BTF type ID is not enough to uniquely identify a BTF type, extend BPF trampoline key to include BTF object ID. To differentiate that from target program BPF ID, set 31st bit of type ID. BTF type IDs (at least currently) are not allowed to take full 32 bits, so there is no danger of confusing that bit with a valid BTF type ID.
--- include/linux/bpf.h | 13 ++++-- include/linux/bpf_verifier.h | 28 +++++++++---- include/linux/btf.h | 5 ++- kernel/bpf/btf.c | 65 ++++++++++++++++++++---------- kernel/bpf/syscall.c | 24 +++++++++-- kernel/bpf/verifier.c | 77 ++++++++++++++++++++++-------------- net/ipv4/bpf_tcp_ca.c | 3 +- 7 files changed, 148 insertions(+), 67 deletions(-)
include/linux/bpf.h +9 -4
@@ -421,7 +421,10 @@ struct bpf_insn_access_aux {enum bpf_reg_type reg_type;union {int ctx_field_size; -u32 btf_id; +struct { +struct btf *btf; +u32 btf_id; + }; };struct bpf_verifier_log *log;/* for verbose logs */ };
@@ -458,6 +461,7 @@ struct bpf_verifier_ops {struct bpf_insn *dst, struct bpf_prog *prog, u32 *target_size);int (*btf_struct_access)(struct bpf_verifier_log *log, +const struct btf *btf,const struct btf_type *t,int off,int size,enum bpf_access_type atype,u32 *next_btf_id);
@@ -771,6 +775,7 @@ struct bpf_prog_aux {u32 ctx_arg_info_size;u32 max_rdonly_access;u32 max_rdwr_access; +struct btf *attach_btf;const struct bpf_ctx_arg_aux *ctx_arg_info;struct mutex dst_mutex;/* protects dst_* pointers below, *after* prog becomes visible */ struct bpf_prog *dst_prog;
@@ -1005,7 +1010,6 @@ struct bpf_event_entry {bool bpf_prog_array_compatible (struct bpf_array *array,const struct bpf_prog *fp);int bpf_prog_calc_tag (struct bpf_prog *fp); -const char *kernel_type_name (u32 btf_type_id);const struct bpf_func_proto *bpf_get_trace_printk_proto (void );
@@ -1430,12 +1434,13 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog,bool btf_ctx_access (int off,int size,enum bpf_access_type type,const struct bpf_prog *prog,struct bpf_insn_access_aux *info); -int btf_struct_access (struct bpf_verifier_log *log, +int btf_struct_access (struct bpf_verifier_log *log,const struct btf *btf,const struct btf_type *t,int off,int size,enum bpf_access_type atype,u32 *next_btf_id);bool btf_struct_ids_match (struct bpf_verifier_log *log, -int off,u32 id,u32 need_type_id); +const struct btf *btf,u32 id,int off, +const struct btf *need_btf,u32 need_type_id); int btf_distill_func_proto(struct bpf_verifier_log *log, struct btf *btf,
include/linux/bpf_verifier.h +21 -7
@@ -5,6 +5,7 @@#define _LINUX_BPF_VERIFIER_H 1#include <linux/bpf.h> /* for enum bpf_reg_type */ +#include <linux/btf.h> /* for struct btf and btf_id() */ #include <linux/filter.h> /* for MAX_BPF_STACK */ #include <linux/tnum.h>
@@ -43,6 +44,8 @@ enum bpf_reg_liveness {struct bpf_reg_state {/* Ordering of fields matters. See states_equal() */ enum bpf_reg_type type; +/* Fixed part of pointer offset, pointer types only */ +s32 off;union {/* valid when type == PTR_TO_PACKET */ int range;
@@ -52,15 +55,20 @@ struct bpf_reg_state {*/ struct bpf_map *map_ptr; -u32 btf_id;/* for PTR_TO_BTF_ID */ +/* for PTR_TO_BTF_ID */ +struct { +struct btf *btf; +u32 btf_id; + };u32 mem_size;/* for PTR_TO_MEM | PTR_TO_MEM_OR_NULL */ /* Max size from any of the above. */ -unsigned long raw; +struct { +unsigned long raw1; +unsigned long raw2; + } raw; }; -/* Fixed part of pointer offset, pointer types only */ -s32 off;/* For PTR_TO_PACKET, used to find other pointers with the same variable * offset, so they can share range knowledge. * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we
@@ -311,7 +319,10 @@ struct bpf_insn_aux_data {struct {enum bpf_reg_type reg_type;/* type of pseudo_btf_id */ union { -u32 btf_id;/* btf_id for struct typed var */ +struct { +struct btf *btf; +u32 btf_id;/* btf_id for struct typed var */ + };u32 mem_size;/* mem_size for non-struct typed var */ }; } btf_var;
@@ -459,9 +470,12 @@ int check_ctx_reg(struct bpf_verifier_env *env,/* this lives here instead of in bpf.h because it needs to dereference tgt_prog */ static inline u64 bpf_trampoline_compute_key (const struct bpf_prog *tgt_prog, -u32 btf_id) +struct btf *btf,u32 btf_id) { -return tgt_prog ? (((u64 )tgt_prog->aux->id) <<32 | btf_id) : btf_id; +if (tgt_prog) +return ((u64 )tgt_prog->aux->id <<32 ) | btf_id; +else +return ((u64 )btf_obj_id (btf) <<32 ) |0x80000000 | btf_id; } int bpf_check_attach_target(struct bpf_verifier_log *log,
include/linux/btf.h +4 -1
@@ -18,6 +18,7 @@ struct btf_show;extern const struct file_operations btf_fops; +void btf_get (struct btf *btf);void btf_put (struct btf *btf);int btf_new_fd (const union bpf_attr *attr);struct btf *btf_get_by_fd (int fd);
@@ -88,7 +89,7 @@ int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj, char *buf, int len, u64 flags);int btf_get_fd_by_id (u32 id); -u32 btf_id (const struct btf *btf); +u32 btf_obj_id (const struct btf *btf);bool btf_member_is_reg_int (const struct btf *btf,const struct btf_type *s,const struct btf_member *m,u32 expected_offset,u32 expected_size);
@@ -206,6 +207,8 @@ static inline const struct btf_var_secinfo *btf_type_var_secinfo( }#ifdef CONFIG_BPF_SYSCALL +struct bpf_prog ; +const struct btf_type *btf_type_by_id (const struct btf *btf,u32 type_id);const char *btf_name_by_offset (const struct btf *btf,u32 offset);struct btf *btf_parse_vmlinux (void );
kernel/bpf/btf.c +44 -21
180 lines not on this page · raw message
kernel/bpf/syscall.c +21 -3
@@ -1726,6 +1726,8 @@ static void __bpf_prog_put_rcu(struct rcu_head *rcu)bpf_prog_uncharge_memlock (aux->prog);security_bpf_prog_free (aux);bpf_prog_free (aux->prog); +if (aux->attach_btf) +btf_put (aux->attach_btf); }static void __bpf_prog_put_noref (struct bpf_prog *prog,bool deferred)
@@ -2155,6 +2157,20 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) prog->expected_attach_type = attr->expected_attach_type; prog->aux->attach_btf_id = attr->attach_btf_id; + +if (attr->attach_btf_id && !attr->attach_prog_fd) { +struct btf *btf; + + btf =bpf_get_btf_vmlinux (); +if (IS_ERR (btf)) +return PTR_ERR (btf); +if (!btf) +return -EINVAL ; + +btf_get (btf); + prog->aux->attach_btf = btf; + } +if (attr->attach_prog_fd) {struct bpf_prog *dst_prog;
@@ -2255,6 +2271,8 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) free_prog_sec:security_bpf_prog_free (prog->aux); free_prog_nouncharge: +if (prog->aux->attach_btf) +btf_put (prog->aux->attach_btf);bpf_prog_free (prog);return err; }
@@ -2612,7 +2630,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, goto out_put_prog; } - key =bpf_trampoline_compute_key (tgt_prog, btf_id); + key =bpf_trampoline_compute_key (tgt_prog,NULL , btf_id); } link =kzalloc (sizeof (*link),GFP_USER );
@@ -3589,7 +3607,7 @@ static int bpf_prog_get_info_by_fd(struct file *file, }if (prog->aux->btf) - info.btf_id =btf_id (prog->aux->btf); + info.btf_id =btf_obj_id (prog->aux->btf); ulen = info.nr_func_info; info.nr_func_info = prog->aux->func_info_cnt;
@@ -3692,7 +3710,7 @@ static int bpf_map_get_info_by_fd(struct file *file,memcpy (info.name, map->name,sizeof (map->name));if (map->btf) { - info.btf_id =btf_id (map->btf); + info.btf_id =btf_obj_id (map->btf); info.btf_key_type_id = map->btf_key_type_id; info.btf_value_type_id = map->btf_value_type_id; }
kernel/bpf/verifier.c +47 -30
256 lines not on this page · raw message
net/ipv4/bpf_tcp_ca.c +2 -1
@@ -95,6 +95,7 @@ static bool bpf_tcp_ca_is_valid_access(int off, int size, }static int bpf_tcp_ca_btf_struct_access (struct bpf_verifier_log *log, +const struct btf *btf,const struct btf_type *t,int off,int size,enum bpf_access_type atype,u32 *next_btf_id)
@@ -102,7 +103,7 @@ static int bpf_tcp_ca_btf_struct_access(struct bpf_verifier_log *log,size_t end;if (atype ==BPF_READ ) -return btf_struct_access (log, t, off, size, atype, next_btf_id); +return btf_struct_access (log, btf, t, off, size, atype, next_btf_id);if (t != tcp_sock_type) {bpf_log (log,"only read is supported\n" );
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH v2 3/8 bpf: allow to specify kernel module BTFs when attaching BPF programs
In reply to Andrii NakryikoAdd ability for user-space programs to specify non-vmlinux BTF when attaching BTF-powered BPF programs: raw_tp, fentry/fexit/fmod_ret, LSM, etc. For this, add attach_btf_obj_id field which contains BTF object ID for either vmlinux or module. For backwards compatibility (and simplicity) reasons, 0 denotes vmlinux BTF. Only kernel BTF (vmlinux or module) can be specified.
--- include/linux/btf.h | 2 ++ include/uapi/linux/bpf.h | 1 + kernel/bpf/btf.c | 25 +++++++++++++++++++------ kernel/bpf/syscall.c | 22 +++++++++++++++++++--- tools/include/uapi/linux/bpf.h | 1 + 5 files changed, 42 insertions(+), 9 deletions(-)
include/linux/btf.h +2 -0
@@ -89,7 +89,9 @@ int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj, char *buf, int len, u64 flags);int btf_get_fd_by_id (u32 id); +struct btf *btf_get_by_id (int id);u32 btf_obj_id (const struct btf *btf); +bool btf_is_kernel (const struct btf *btf);bool btf_member_is_reg_int (const struct btf *btf,const struct btf_type *s,const struct btf_member *m,u32 expected_offset,u32 expected_size);
include/uapi/linux/bpf.h +1 -0
@@ -558,6 +558,7 @@ union bpf_attr {__u32 line_info_cnt;/* number of bpf_line_info records */ __u32 attach_btf_id;/* in-kernel BTF type id to attach to */ __u32 attach_prog_fd;/* 0 to attach to vmlinux */ +__u32 attach_btf_obj_id;/* vmlinux/module BTF object ID for BTF type */ };struct {/* anonymous struct used by BPF_OBJ_* commands */
kernel/bpf/btf.c +19 -6
@@ -5652,6 +5652,19 @@ struct btf *btf_get_by_fd(int fd)return btf; } +struct btf *btf_get_by_id (int id) +{ +struct btf *btf; + +rcu_read_lock (); + btf =idr_find (&btf_idr, id); +if (!btf || !refcount_inc_not_zero (&btf->refcnt)) + btf =ERR_PTR (-ENOENT ); +rcu_read_unlock (); + +return btf; +} +int btf_get_info_by_fd (const struct btf *btf,const union bpf_attr *attr,union bpf_attr __user *uattr)
@@ -5717,12 +5730,7 @@ int btf_get_fd_by_id(u32 id)struct btf *btf;int fd; -rcu_read_lock (); - btf =idr_find (&btf_idr, id); -if (!btf || !refcount_inc_not_zero (&btf->refcnt)) - btf =ERR_PTR (-ENOENT ); -rcu_read_unlock (); - + btf =btf_get_by_id (id);if (IS_ERR (btf))return PTR_ERR (btf);
@@ -5738,6 +5746,11 @@ u32 btf_obj_id(const struct btf *btf)return btf->id; } +bool btf_is_kernel (const struct btf *btf) +{ +return btf->kernel_btf; +} +static int btf_id_cmp_func (const void *a,const void *b) {const int *pa = a, *pb = b;
kernel/bpf/syscall.c +19 -3
@@ -1968,7 +1968,7 @@ static void bpf_prog_load_fixup_attach_type(union bpf_attr *attr)static int bpf_prog_load_check_attach (enum bpf_prog_type prog_type,enum bpf_attach_type expected_attach_type, -u32 btf_id,u32 prog_fd) +u32 btf_obj_id,u32 btf_id,u32 prog_fd) {if (btf_id) {if (btf_id >BTF_MAX_TYPE )
@@ -1985,6 +1985,9 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type, } } +if (btf_obj_id && (!btf_id || prog_fd)) +return -EINVAL ; +if (prog_fd && prog_type !=BPF_PROG_TYPE_TRACING && prog_type !=BPF_PROG_TYPE_EXT )return -EINVAL ;
@@ -2097,7 +2100,7 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type) }/* last field in 'union bpf_attr' used by this command */ -#define BPF_PROG_LOAD_LAST_FIELD attach_prog_fd +#define BPF_PROG_LOAD_LAST_FIELD attach_btf_obj_idstatic int bpf_prog_load (union bpf_attr *attr,union bpf_attr __user *uattr) {
@@ -2146,6 +2149,7 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr)bpf_prog_load_fixup_attach_type (attr);if (bpf_prog_load_check_attach (type, attr->expected_attach_type, + attr->attach_btf_obj_id, attr->attach_btf_id, attr->attach_prog_fd))return -EINVAL ;
@@ -2158,7 +2162,19 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) prog->expected_attach_type = attr->expected_attach_type; prog->aux->attach_btf_id = attr->attach_btf_id; -if (attr->attach_btf_id && !attr->attach_prog_fd) { +if (attr->attach_btf_obj_id) { +struct btf *btf; + + btf =btf_get_by_id (attr->attach_btf_obj_id); +if (IS_ERR (btf)) +return PTR_ERR (btf); +if (!btf_is_kernel (btf)) { +btf_put (btf); +return -EINVAL ; + } + + prog->aux->attach_btf = btf; + }else if (attr->attach_btf_id && !attr->attach_prog_fd) {struct btf *btf; btf =bpf_get_btf_vmlinux ();
tools/include/uapi/linux/bpf.h +1 -0
@@ -558,6 +558,7 @@ union bpf_attr {__u32 line_info_cnt;/* number of bpf_line_info records */ __u32 attach_btf_id;/* in-kernel BTF type id to attach to */ __u32 attach_prog_fd;/* 0 to attach to vmlinux */ +__u32 attach_btf_obj_id;/* vmlinux/module BTF object ID for BTF type */ };struct {/* anonymous struct used by BPF_OBJ_* commands */
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH v2 4/8 libbpf: factor out low-level BPF program loading helper
In reply to Andrii NakryikoRefactor low-level API for BPF program loading to not rely on public API types. This allows painless extension without constant efforts to cleverly not break backwards compatibility.
--- tools/lib/bpf/bpf.c | 100 ++++++++++++++++++++++---------- tools/lib/bpf/libbpf.c | 34 +++++------ tools/lib/bpf/libbpf_internal.h | 29 +++++++++ 3 files changed, 113 insertions(+), 50 deletions(-)
tools/lib/bpf/bpf.c +68 -32
@@ -214,59 +214,52 @@ alloc_zero_tailing_info(const void *orecord, __u32 cnt,return info; } -int bpf_load_program_xattr (const struct bpf_load_program_attr *load_attr, -char *log_buf,size_t log_buf_sz) +int libbpf__bpf_prog_load (const struct bpf_prog_load_params *load_attr) {void *finfo =NULL , *linfo =NULL ;union bpf_attr attr; -__u32 log_level;int fd; -if (!load_attr || !log_buf != !log_buf_sz) +if (!load_attr->log_buf != !load_attr->log_buf_sz)return -EINVAL ; - log_level = load_attr->log_level; -if (log_level > (4 |2 |1 ) || (log_level && !log_buf)) +if (load_attr->log_level > (4 |2 |1 ) || (load_attr->log_level && !load_attr->log_buf))return -EINVAL ;memset (&attr,0 ,sizeof (attr)); attr.prog_type = load_attr->prog_type; attr.expected_attach_type = load_attr->expected_attach_type; -if (attr.prog_type ==BPF_PROG_TYPE_STRUCT_OPS || - attr.prog_type ==BPF_PROG_TYPE_LSM ) { - attr.attach_btf_id = load_attr->attach_btf_id; - }else if (attr.prog_type ==BPF_PROG_TYPE_TRACING || - attr.prog_type ==BPF_PROG_TYPE_EXT ) { - attr.attach_btf_id = load_attr->attach_btf_id; - attr.attach_prog_fd = load_attr->attach_prog_fd; - }else { - attr.prog_ifindex = load_attr->prog_ifindex; - attr.kern_version = load_attr->kern_version; - } - attr.insn_cnt = (__u32 )load_attr->insns_cnt; + + attr.attach_btf_id = load_attr->attach_btf_id; + attr.attach_prog_fd = load_attr->attach_prog_fd; + + attr.prog_ifindex = load_attr->prog_ifindex; + attr.kern_version = load_attr->kern_version; + + attr.insn_cnt = (__u32 )load_attr->insn_cnt; attr.insns =ptr_to_u64 (load_attr->insns); attr.license =ptr_to_u64 (load_attr->license); - attr.log_level = log_level; -if (log_level) { - attr.log_buf =ptr_to_u64 (log_buf); - attr.log_size = log_buf_sz; - }else { - attr.log_buf =ptr_to_u64 (NULL ); - attr.log_size =0 ; + attr.log_level = load_attr->log_level; +if (attr.log_level) { + attr.log_buf =ptr_to_u64 (load_attr->log_buf); + attr.log_size = load_attr->log_buf_sz; } attr.prog_btf_fd = load_attr->prog_btf_fd; + attr.prog_flags = load_attr->prog_flags; + attr.func_info_rec_size = load_attr->func_info_rec_size; attr.func_info_cnt = load_attr->func_info_cnt; attr.func_info =ptr_to_u64 (load_attr->func_info); + attr.line_info_rec_size = load_attr->line_info_rec_size; attr.line_info_cnt = load_attr->line_info_cnt; attr.line_info =ptr_to_u64 (load_attr->line_info); +if (load_attr->name)memcpy (attr.prog_name, load_attr->name, -min (strlen (load_attr->name),BPF_OBJ_NAME_LEN -1 )); - attr.prog_flags = load_attr->prog_flags; +min (strlen (load_attr->name), (size_t )BPF_OBJ_NAME_LEN -1 )); fd =sys_bpf_prog_load (&attr,sizeof (attr));if (fd >=0 )
@@ -306,19 +299,19 @@ int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr, } fd =sys_bpf_prog_load (&attr,sizeof (attr)); -if (fd >=0 ) goto done; } -if (log_level || !log_buf) +if (load_attr->log_level || !load_attr->log_buf) goto done;/* Try again with log */ - attr.log_buf =ptr_to_u64 (log_buf); - attr.log_size = log_buf_sz; + attr.log_buf =ptr_to_u64 (load_attr->log_buf); + attr.log_size = load_attr->log_buf_sz; attr.log_level =1 ; - log_buf[0 ] =0 ; + load_attr->log_buf[0 ] =0 ; + fd =sys_bpf_prog_load (&attr,sizeof (attr)); done:free (finfo);
@@ -326,6 +319,49 @@ int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr,return fd; } +int bpf_load_program_xattr (const struct bpf_load_program_attr *load_attr, +char *log_buf,size_t log_buf_sz) +{ +struct bpf_prog_load_params p = {}; + +if (!load_attr || !log_buf != !log_buf_sz) +return -EINVAL ; + + p.prog_type = load_attr->prog_type; + p.expected_attach_type = load_attr->expected_attach_type; +switch (p.prog_type) { +case BPF_PROG_TYPE_STRUCT_OPS : +case BPF_PROG_TYPE_LSM : + p.attach_btf_id = load_attr->attach_btf_id; +break ; +case BPF_PROG_TYPE_TRACING : +case BPF_PROG_TYPE_EXT : + p.attach_btf_id = load_attr->attach_btf_id; + p.attach_prog_fd = load_attr->attach_prog_fd; +break ; +default : + p.prog_ifindex = load_attr->prog_ifindex; + p.kern_version = load_attr->kern_version; + } + p.insn_cnt = load_attr->insns_cnt; + p.insns = load_attr->insns; + p.license = load_attr->license; + p.log_level = load_attr->log_level; + p.log_buf = log_buf; + p.log_buf_sz = log_buf_sz; + p.prog_btf_fd = load_attr->prog_btf_fd; + p.func_info_rec_size = load_attr->func_info_rec_size; + p.func_info_cnt = load_attr->func_info_cnt; + p.func_info = load_attr->func_info; + p.line_info_rec_size = load_attr->line_info_rec_size; + p.line_info_cnt = load_attr->line_info_cnt; + p.line_info = load_attr->line_info; + p.name = load_attr->name; + p.prog_flags = load_attr->prog_flags; + +return libbpf__bpf_prog_load (&p); +} + int bpf_load_program(enum bpf_prog_type type, const struct bpf_insn *insns, size_t insns_cnt, const char *license, __u32 kern_version, char *log_buf,
tools/lib/bpf/libbpf.c +16 -18
@@ -6808,7 +6808,7 @@ static int load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt, char *license, __u32 kern_version, int *pfd) { - struct bpf_load_program_attr load_attr; + struct bpf_prog_load_params load_attr = {};char *cp, errmsg[STRERR_BUFSIZE ];size_t log_buf_size =0 ;char *log_buf =NULL ;
@@ -6817,7 +6817,6 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt,if (!insns || !insns_cnt)return -EINVAL ; -memset (&load_attr,0 ,sizeof (struct bpf_load_program_attr )); load_attr.prog_type = prog->type;/* old kernels might not support specifying expected_attach_type */ if (!kernel_supports(FEAT_EXP_ATTACH_TYPE) && prog->sec_def &&
@@ -6828,19 +6827,14 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt,if (kernel_supports (FEAT_PROG_NAME )) load_attr.name = prog->name; load_attr.insns = insns; - load_attr.insns_cnt = insns_cnt; + load_attr.insn_cnt = insns_cnt; load_attr.license = license; -if (prog->type ==BPF_PROG_TYPE_STRUCT_OPS || - prog->type ==BPF_PROG_TYPE_LSM ) { - load_attr.attach_btf_id = prog->attach_btf_id; - }else if (prog->type ==BPF_PROG_TYPE_TRACING || - prog->type ==BPF_PROG_TYPE_EXT ) { - load_attr.attach_prog_fd = prog->attach_prog_fd; - load_attr.attach_btf_id = prog->attach_btf_id; - }else { - load_attr.kern_version = kern_version; - load_attr.prog_ifindex = prog->prog_ifindex; - } + load_attr.attach_btf_id = prog->attach_btf_id; + load_attr.attach_prog_fd = prog->attach_prog_fd; + load_attr.attach_btf_id = prog->attach_btf_id; + load_attr.kern_version = kern_version; + load_attr.prog_ifindex = prog->prog_ifindex; +/* specify func_info/line_info only if kernel supports them */ btf_fd =bpf_object__btf_fd (prog->obj);if (btf_fd >=0 &&kernel_supports (FEAT_BTF_FUNC )) {
@@ -6864,7 +6858,9 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt, *log_buf =0 ; } - ret =bpf_load_program_xattr (&load_attr, log_buf, log_buf_size); + load_attr.log_buf = log_buf; + load_attr.log_buf_sz = log_buf_size; + ret =libbpf__bpf_prog_load (&load_attr);if (ret >=0 ) {if (log_buf && load_attr.log_level)
@@ -6905,9 +6901,9 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt,pr_warn ("-- BEGIN DUMP LOG ---\n" );pr_warn ("\n%s\n" , log_buf);pr_warn ("-- END LOG --\n" ); - }else if (load_attr.insns_cnt >=BPF_MAXINSNS ) { + }else if (load_attr.insn_cnt >=BPF_MAXINSNS ) {pr_warn ("Program too large (%zu insns), at most %d insns\n" , - load_attr.insns_cnt,BPF_MAXINSNS ); + load_attr.insn_cnt,BPF_MAXINSNS ); ret = -LIBBPF_ERRNO__PROG2BIG ; }else if (load_attr.prog_type !=BPF_PROG_TYPE_KPROBE ) {/* Wrong program type? */
@@ -6915,7 +6911,9 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt, load_attr.prog_type =BPF_PROG_TYPE_KPROBE ; load_attr.expected_attach_type =0 ; - fd =bpf_load_program_xattr (&load_attr,NULL ,0 ); + load_attr.log_buf =NULL ; + load_attr.log_buf_sz =0 ; + fd =libbpf__bpf_prog_load (&load_attr);if (fd >=0 ) {close (fd); ret = -LIBBPF_ERRNO__PROGTYPE ;
tools/lib/bpf/libbpf_internal.h +29 -0
@@ -151,6 +151,35 @@ int parse_cpu_mask_file(const char *fcpu, bool **mask, int *mask_sz);int libbpf__load_raw_btf (const char *raw_types,size_t types_len,const char *str_sec,size_t str_len); +struct bpf_prog_load_params { +enum bpf_prog_type prog_type; +enum bpf_attach_type expected_attach_type; +const char *name; +const struct bpf_insn *insns; +size_t insn_cnt; +const char *license; +__u32 kern_version; +__u32 attach_prog_fd; +__u32 attach_btf_id; +__u32 prog_ifindex; +__u32 prog_btf_fd; +__u32 prog_flags; + +__u32 func_info_rec_size; +const void *func_info; +__u32 func_info_cnt; + +__u32 line_info_rec_size; +const void *line_info; +__u32 line_info_cnt; + +__u32 log_level; +char *log_buf; +size_t log_buf_sz; +}; + +int libbpf__bpf_prog_load (const struct bpf_prog_load_params *load_attr); +int bpf_object__section_size (const struct bpf_object *obj,const char *name,__u32 *size); int bpf_object__variable_offset(const struct bpf_object *obj, const char *name,
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH v2 5/8 libbpf: support attachment of BPF tracing programs to kernel modules
In reply to Andrii NakryikoTeach libbpf to search for BTF types in kernel modules for tracing BPF programs. This allows attachment of raw_tp/fentry/fexit/fmod_ret/etc BPF program types to tracepoints and functions in kernel modules.
--- tools/lib/bpf/bpf.c | 1 + tools/lib/bpf/libbpf.c | 111 +++++++++++++++++++++++++------- tools/lib/bpf/libbpf_internal.h | 1 + 3 files changed, 90 insertions(+), 23 deletions(-)
tools/lib/bpf/bpf.c +1 -0
@@ -230,6 +230,7 @@ int libbpf__bpf_prog_load(const struct bpf_prog_load_params *load_attr) attr.prog_type = load_attr->prog_type; attr.expected_attach_type = load_attr->expected_attach_type; + attr.attach_btf_obj_id = load_attr->attach_btf_obj_id; attr.attach_btf_id = load_attr->attach_btf_id; attr.attach_prog_fd = load_attr->attach_prog_fd;
tools/lib/bpf/libbpf.c +88 -23
179 lines not on this page · raw message
tools/lib/bpf/libbpf_internal.h +1 -0
@@ -160,6 +160,7 @@ struct bpf_prog_load_params {const char *license;__u32 kern_version;__u32 attach_prog_fd; +__u32 attach_btf_obj_id;__u32 attach_btf_id;__u32 prog_ifindex;__u32 prog_btf_fd;
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH v2 6/8 selftests/bpf: add tp_btf CO-RE reloc test for modules
In reply to Andrii NakryikoAdd another CO-RE relocation test for kernel module relocations. This time for tp_btf with direct memory reads.
--- .../selftests/bpf/prog_tests/core_reloc.c | 3 +- .../bpf/progs/test_core_reloc_module.c | 32 ++++++++++++++++++- 2 files changed, 33 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/prog_tests/core_reloc.c +2 -1
@@ -514,7 +514,8 @@ static struct core_reloc_test_case test_cases[] = { },/* validate we can find kernel module BTF types for relocs/attach */ - MODULES_CASE("module", "raw_tp/bpf_testmod_test_read", "bpf_testmod_test_read"), +MODULES_CASE ("module_probed" ,"raw_tp/bpf_testmod_test_read" ,"bpf_testmod_test_read" ), + MODULES_CASE("module_direct", "tp_btf/bpf_testmod_test_read", NULL),/* validate BPF program can use multiple flavors to match against * single target BTF type
tools/testing/selftests/bpf/progs/test_core_reloc_module.c +31 -1
@@ -36,7 +36,7 @@ struct core_reloc_module_output { }; SEC("raw_tp/bpf_testmod_test_read") -int BPF_PROG (test_core_module , +int BPF_PROG (test_core_module_probed ,struct task_struct *task,struct bpf_testmod_test_read_ctx *read_ctx) {
@@ -64,3 +64,33 @@ int BPF_PROG(test_core_module,return 0 ; } + +SEC("tp_btf/bpf_testmod_test_read") +int BPF_PROG (test_core_module_direct , +struct task_struct *task, +struct bpf_testmod_test_read_ctx *read_ctx) +{ +struct core_reloc_module_output *out = (void *)&data.out; +__u64 pid_tgid =bpf_get_current_pid_tgid (); +__u32 real_tgid = (__u32)(pid_tgid >>32 ); +__u32 real_pid = (__u32 )pid_tgid; + +if (data.my_pid_tgid != pid_tgid) +return 0 ; + +if (task->pid != real_pid || task->tgid != real_tgid) +return 0 ; + + out->len = read_ctx->len; + out->off = read_ctx->off; + + out->read_ctx_sz = bpf_core_type_size(struct bpf_testmod_test_read_ctx); + out->read_ctx_exists = bpf_core_type_exists(struct bpf_testmod_test_read_ctx); + out->buf_exists =bpf_core_field_exists (read_ctx->buf); + out->off_exists =bpf_core_field_exists (read_ctx->off); + out->len_exists =bpf_core_field_exists (read_ctx->len); + + out->comm_len =BPF_CORE_READ_STR_INTO (&out->comm, task, comm); + +return 0 ; +}
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH v2 7/8 selftests/bpf: make bpf_testmod's traceable function global
In reply to Andrii NakryikoPahole currently isn't able to generate BTF for static functions in kernel modules, so make sure traced sidecar function is global.
--- tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +2 -1
@@ -10,7 +10,7 @@#define CREATE_TRACE_POINTS #include "bpf_testmod-events.h" -static noinline ssize_t +noinline ssize_tbpf_testmod_test_read (struct file *file,struct kobject *kobj,struct bin_attribute *bin_attr,char *buf,loff_t off,size_t len)
@@ -25,6 +25,7 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj,return -EIO ;/* always fail */ } +EXPORT_SYMBOL (bpf_testmod_test_read);ALLOW_ERROR_INJECTION (bpf_testmod_test_read,ERRNO );static struct bin_attribute bin_attr_bpf_testmod_file__ro_after_init = {
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH v2 8/8 selftests/bpf: add fentry/fexit/fmod_ret selftest for kernel module
In reply to Andrii NakryikoAdd new selftest checking attachment of fentry/fexit/fmod_ret (and raw tracepoint ones for completeness) BPF programs to kernel module function.
--- .../selftests/bpf/prog_tests/module_attach.c | 53 +++++++++++++++ .../selftests/bpf/progs/test_module_attach.c | 66 +++++++++++++++++++ 2 files changed, 119 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/module_attach.c create mode 100644 tools/testing/selftests/bpf/progs/test_module_attach.c
tools/testing/selftests/bpf/prog_tests/module_attach.c new file +53 -0
@@ -0,0 +1,53 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ + +#include <test_progs.h> +#include "test_module_attach.skel.h" + +static int duration; + +static int trigger_module_test_read (int read_sz) +{ +int fd, err; + + fd =open ("/sys/kernel/bpf_testmod" ,O_RDONLY ); + err = -errno; +if (CHECK (fd <0 ,"testmod_file_open" ,"failed: %d\n" , err)) +return err; + +read (fd,NULL , read_sz); +close (fd); + +return 0 ; +} + +void test_module_attach (void ) +{ +const int READ_SZ =456 ; +struct test_module_attach * skel; +struct test_module_attach__bss *bss; +int err; + + skel =test_module_attach__open_and_load (); +if (CHECK (!skel,"skel_open" ,"failed to open skeleton\n" )) +return ; + + bss = skel->bss; + + err =test_module_attach__attach (skel); +if (CHECK (err,"skel_attach" ,"skeleton attach failed: %d\n" , err)) + goto cleanup; + +/* trigger tracepoint */ +ASSERT_OK (trigger_module_test_read (READ_SZ ),"trigger_read" ); + +ASSERT_EQ (bss->raw_tp_read_sz,READ_SZ ,"raw_tp" ); +ASSERT_EQ (bss->tp_btf_read_sz,READ_SZ ,"tp_btf" ); +ASSERT_EQ (bss->fentry_read_sz,READ_SZ ,"fentry" ); +ASSERT_EQ (bss->fexit_read_sz,READ_SZ ,"fexit" ); +ASSERT_EQ (bss->fexit_ret, -EIO ,"fexit_tet" ); +ASSERT_EQ (bss->fmod_ret_read_sz,READ_SZ ,"fmod_ret" ); + +cleanup: +test_module_attach__destroy (skel); +}
tools/testing/selftests/bpf/progs/test_module_attach.c new file +66 -0
66 lines not on this page · raw message
Signature · 1 line
2.24.1
Quoted 9 lines · Andrii Nakryiko
Building on top of two previous patch sets ([0] and not yet landed [1]), this patch sets extends kernel and libbpf with support for attaching BTF-powered raw tracepoint (
tp_btf) and tracing (fentry/fexit/fmod_ret/lsm) BPF programs to BPF hooks defined in kernel modules.Kernel UAPI for
BPF_PROG_LOADis extended with extra parameter (attach_btf_obj_id) which allows to specify kernel module BTF in which the BTF type is identifed byattach_btf_id.
Quoted 8 lines
From end user perspective there are no extra actions that need to happen.
Libbpf will continue searching across all kernel module BTFs, if desired attach BTF type is not found in vmlinux. That way it doesn't matter if BPF hook that user is trying to attach to is built-in into vmlinux image or is loaded in kernel module.Currently pahole doesn't generate
BTF_KIND_FUNCinfo for ftrace-able static functions in kernel modules, so expose traced function in bpf_sidecar.ko. Once
bpf_sidecar.ko => bpf_testmod.ko
Quoted 42 lines
pahole is enhanced, we can go back to static function.
[0] https://patchwork.kernel.org/project/netdevbpf/list/?series=380759&state=* [1] https://patchwork.kernel.org/project/netdevbpf/list/?series=393677&state=*
v1->v2:
- avoid increasingbpf_reg_stateby reordering fields (Alexei);
- preservebtf_data_sizein struct module;
- rebase on top of v3 of patch [1].Andrii Nakryiko (8):
bpf: keep module'sbtf_data_sizeintact after load
bpf: remove hard-codedbtf_vmlinuxassumption from BPF verifier
bpf: allow to specify kernel module BTFs when attaching BPF programs
libbpf: factor out low-level BPF program loading helper
libbpf: support attachment of BPF tracing programs to kernel modules selftests/bpf: addtp_btfCO-RE reloc test for modules selftests/bpf: make bpf_testmod's traceable function global selftests/bpf: add fentry/fexit/fmod_ret selftest for kernel moduleinclude/linux/bpf.h | 13 +- include/linux/bpf_verifier.h | 28 +++- include/linux/btf.h | 7 +- include/uapi/linux/bpf.h | 1 + kernel/bpf/btf.c | 90 +++++++---- kernel/bpf/syscall.c | 44 +++++- kernel/bpf/verifier.c | 77 ++++++---- kernel/module.c | 1 - net/ipv4/bpf_tcp_ca.c | 3 +- tools/include/uapi/linux/bpf.h | 1 + tools/lib/bpf/bpf.c | 101 ++++++++---- tools/lib/bpf/libbpf.c | 145 +++++++++++++----- tools/lib/bpf/libbpf_internal.h | 30 ++++ .../selftests/bpf/bpf_testmod/bpf_testmod.c | 3 +- .../selftests/bpf/prog_tests/core_reloc.c | 3 +- .../selftests/bpf/prog_tests/module_attach.c | 53 +++++++ .../bpf/progs/test_core_reloc_module.c | 32 +++- .../selftests/bpf/progs/test_module_attach.c | 66 ++++++++ 18 files changed, 546 insertions(+), 152 deletions(-)create mode 100644
tools/testing/selftests/bpf/prog_tests/module_attach.ccreate mode 100644tools/testing/selftests/bpf/progs/test_module_attach.c
Quoted 19 lines · Yonghong Song
Quoted 15 lines · Andrii Nakryiko
Building on top of two previous patch sets ([0] and not yet landed [1]), this patch sets extends kernel and libbpf with support for attaching BTF-powered raw tracepoint (
tp_btf) and tracing (fentry/fexit/fmod_ret/lsm) BPF programs to BPF hooks defined in kernel modules.Kernel UAPI for
BPF_PROG_LOADis extended with extra parameter (attach_btf_obj_id) which allows to specify kernel module BTF in which the BTF type is identifed byattach_btf_id.From end user perspective there are no extra actions that need to happen.
Libbpf will continue searching across all kernel module BTFs, if desired attach BTF type is not found in vmlinux. That way it doesn't matter if BPF hook that user is trying to attach to is built-in into vmlinux image or is loaded in kernel module.
Quoted 5 lines
Quoted 2 lines
Currently pahole doesn't generate
BTF_KIND_FUNCinfo for ftrace-able static functions in kernel modules, so expose traced function in bpf_sidecar.ko. Oncebpf_sidecar.ko => bpf_testmod.ko
will fix, thanks!
Quoted 42 lines
Quoted 42 lines
pahole is enhanced, we can go back to static function.
[0] https://patchwork.kernel.org/project/netdevbpf/list/?series=380759&state=* [1] https://patchwork.kernel.org/project/netdevbpf/list/?series=393677&state=*
v1->v2:
- avoid increasingbpf_reg_stateby reordering fields (Alexei);
- preservebtf_data_sizein struct module;
- rebase on top of v3 of patch [1].Andrii Nakryiko (8):
bpf: keep module'sbtf_data_sizeintact after load
bpf: remove hard-codedbtf_vmlinuxassumption from BPF verifier
bpf: allow to specify kernel module BTFs when attaching BPF programs
libbpf: factor out low-level BPF program loading helper
libbpf: support attachment of BPF tracing programs to kernel modules selftests/bpf: addtp_btfCO-RE reloc test for modules selftests/bpf: make bpf_testmod's traceable function global selftests/bpf: add fentry/fexit/fmod_ret selftest for kernel moduleinclude/linux/bpf.h | 13 +- include/linux/bpf_verifier.h | 28 +++- include/linux/btf.h | 7 +- include/uapi/linux/bpf.h | 1 + kernel/bpf/btf.c | 90 +++++++---- kernel/bpf/syscall.c | 44 +++++- kernel/bpf/verifier.c | 77 ++++++---- kernel/module.c | 1 - net/ipv4/bpf_tcp_ca.c | 3 +- tools/include/uapi/linux/bpf.h | 1 + tools/lib/bpf/bpf.c | 101 ++++++++---- tools/lib/bpf/libbpf.c | 145 +++++++++++++----- tools/lib/bpf/libbpf_internal.h | 30 ++++ .../selftests/bpf/bpf_testmod/bpf_testmod.c | 3 +- .../selftests/bpf/prog_tests/core_reloc.c | 3 +- .../selftests/bpf/prog_tests/module_attach.c | 53 +++++++ .../bpf/progs/test_core_reloc_module.c | 32 +++- .../selftests/bpf/progs/test_module_attach.c | 66 ++++++++ 18 files changed, 546 insertions(+), 152 deletions(-)create mode 100644
tools/testing/selftests/bpf/prog_tests/module_attach.ccreate mode 100644tools/testing/selftests/bpf/progs/test_module_attach.c