bpf: implement bpf iterator for kernel data
Subsystems (14)
v3 cover bpf: implement bpf iterator for kernel data
Motivation:
The current way to dump kernel data structures mostly:
1. /proc system
2. various specific tools like "ss" which requires kernel support.
3. drgn
The dropback for the first two is that whenever you want to dump more, you need change the kernel. For example, Martin wants to dump socket local storage with "ss". Kernel change is needed for it to work ([1]). This is also the direct motivation for this work.
drgn ([2]) solves this proble nicely and no kernel change is not needed. But since drgn is not able to verify the validity of a particular pointer value, it might present the wrong results in rare cases.
In this patch set, we introduce bpf iterator. Initial kernel changes are still needed for interested kernel data, but a later data structure change will not require kernel changes any more. bpf program itself can adapt to new data structure changes. This will give certain flexibility with guaranteed correctness.
In this patch set, kernel seq_ops is used to facilitate iterating through kernel data, similar to current /proc and many other lossless kernel dumping facilities. In the future, different iterators can be implemented to trade off losslessness for other criteria e.g. no repeated object visits, etc.
User Interface:
1. Similar to prog/map/link, the iterator can be pinned into a path within a bpffs mount point.
2. The bpftool command can pin an iterator to a file bpftool iter pin <bpf_prog.o> <path>
3. Use cat <path> to dump the contents.
Use rm -f <path> to remove the pinned iterator.
4. The anonymous iterator can be created as well.
Please see patch #19 andd #20 for bpf programs and bpf iterator output examples.
Note that certain iterators are namespace aware. For example, task and task_file targets only iterate through current pid namespace. ipv6_route and netlink will iterate through current net namespace.
Please see individual patches for implementation details.
Performance:
The bpf iterator provides in-kernel aggregation abilities
for kernel data. This can greatly improve performance
compared to e.g., iterating all process directories under /proc.
For example, I did an experiment on my VM with an application forking different number of tasks and each forked process opening various number of files. The following is the result with the latency with unit of microseconds:
# of forked tasks # of open files # of bpf_prog calls # latency (us)
100 100 11503 7586
1000 1000 1013203 709513
10000 100 1130203 764519 The number of bpf_prog calls may be more than forked tasks multipled by open files since there are other tasks running on the system. The bpf program is a do-nothing program. One millions of bpf calls takes less than one second.
Future Work:
Although the initial motivation is from Martin's sk_local_storage, this patch didn't implement tcp6 sockets and sk_local_storage. The /proc/net/tcp6 involves three types of sockets, timewait, request and tcp6 sockets. Some kind of type casting or other mechanism is needed to handle all these socket types in one bpf program. This will be addressed in future work.
Currently, we do not support kernel data generated under module. This requires some BTF work.
More work for more iterators, e.g., tcp, udp, bpf_map elements, etc.
Changelog:
v2 -> v3:
- add bpf_iter_unreg_target() to unregister a target, used in the error path of the __init functions.
- handle err != 0 before handling overflow (Andrii)
- reference count "task" for task_file target (Andrii)
- remove some redundancy for bpf_map/task/task_file targets
- add bpf_iter_unreg_target() in ip6_route_cleanup()
- Handling "%%" format in bpf_seq_printf() (Andrii)
- implement auto-attach for bpf_iter in libbpf (Andrii)
- add macros offsetof and container_of in bpf_helpers.h (Andrii)
- add tests for auto-attach and program-return-1 cases
- some other minor fixes
v1 -> v2:
- removed target_feature, using callback functions instead
- checking target to ensure program specified btf_id supported (Martin)
- link_create change with new changes from Andrii
- better handling of btf_iter vs. seq_file private data (Martin, Andrii)
- implemented bpf_seq_read() (Andrii, Alexei)
- percpu buffer for bpf_seq_printf() (Andrii)
- better syntax for BPF_SEQ_PRINTF macro (Andrii)
- bpftool fixes (Quentin)
- a lot of other fixes
RFC v2 -> v1:
- rename bpfdump to bpf_iter
- use bpffs instead of a new file system
- use bpf_link to streamline and simplify iterator creation.
References:
[1]: https://lore.kernel.org/bpf/[email protected]
[2]: https://github.com/osandov/drgn
Yonghong Song (21):
bpf: implement an interface to register bpf_iter targets
bpf: allow loading of a bpf_iter program
bpf: support bpf tracing/iter programs for BPF_LINK_CREATE
bpf: support bpf tracing/iter programs for BPF_LINK_UPDATE
bpf: implement bpf_seq_read() for bpf iterator
bpf: create anonymous bpf iterator
bpf: create file bpf iterator
bpf: implement common macros/helpers for target iterators
bpf: add bpf_map iterator
net: bpf: add netlink and ipv6_route bpf_iter targets
bpf: add task and task/file iterator targets
bpf: add PTR_TO_BTF_ID_OR_NULL support
bpf: add bpf_seq_printf and bpf_seq_write helpers
bpf: handle spilled PTR_TO_BTF_ID properly when checking
stack_boundary
bpf: support variable length array in tracing programs tools/libbpf: add bpf_iter support
tools/libpf: add offsetof/container_of macro in bpf_helpers.h tools/bpftool: add bpf_iter support for bptool
tools/bpf: selftests: add iterator programs for ipv6_route and netlink tools/bpf: selftests: add iter progs for bpf_map/task/task_file tools/bpf: selftests: add bpf_iter selftests
fs/proc/proc_net.c | 19 + include/linux/bpf.h | 36 ++ include/linux/bpf_types.h | 1 + include/linux/proc_fs.h | 3 + include/uapi/linux/bpf.h | 40 +- kernel/bpf/Makefile | 2 +- kernel/bpf/bpf_iter.c | 526 ++++++++++++++++++ kernel/bpf/btf.c | 42 +- kernel/bpf/inode.c | 5 +- kernel/bpf/map_iter.c | 97 ++++ kernel/bpf/syscall.c | 59 ++ kernel/bpf/task_iter.c | 332 +++++++++++ kernel/bpf/verifier.c | 42 +- kernel/trace/bpf_trace.c | 200 +++++++ net/ipv6/ip6_fib.c | 65 ++- net/ipv6/route.c | 37 ++ net/netlink/af_netlink.c | 87 ++- scripts/bpf_helpers_doc.py | 2 + .../bpftool/Documentation/bpftool-iter.rst | 83 +++ tools/bpf/bpftool/bash-completion/bpftool | 13 + tools/bpf/bpftool/iter.c | 84 +++ tools/bpf/bpftool/link.c | 1 + tools/bpf/bpftool/main.c | 3 +- tools/bpf/bpftool/main.h | 1 + tools/include/uapi/linux/bpf.h | 40 +- tools/lib/bpf/bpf.c | 10 + tools/lib/bpf/bpf.h | 2 + tools/lib/bpf/bpf_helpers.h | 14 + tools/lib/bpf/bpf_tracing.h | 16 + tools/lib/bpf/libbpf.c | 52 ++ tools/lib/bpf/libbpf.h | 9 + tools/lib/bpf/libbpf.map | 2 + .../selftests/bpf/prog_tests/bpf_iter.c | 408 ++++++++++++++ .../selftests/bpf/progs/bpf_iter_bpf_map.c | 28 + .../selftests/bpf/progs/bpf_iter_ipv6_route.c | 62 +++ .../selftests/bpf/progs/bpf_iter_netlink.c | 66 +++ .../selftests/bpf/progs/bpf_iter_task.c | 25 + .../selftests/bpf/progs/bpf_iter_task_file.c | 26 + .../selftests/bpf/progs/bpf_iter_test_kern1.c | 4 + .../selftests/bpf/progs/bpf_iter_test_kern2.c | 4 + .../selftests/bpf/progs/bpf_iter_test_kern3.c | 18 + .../selftests/bpf/progs/bpf_iter_test_kern4.c | 52 ++ .../bpf/progs/bpf_iter_test_kern_common.h | 22 + 43 files changed, 2626 insertions(+), 14 deletions(-) create mode 100644 kernel/bpf/bpf_iter.c create mode 100644 kernel/bpf/map_iter.c create mode 100644 kernel/bpf/task_iter.c create mode 100644 tools/bpf/bpftool/Documentation/bpftool-iter.rst create mode 100644 tools/bpf/bpftool/iter.c create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_iter.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_bpf_map.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_ipv6_route.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_netlink.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_task.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_task_file.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern1.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern2.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern3.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern4.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern_common.h
Signature · 1 line
2.24.1
PATCH v3 1/21 bpf: implement an interface to register bpf_iter targets
The target can call bpf_iter_reg_target() to register itself. The needed information:
target: target name seq_ops: the seq_file operations for the target init_seq_private target callback to initialize seq_priv during file open fini_seq_private target callback to clean up seq_priv during file release seq_priv_size: the private_data size needed by the seq_file
operations
The target name represents a target which provides a seq_ops for iterating objects.
The target can provide two callback functions, init_seq_private and fini_seq_private, called during file open/release time. For example, /proc/net/{tcp6, ipv6_route, netlink, ...}, net name space needs to be setup properly during file open and released properly during file release.
Function bpf_iter_unreg_target() is also implemented to unregister a particular target.
--- include/linux/bpf.h | 15 +++++++++++ kernel/bpf/Makefile | 2 +- kernel/bpf/bpf_iter.c | 59 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 75 insertions(+), 1 deletion(-) create mode 100644 kernel/bpf/bpf_iter.c
include/linux/bpf.h +15 -0
@@ -31,6 +31,7 @@ struct seq_file;struct btf ;struct btf_type ;struct exception_table_entry ; +struct seq_operations ;extern struct idr btf_idr;extern spinlock_t btf_idr_lock;
@@ -1126,6 +1127,20 @@ struct bpf_link *bpf_link_get_from_fd(u32 ufd);int bpf_obj_pin_user (u32 ufd,const char __user *pathname);int bpf_obj_get_user (const char __user *pathname,int flags); +typedef int (*bpf_iter_init_seq_priv_t )(void *private_data); +typedef void (*bpf_iter_fini_seq_priv_t )(void *private_data); + +struct bpf_iter_reg { +const char *target; +const struct seq_operations *seq_ops; +bpf_iter_init_seq_priv_t init_seq_private; +bpf_iter_fini_seq_priv_t fini_seq_private; +u32 seq_priv_size; +}; + +int bpf_iter_reg_target (struct bpf_iter_reg *reg_info); +void bpf_iter_unreg_target (const char *target); +int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value);int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value); int bpf_percpu_hash_update(struct bpf_map *map, void *key, void *value,
kernel/bpf/Makefile +1 -1
@@ -2,7 +2,7 @@ obj-y := core.o CFLAGS_core.o += $(call cc-disable-warning, override-init) -obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o +obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o
kernel/bpf/bpf_iter.c new file +59 -0
59 lines not on this page · raw message
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 01/21] bpf: implement an interface to register bpf_iter targets
Quoted 19 lines · Yonghong Song
The target can call
bpf_iter_reg_target()to register itself. The needed information:target: target name seq_ops: the seq_file operations for the target init_seq_private target callback to initialize seq_priv during file open fini_seq_private target callback to clean up seq_priv during file release seq_priv_size: the private_data size needed by the seq_fileoperations
The target name represents a target which provides a
seq_opsfor iterating objects.The target can provide two callback functions,
init_seq_privateandfini_seq_private, called during file open/release time. For example,/proc/net/{tcp6,ipv6_route, netlink, ...}, net name space needs to be setup properly during file open and released properly during file release.
Quoted 5 lines
Function
bpf_iter_unreg_target()is also implemented to unregister a particular target.Signed-off-by Yonghong Song <[email protected]>---
Lost my initial ack? :) Here we go again:
Quoted 6 lines
include/linux/bpf.h | 15 +++++++++++ kernel/bpf/Makefile | 2 +- kernel/bpf/bpf_iter.c | 59 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 75 insertions(+), 1 deletion(-) create mode 100644 kernel/bpf/bpf_iter.c
[...]
PATCH v3 2/21 bpf: allow loading of a bpf_iter program
In reply to Yonghong SongA bpf_iter program is a tracing program with attach type BPF_TRACE_ITER. The load attribute
attach_btf_id
is used by the verifier against a particular kernel function, which represents a target, e.g., __bpf_iter__bpf_map for target bpf_map which is implemented later.
The program return value must be 0 or 1 for now.
0 : successful, except potential seq_file buffer overflow
which is handled by seq_file reader.
1 : request to restart the same object
In the future, other return values may be used for filtering or teminating the iterator.
--- include/linux/bpf.h | 3 +++ include/uapi/linux/bpf.h | 1 + kernel/bpf/bpf_iter.c | 36 ++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 21 ++++++++++++++++++++ tools/include/uapi/linux/bpf.h | 1 + 5 files changed, 62 insertions(+)
include/linux/bpf.h +3 -0
@@ -1127,6 +1127,8 @@ struct bpf_link *bpf_link_get_from_fd(u32 ufd);int bpf_obj_pin_user (u32 ufd,const char __user *pathname);int bpf_obj_get_user (const char __user *pathname,int flags); +#define BPF_ITER_FUNC_PREFIX "__bpf_iter__" +typedef int (*bpf_iter_init_seq_priv_t )(void *private_data);typedef void (*bpf_iter_fini_seq_priv_t )(void *private_data);
@@ -1140,6 +1142,7 @@ struct bpf_iter_reg {int bpf_iter_reg_target (struct bpf_iter_reg *reg_info);void bpf_iter_unreg_target (const char *target); +bool bpf_iter_prog_supported (struct bpf_prog *prog);int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value);int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value);
include/uapi/linux/bpf.h +1 -0
@@ -218,6 +218,7 @@ enum bpf_attach_type {BPF_TRACE_FEXIT ,BPF_MODIFY_RETURN ,BPF_LSM_MAC , +BPF_TRACE_ITER , __MAX_BPF_ATTACH_TYPE };
kernel/bpf/bpf_iter.c +36 -0
@@ -12,6 +12,7 @@ struct bpf_iter_target_info {bpf_iter_init_seq_priv_t init_seq_private;bpf_iter_fini_seq_priv_t fini_seq_private;u32 seq_priv_size; +u32 btf_id;/* cached value */ };static struct list_head targets =LIST_HEAD_INIT (targets);
@@ -57,3 +58,38 @@ void bpf_iter_unreg_target(const char *target)WARN_ON (found == false); } + +static void cache_btf_id (struct bpf_iter_target_info *tinfo, +struct bpf_prog *prog) +{ + tinfo->btf_id = prog->aux->attach_btf_id; +} + +bool bpf_iter_prog_supported (struct bpf_prog *prog) +{ +const char *attach_fname = prog->aux->attach_func_name; +u32 prog_btf_id = prog->aux->attach_btf_id; +const char *prefix =BPF_ITER_FUNC_PREFIX ; +struct bpf_iter_target_info *tinfo; +int prefix_len =strlen (prefix); +bool supported = false; + +if (strncmp (attach_fname, prefix, prefix_len)) +return false; + +mutex_lock (&targets_mutex); +list_for_each_entry (tinfo, &targets, list) { +if (tinfo->btf_id && tinfo->btf_id == prog_btf_id) { + supported = true; +break ; + } +if (!strcmp (attach_fname + prefix_len, tinfo->target)) { +cache_btf_id (tinfo, prog); + supported = true; +break ; + } + } +mutex_unlock (&targets_mutex); + +return supported; +}
kernel/bpf/verifier.c +21 -0
@@ -7101,6 +7101,10 @@ static int check_return_code(struct bpf_verifier_env *env)return 0 ; range =tnum_const (0 );break ; +case BPF_PROG_TYPE_TRACING : +if (env->prog->expected_attach_type !=BPF_TRACE_ITER ) +return 0 ; +break ;default :return 0 ; }
@@ -10481,6 +10485,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)struct bpf_prog *tgt_prog = prog->aux->linked_prog;u32 btf_id = prog->aux->attach_btf_id;const char prefix[] ="btf_trace_" ; +struct btf_func_model fmodel;int ret =0 , subprog =-1 , i;struct bpf_trampoline *tr;const struct btf_type *t;
@@ -10622,6 +10627,22 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) prog->aux->attach_func_proto = t; prog->aux->attach_btf_trace = true;return 0 ; +case BPF_TRACE_ITER : +if (!btf_type_is_func (t)) { +verbose (env,"attach_btf_id %u is not a function\n" , + btf_id); +return -EINVAL ; + } + t =btf_type_by_id (btf, t->type); +if (!btf_type_is_func_proto (t)) +return -EINVAL ; + prog->aux->attach_func_name = tname; + prog->aux->attach_func_proto = t; +if (!bpf_iter_prog_supported (prog)) +return -EINVAL ; + ret =btf_distill_func_proto (&env->log, btf, t, + tname, &fmodel); +return ret;default :if (!prog_extension)return -EINVAL ;
tools/include/uapi/linux/bpf.h +1 -0
@@ -218,6 +218,7 @@ enum bpf_attach_type {BPF_TRACE_FEXIT ,BPF_MODIFY_RETURN ,BPF_LSM_MAC , +BPF_TRACE_ITER , __MAX_BPF_ATTACH_TYPE };
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 02/21] bpf: allow loading of a bpf_iter program
Quoted 13 lines · Yonghong Song
A
bpf_iterprogram is a tracing program with attach typeBPF_TRACE_ITER. The load attribute
attach_btf_id
is used by the verifier against a particular kernel function, which represents a target, e.g.,__bpf_iter__bpf_mapfor targetbpf_mapwhich is implemented later.The program return value must be 0 or 1 for now.
0 : successful, except potentialseq_filebuffer overflow
which is handled byseq_filereader.
1 : request to restart the same object
Quoted 5 lines
In the future, other return values may be used for filtering or teminating the iterator.
Signed-off-by Yonghong Song <[email protected]>---
Looks good.
Quoted 7 lines
include/linux/bpf.h | 3 +++ include/uapi/linux/bpf.h | 1 + kernel/bpf/bpf_iter.c | 36 ++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 21 ++++++++++++++++++++ tools/include/uapi/linux/bpf.h | 1 + 5 files changed, 62 insertions(+)
[...]
PATCH v3 3/21 bpf: support bpf tracing/iter programs for BPF_LINK_CREATE
In reply to Yonghong SongGiven a bpf program, the step to create an anonymous bpf iterator is:
- create a bpf_iter_link, which combines bpf program and the target. In the future, there could be more information recorded in the link. A link_fd will be returned to the user space.
- create an anonymous bpf iterator with the given link_fd.
The bpf_iter_link can be pinned to bpffs mount file system to create a file based bpf iterator as well.
The benefit to use of bpf_iter_link:
- using bpf link simplifies design and implementation as bpf link is used for other tracing bpf programs.
- for file based bpf iterator, bpf_iter_link provides a standard way to replace underlying bpf programs.
- for both anonymous and free based iterators, bpf link query capability can be leveraged.
The patch added support of tracing/iter programs for BPF_LINK_CREATE. A new link type BPF_LINK_TYPE_ITER is added to facilitate link querying. Currently, only prog_id is needed, so there is no additional in-kernel show_fdinfo() and fill_link_info() hook is needed for BPF_LINK_TYPE_ITER link.
--- include/linux/bpf.h | 1 + include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/bpf_iter.c | 62 ++++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 14 ++++++++ tools/include/uapi/linux/bpf.h | 1 + 6 files changed, 80 insertions(+)
include/linux/bpf.h +1 -0
@@ -1143,6 +1143,7 @@ struct bpf_iter_reg {int bpf_iter_reg_target (struct bpf_iter_reg *reg_info);void bpf_iter_unreg_target (const char *target);bool bpf_iter_prog_supported (struct bpf_prog *prog); +int bpf_iter_link_attach (const union bpf_attr *attr,struct bpf_prog *prog);int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value);int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value);
include/linux/bpf_types.h +1 -0
@@ -124,3 +124,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing)#ifdef CONFIG_CGROUP_BPF BPF_LINK_TYPE(BPF_LINK_TYPE_CGROUP, cgroup)#endif +BPF_LINK_TYPE (BPF_LINK_TYPE_ITER , iter)
include/uapi/linux/bpf.h +1 -0
@@ -229,6 +229,7 @@ enum bpf_link_type {BPF_LINK_TYPE_RAW_TRACEPOINT =1 ,BPF_LINK_TYPE_TRACING =2 ,BPF_LINK_TYPE_CGROUP =3 , +BPF_LINK_TYPE_ITER =4 , MAX_BPF_LINK_TYPE, };
kernel/bpf/bpf_iter.c +62 -0
71 lines not on this page · raw message
kernel/bpf/syscall.c +14 -0
@@ -2729,6 +2729,8 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type)case BPF_CGROUP_GETSOCKOPT :case BPF_CGROUP_SETSOCKOPT :return BPF_PROG_TYPE_CGROUP_SOCKOPT ; +case BPF_TRACE_ITER : +return BPF_PROG_TYPE_TRACING ;default :return BPF_PROG_TYPE_UNSPEC ; }
@@ -3729,6 +3731,15 @@ static int bpf_map_do_batch(const union bpf_attr *attr,return err; } +static int tracing_bpf_link_attach (const union bpf_attr *attr,struct bpf_prog *prog) +{ +if (attr->link_create.attach_type ==BPF_TRACE_ITER && + prog->expected_attach_type ==BPF_TRACE_ITER ) +return bpf_iter_link_attach (attr, prog); + +return -EINVAL ; +} +#define BPF_LINK_CREATE_LAST_FIELD link_create.flagsstatic int link_create (union bpf_attr *attr) {
@@ -3765,6 +3776,9 @@ static int link_create(union bpf_attr *attr)case BPF_PROG_TYPE_CGROUP_SOCKOPT : ret =cgroup_bpf_link_attach (attr, prog);break ; +case BPF_PROG_TYPE_TRACING : + ret =tracing_bpf_link_attach (attr, prog); +break ;default : ret = -EINVAL ; }
tools/include/uapi/linux/bpf.h +1 -0
@@ -229,6 +229,7 @@ enum bpf_link_type {BPF_LINK_TYPE_RAW_TRACEPOINT =1 ,BPF_LINK_TYPE_TRACING =2 ,BPF_LINK_TYPE_CGROUP =3 , +BPF_LINK_TYPE_ITER =4 , MAX_BPF_LINK_TYPE, };
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 03/21] bpf: support bpf tracing/iter programs for BPF_LINK_CREATE
Quoted 24 lines · Yonghong Song
Given a bpf program, the step to create an anonymous bpf iterator is:
- create abpf_iter_link, which combines bpf program and the target. In the future, there could be more information recorded in the link. Alink_fdwill be returned to the user space.
- create an anonymous bpf iterator with the givenlink_fd.The
bpf_iter_linkcan be pinned to bpffs mount file system to create a file based bpf iterator as well.The benefit to use of
bpf_iter_link:
- using bpf link simplifies design and implementation as bpf link is used for other tracing bpf programs.
- for file based bpf iterator,bpf_iter_linkprovides a standard way to replace underlying bpf programs.
- for both anonymous and free based iterators, bpf link query capability can be leveraged.The patch added support of tracing/iter programs for
BPF_LINK_CREATE. A new link typeBPF_LINK_TYPE_ITERis added to facilitate link querying. Currently, onlyprog_idis needed, so there is no additional in-kernelshow_fdinfo()andfill_link_info()hook is needed forBPF_LINK_TYPE_ITERlink.
Quoted 3 lines
Acked-by Andrii Nakryiko <[email protected]>Signed-off-by Yonghong Song <[email protected]>---
still looks good, but I realized show_fdinfo and fill_link_info is missing, see request for a follow-up below :)
Quoted 8 lines
include/linux/bpf.h | 1 + include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/bpf_iter.c | 62 ++++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 14 ++++++++ tools/include/uapi/linux/bpf.h | 1 + 6 files changed, 80 insertions(+)
[...]
kernel/bpf/bpf_iter.c bool bpf_iter_prog_supported(struct bpf_prog *prog) · line 117
Quoted 4 lines
kernel/bpf/bpf_iter.c +4
-0+static const struct bpf_link_ops bpf_iter_link_lops = { + .release = bpf_iter_link_release, + .dealloc = bpf_iter_link_dealloc, +};
Link infra supports .show_fdinfo and .fill_link_info methods, there is no need to block on this, but it would be great to implement them from BPF_LINK_TYPE_ITER as well in the same release as a follow-up. Thanks!
[...]
Re: [PATCH bpf-next v3 03/21] bpf: support bpf tracing/iter programs for BPF_LINK_CREATE
Quoted 44 lines · Andrii Nakryiko
Quoted 24 lines · Yonghong Song
Given a bpf program, the step to create an anonymous bpf iterator is:
- create abpf_iter_link, which combines bpf program and the target. In the future, there could be more information recorded in the link. Alink_fdwill be returned to the user space.
- create an anonymous bpf iterator with the givenlink_fd.The
bpf_iter_linkcan be pinned to bpffs mount file system to create a file based bpf iterator as well.The benefit to use of
bpf_iter_link:
- using bpf link simplifies design and implementation as bpf link is used for other tracing bpf programs.
- for file based bpf iterator,bpf_iter_linkprovides a standard way to replace underlying bpf programs.
- for both anonymous and free based iterators, bpf link query capability can be leveraged.The patch added support of tracing/iter programs for
BPF_LINK_CREATE. A new link typeBPF_LINK_TYPE_ITERis added to facilitate link querying. Currently, onlyprog_idis needed, so there is no additional in-kernelshow_fdinfo()andfill_link_info()hook is needed forBPF_LINK_TYPE_ITERlink.Quoted 3 lines
Acked-by Andrii Nakryiko <[email protected]>Signed-off-by Yonghong Song <[email protected]>---
still looks good, but I realized
show_fdinfoandfill_link_infois missing, see request for a follow-up below :)Quoted 8 lines
include/linux/bpf.h | 1 + include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/bpf_iter.c | 62 ++++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 14 ++++++++ tools/include/uapi/linux/bpf.h | 1 + 6 files changed, 80 insertions(+)[...]
Quoted 8 lines
kernel/bpf/bpf_iter.c bool bpf_iter_prog_supported(struct bpf_prog *prog) · line 117
Quoted 4 lines
kernel/bpf/bpf_iter.c +4
-0+static const struct bpf_link_ops bpf_iter_link_lops = { + .release = bpf_iter_link_release, + .dealloc = bpf_iter_link_dealloc, +};Link infra supports .show_fdinfo and .fill_link_info methods, there is no need to block on this, but it would be great to implement them from
BPF_LINK_TYPE_ITERas well in the same release as a follow-up. Thanks!
The reason I did not implement is due to we do not have additional information beyond prog_id to present. The prog_id itself gives all information about this link. I looked at tracing program show_fdinfo/fill_link_info, the additional attach_type is printed. But attach_type is obvious for BPF_LINK_TYPE_ITER which does not
need print.
In the future when we add more stuff to parameterize the bpf_iter, will need to implement these two callbacks as well as bpftool.
Quoted 4 lines
[...]
Re: [PATCH bpf-next v3 03/21] bpf: support bpf tracing/iter programs for BPF_LINK_CREATE
Quoted 57 lines · Yonghong Song
Quoted 44 lines · Andrii Nakryiko
Quoted 24 lines · Yonghong Song
Given a bpf program, the step to create an anonymous bpf iterator is:
- create abpf_iter_link, which combines bpf program and the target. In the future, there could be more information recorded in the link. Alink_fdwill be returned to the user space.
- create an anonymous bpf iterator with the givenlink_fd.The
bpf_iter_linkcan be pinned to bpffs mount file system to create a file based bpf iterator as well.The benefit to use of
bpf_iter_link:
- using bpf link simplifies design and implementation as bpf link is used for other tracing bpf programs.
- for file based bpf iterator,bpf_iter_linkprovides a standard way to replace underlying bpf programs.
- for both anonymous and free based iterators, bpf link query capability can be leveraged.The patch added support of tracing/iter programs for
BPF_LINK_CREATE. A new link typeBPF_LINK_TYPE_ITERis added to facilitate link querying. Currently, onlyprog_idis needed, so there is no additional in-kernelshow_fdinfo()andfill_link_info()hook is needed forBPF_LINK_TYPE_ITERlink.Quoted 3 lines
Acked-by Andrii Nakryiko <[email protected]>Signed-off-by Yonghong Song <[email protected]>---
still looks good, but I realized
show_fdinfoandfill_link_infois missing, see request for a follow-up below :)Quoted 8 lines
include/linux/bpf.h | 1 + include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/bpf_iter.c | 62 ++++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 14 ++++++++ tools/include/uapi/linux/bpf.h | 1 + 6 files changed, 80 insertions(+)[...]
Quoted 8 lines
kernel/bpf/bpf_iter.c bool bpf_iter_prog_supported(struct bpf_prog *prog) · line 117
Quoted 4 lines
kernel/bpf/bpf_iter.c +4
-0+static const struct bpf_link_ops bpf_iter_link_lops = { + .release = bpf_iter_link_release, + .dealloc = bpf_iter_link_dealloc, +};Link infra supports .show_fdinfo and .fill_link_info methods, there is no need to block on this, but it would be great to implement them from
BPF_LINK_TYPE_ITERas well in the same release as a follow-up. Thanks!
Quoted 3 lines
The reason I did not implement is due to we do not have additional information beyond
prog_idto present. Theprog_iditself gives all information about this link. I looked at tracing program
Not all, e.g., bpf_iter target is invisible right now. It's good to have this added in a follow up, but certainly not a blocker.
Quoted 6 lines
show_fdinfo/fill_link_info, the additional
attach_typeis printed. Butattach_typeis obvious forBPF_LINK_TYPE_ITERwhich does not need print.In the future when we add more stuff to parameterize the
bpf_iter, will need to implement these two callbacks as well as bpftool.
yep
Quoted 5 lines
Quoted 5 lines
[...]
Re: [PATCH bpf-next v3 03/21] bpf: support bpf tracing/iter programs for BPF_LINK_CREATE
Quoted 58 lines · Andrii Nakryiko
Quoted 57 lines · Yonghong Song
Quoted 52 lines · Andrii Nakryiko
Quoted 24 lines · Yonghong Song
Given a bpf program, the step to create an anonymous bpf iterator is:
- create abpf_iter_link, which combines bpf program and the target. In the future, there could be more information recorded in the link. Alink_fdwill be returned to the user space.
- create an anonymous bpf iterator with the givenlink_fd.The
bpf_iter_linkcan be pinned to bpffs mount file system to create a file based bpf iterator as well.The benefit to use of
bpf_iter_link:
- using bpf link simplifies design and implementation as bpf link is used for other tracing bpf programs.
- for file based bpf iterator,bpf_iter_linkprovides a standard way to replace underlying bpf programs.
- for both anonymous and free based iterators, bpf link query capability can be leveraged.The patch added support of tracing/iter programs for
BPF_LINK_CREATE. A new link typeBPF_LINK_TYPE_ITERis added to facilitate link querying. Currently, onlyprog_idis needed, so there is no additional in-kernelshow_fdinfo()andfill_link_info()hook is needed forBPF_LINK_TYPE_ITERlink.Quoted 3 lines
Acked-by Andrii Nakryiko <[email protected]>Signed-off-by Yonghong Song <[email protected]>---
still looks good, but I realized
show_fdinfoandfill_link_infois missing, see request for a follow-up below :)Quoted 8 lines
include/linux/bpf.h | 1 + include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/bpf_iter.c | 62 ++++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 14 ++++++++ tools/include/uapi/linux/bpf.h | 1 + 6 files changed, 80 insertions(+)[...]
kernel/bpf/bpf_iter.c bool bpf_iter_prog_supported(struct bpf_prog *prog) · line 117
Quoted 4 lines
kernel/bpf/bpf_iter.c +4
-0+static const struct bpf_link_ops bpf_iter_link_lops = { + .release = bpf_iter_link_release, + .dealloc = bpf_iter_link_dealloc, +};Link infra supports .show_fdinfo and .fill_link_info methods, there is no need to block on this, but it would be great to implement them from
BPF_LINK_TYPE_ITERas well in the same release as a follow-up. Thanks!
Quoted 6 lines
Quoted 3 lines
The reason I did not implement is due to we do not have additional information beyond
prog_idto present. Theprog_iditself gives all information about this link. I looked at tracing programNot all, e.g.,
bpf_itertarget is invisible right now. It's good to have this added in a follow up, but certainly not a blocker.
Indeed, bpf_iter target is not visible now. We can add it to program or to link. Adding to link is a reasonable idea as link itself is the concept to merge program and target.
Will have a follow up patch for this later.
Quoted 16 lines
Quoted 8 lines
show_fdinfo/fill_link_info, the additional
attach_typeis printed. Butattach_typeis obvious forBPF_LINK_TYPE_ITERwhich does not need print.In the future when we add more stuff to parameterize the
bpf_iter, will need to implement these two callbacks as well as bpftool.yep
Quoted 5 lines
Quoted 5 lines
[...]
PATCH v3 4/21 bpf: support bpf tracing/iter programs for BPF_LINK_UPDATE
In reply to Yonghong SongAdded BPF_LINK_UPDATE support for tracing/iter programs. This way, a file based bpf iterator, which holds a reference to the link, can have its bpf program updated without creating new files.
--- kernel/bpf/bpf_iter.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
kernel/bpf/bpf_iter.c +31 -0
@@ -23,6 +23,9 @@ struct bpf_iter_link {static struct list_head targets =LIST_HEAD_INIT (targets);static DEFINE_MUTEX (targets_mutex); +/* protect bpf_iter_link changes */ +static DEFINE_MUTEX (link_mutex); +int bpf_iter_reg_target (struct bpf_iter_reg *reg_info) {struct bpf_iter_target_info *tinfo;
@@ -111,9 +114,37 @@ static void bpf_iter_link_dealloc(struct bpf_link *link)kfree (iter_link); } +static int bpf_iter_link_replace (struct bpf_link *link, +struct bpf_prog *new_prog, +struct bpf_prog *old_prog) +{ +int ret =0 ; + +mutex_lock (&link_mutex); +if (old_prog && link->prog != old_prog) { + ret = -EPERM ; + goto out_unlock; + } + +if (link->prog->type != new_prog->type || + link->prog->expected_attach_type != new_prog->expected_attach_type || + link->prog->aux->attach_btf_id != new_prog->aux->attach_btf_id) { + ret = -EINVAL ; + goto out_unlock; + } + + old_prog =xchg (&link->prog, new_prog); +bpf_prog_put (old_prog); + +out_unlock: +mutex_unlock (&link_mutex); +return ret; +} +static const struct bpf_link_ops bpf_iter_link_lops = { .release = bpf_iter_link_release, .dealloc = bpf_iter_link_dealloc, + .update_prog = bpf_iter_link_replace, };int bpf_iter_link_attach (const union bpf_attr *attr,struct bpf_prog *prog)
Signature · 1 line
2.24.1
PATCH v3 5/21 bpf: implement bpf_seq_read() for bpf iterator
In reply to Yonghong Songbpf iterator uses seq_file to provide a lossless way to transfer data to user space. But we want to call bpf program after all objects have been traversed, and bpf program may write additional data to the seq_file buffer. The current seq_read() does not work for this use case.
Besides allowing stop() function to write to the buffer, the bpf_seq_read() also fixed the buffer size to one page. If any single call of show() or stop() will emit data more than one page to cause overflow, -E2BIG error code will be returned to user space.
--- kernel/bpf/bpf_iter.c | 118 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+)
kernel/bpf/bpf_iter.c +118 -0
@@ -26,6 +26,124 @@ static DEFINE_MUTEX(targets_mutex);/* protect bpf_iter_link changes */ static DEFINE_MUTEX (link_mutex); +/* bpf_seq_read, a customized and simpler version for bpf iterator. +* no_llseek is assumed for this file. +* The following are differences from seq_read(): +* . fixed buffer size (PAGE_SIZE) +* . assuming no_llseek +* . stop() may call bpf program, handling potential overflow there +*/ +static ssize_t bpf_seq_read (struct file *file,char __user *buf,size_t size, +loff_t *ppos) +{ +struct seq_file *seq = file->private_data; +size_t n, offs, copied =0 ; +int err =0 ; +void *p; + +mutex_lock (&seq->lock); + +if (!seq->buf) { + seq->size =PAGE_SIZE ; + seq->buf =kmalloc (seq->size,GFP_KERNEL ); +if (!seq->buf) { + err = -ENOMEM ; + goto done; + } + } + +if (seq->count) { + n =min (seq->count, size); + err =copy_to_user (buf, seq->buf + seq->from, n); +if (err) { + err = -EFAULT ; + goto done; + } + seq->count -= n; + seq->from += n; + copied = n; + goto done; + } + + seq->from =0 ; + p = seq->op->start (seq, &seq->index); +if (IS_ERR_OR_NULL (p)) + goto stop; + + err = seq->op->show (seq, p); +if (err >0 ) { + seq->count =0 ; + }else if (err <0 ||seq_has_overflowed (seq)) { +if (!err) + err = -E2BIG ; + seq->count =0 ; + seq->op->stop (seq, p); + goto done; + } + +while (1 ) { +loff_t pos = seq->index; + + offs = seq->count; + p = seq->op->next (seq, p, &seq->index); +if (pos == seq->index) { +pr_info_ratelimited ("buggy seq_file .next function %ps " +"did not updated position index\n" , + seq->op->next); + seq->index++; + } + +if (IS_ERR_OR_NULL (p)) { + err =PTR_ERR (p); +break ; + } +if (seq->count >= size) +break ; + + err = seq->op->show (seq, p); +if (err >0 ) { + seq->count = offs; + }else if (err <0 ||seq_has_overflowed (seq)) { + seq->count = offs; +if (!err) + err = -E2BIG ; +if (offs ==0 ) { + seq->op->stop (seq, p); + goto done; + } +break ; + } + } +stop: + offs = seq->count; +/* bpf program called if !p */ + seq->op->stop (seq, p); +if (!p &&seq_has_overflowed (seq)) { + seq->count = offs; +if (offs ==0 ) { + err = -E2BIG ; + goto done; + } + } + + n =min (seq->count, size); + err =copy_to_user (buf, seq->buf, n); +if (err) { + err = -EFAULT ; + goto done; + } + copied = n; + seq->count -= n; + seq->from = n; +done: +if (!copied) + copied = err; +else + *ppos += copied; +mutex_unlock (&seq->lock); +return copied; +} +int bpf_iter_reg_target (struct bpf_iter_reg *reg_info) {struct bpf_iter_target_info *tinfo;
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 05/21] bpf: implement bpf_seq_read() for bpf iterator
Quoted 8 lines · Yonghong Song
bpf iterator uses
seq_fileto provide a lossless way to transfer data to user space. But we want to call bpf program after all objects have been traversed, and bpf program may write additional data to theseq_filebuffer. The currentseq_read()does not work for this use case.
Quoted 8 lines
Besides allowing
stop()function to write to the buffer, thebpf_seq_read()also fixed the buffer size to one page. If any single call ofshow()orstop()will emit data more than one page to cause overflow,-E2BIGerror code will be returned to user space.Signed-off-by Yonghong Song <[email protected]>---
This loop is much simpler and more streamlined now, thanks a lot! I think it's correct, see below about one confusing (but apparently correct) bit, though. Either way:
Quoted 24 lines
kernel/bpf/bpf_iter.c | 118 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+)kernel/bpf/bpf_iter.c +13
-0@@ -26,6 +26,124 @@ static DEFINE_MUTEX(targets_mutex);/* protect bpf_iter_link changes */ static DEFINE_MUTEX (link_mutex); +/* bpf_seq_read, a customized and simpler version for bpf iterator. +* no_llseek is assumed for this file. +* The following are differences from seq_read(): +* . fixed buffer size (PAGE_SIZE) +* . assuming no_llseek +* . stop() may call bpf program, handling potential overflow there +*/ +static ssize_t bpf_seq_read (struct file *file,char __user *buf,size_t size, +loff_t *ppos) +{ +struct seq_file *seq = file->private_data; +size_t n, offs, copied =0 ; +int err =0 ;
kernel/bpf/bpf_iter.c static DEFINE_MUTEX(targets_mutex); · line 51
Quoted 10 lines
kernel/bpf/bpf_iter.c +10
-0+void *p; + +mutex_lock (&seq->lock); + +if (!seq->buf) { + seq->size =PAGE_SIZE ; + seq->buf =kmalloc (seq->size,GFP_KERNEL ); +if (!seq->buf) { + err = -ENOMEM ; + goto done;
oh, thank you for converting to all lower-case label names! :)
Quoted 10 lines
kernel/bpf/bpf_iter.c +10
-0+ } + } + +if (seq->count) { + n =min (seq->count, size); + err =copy_to_user (buf, seq->buf + seq->from, n); +if (err) { + err = -EFAULT ; + goto done; + }
kernel/bpf/bpf_iter.c static DEFINE_MUTEX(targets_mutex); · line 71
Quoted 10 lines
kernel/bpf/bpf_iter.c +10
-0+ seq->count -= n; + seq->from += n; + copied = n; + goto done; + } + + seq->from =0 ; + p = seq->op->start (seq, &seq->index); +if (IS_ERR_OR_NULL (p)) + goto stop;
if start() returns IS_ERR(p), stop(p) below won't produce any output (because BPF program is called only for p == NULL), so we'll just return 0 with no error, do I interpret the code correctly? I think seq_file's read actually returns PTR_ERR(p) as a result in this case.
so I think you need err = PTR_ERR(p); before goto stop here?
Quoted 28 lines
kernel/bpf/bpf_iter.c +28
-0+ + err = seq->op->show (seq, p); +if (err >0 ) { + seq->count =0 ; + }else if (err <0 ||seq_has_overflowed (seq)) { +if (!err) + err = -E2BIG ; + seq->count =0 ; + seq->op->stop (seq, p); + goto done; + } + +while (1 ) { +loff_t pos = seq->index; + + offs = seq->count; + p = seq->op->next (seq, p, &seq->index); +if (pos == seq->index) { +pr_info_ratelimited ("buggy seq_file .next function %ps " +"did not updated position index\n" , + seq->op->next); + seq->index++; + } + +if (IS_ERR_OR_NULL (p)) { + err =PTR_ERR (p); +break ; + }
kernel/bpf/bpf_iter.c static DEFINE_MUTEX(targets_mutex); · line 109
Quoted 10 lines
kernel/bpf/bpf_iter.c +10
-0+if (seq->count >= size) +break ; + + err = seq->op->show (seq, p); +if (err >0 ) { + seq->count = offs; + }else if (err <0 ||seq_has_overflowed (seq)) { + seq->count = offs; +if (!err) + err = -E2BIG ;
nit: this -E2BIG is set unconditionally even for 2nd+ show(). This will work, because it will get ignored on next iteration, but I think it will be much more obvious if written as:
if (!err && offs = 0)
err = -E2BIG;
It took me few re-readings of the code I'm pretty familiar with already to realize that this is ok.
I had to write the below piece to realize that this is fine :) Just leaving here just in case you find it useful:
else if (err < 0 || seq_has_overflowed(seq)) {
if (!err && offs == 0) /* overflow in first show() output */
err = -E2BIG;
if (err) { /* overflow in first show() or real error happened */
seq->count = 0; /* not strictly necessary, but shows that we
are truncating output */
seq->op->stop(seq, p);
goto done; /* done will return err */
}
/* no error and overflow for 2nd+ show(), roll back output and stop */
seq->count = offs;
break;
}Quoted 43 lines
kernel/bpf/bpf_iter.c +37
-1+if (offs ==0 ) { + seq->op->stop (seq, p); + goto done; + } +break ; + } + } +stop: + offs = seq->count; +/* bpf program called if !p */ + seq->op->stop (seq, p); +if (!p &&seq_has_overflowed (seq)) { + seq->count = offs; +if (offs ==0 ) { + err = -E2BIG ; + goto done; + } + } + + n =min (seq->count, size); + err =copy_to_user (buf, seq->buf, n); +if (err) { + err = -EFAULT ; + goto done; + } + copied = n; + seq->count -= n; + seq->from = n; +done: +if (!copied) + copied = err; +else + *ppos += copied; +mutex_unlock (&seq->lock); +return copied; +} +int bpf_iter_reg_target (struct bpf_iter_reg *reg_info) {struct bpf_iter_target_info *tinfo; --2.24.1
Re: [PATCH bpf-next v3 05/21] bpf: implement bpf_seq_read() for bpf iterator
Quoted 87 lines · Andrii Nakryiko
Quoted 8 lines · Yonghong Song
bpf iterator uses
seq_fileto provide a lossless way to transfer data to user space. But we want to call bpf program after all objects have been traversed, and bpf program may write additional data to theseq_filebuffer. The currentseq_read()does not work for this use case.Quoted 8 lines
Besides allowing
stop()function to write to the buffer, thebpf_seq_read()also fixed the buffer size to one page. If any single call ofshow()orstop()will emit data more than one page to cause overflow,-E2BIGerror code will be returned to user space.Signed-off-by Yonghong Song <[email protected]>---
This loop is much simpler and more streamlined now, thanks a lot! I think it's correct, see below about one confusing (but apparently correct) bit, though. Either way:
Acked-by Andrii Nakryiko <[email protected]>Quoted 24 lines
kernel/bpf/bpf_iter.c | 118 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+)kernel/bpf/bpf_iter.c +13
-0@@ -26,6 +26,124 @@ static DEFINE_MUTEX(targets_mutex);/* protect bpf_iter_link changes */ static DEFINE_MUTEX (link_mutex); +/* bpf_seq_read, a customized and simpler version for bpf iterator. +* no_llseek is assumed for this file. +* The following are differences from seq_read(): +* . fixed buffer size (PAGE_SIZE) +* . assuming no_llseek +* . stop() may call bpf program, handling potential overflow there +*/ +static ssize_t bpf_seq_read (struct file *file,char __user *buf,size_t size, +loff_t *ppos) +{ +struct seq_file *seq = file->private_data; +size_t n, offs, copied =0 ; +int err =0 ;kernel/bpf/bpf_iter.c static DEFINE_MUTEX(targets_mutex); · line 51
Quoted 10 lines
kernel/bpf/bpf_iter.c +10
-0+void *p; + +mutex_lock (&seq->lock); + +if (!seq->buf) { + seq->size =PAGE_SIZE ; + seq->buf =kmalloc (seq->size,GFP_KERNEL ); +if (!seq->buf) { + err = -ENOMEM ; + goto done;oh, thank you for converting to all lower-case label names! :)
Quoted 10 lines
kernel/bpf/bpf_iter.c +10
-0+ } + } + +if (seq->count) { + n =min (seq->count, size); + err =copy_to_user (buf, seq->buf + seq->from, n); +if (err) { + err = -EFAULT ; + goto done; + }kernel/bpf/bpf_iter.c static DEFINE_MUTEX(targets_mutex); · line 71
Quoted 10 lines
kernel/bpf/bpf_iter.c +10
-0+ seq->count -= n; + seq->from += n; + copied = n; + goto done; + } + + seq->from =0 ; + p = seq->op->start (seq, &seq->index); +if (IS_ERR_OR_NULL (p)) + goto stop;if
start()returnsIS_ERR(p), stop(p) below won't produce any output (because BPF program is called only for p == NULL), so we'll just return 0 with no error, do I interpret the code correctly? I think seq_file's read actually returnsPTR_ERR(p)as a result in this case.
Quoted 1 line
so I think you need err =
PTR_ERR(p); before goto stop here?
Thanks for catching this!
Yes, seq_file() indeed returns PTR_ERR(p) to user space here. Will make the change.
Quoted 44 lines
Quoted 29 lines
kernel/bpf/bpf_iter.c +28
-0+ + err = seq->op->show (seq, p); +if (err >0 ) { + seq->count =0 ; + }else if (err <0 ||seq_has_overflowed (seq)) { +if (!err) + err = -E2BIG ; + seq->count =0 ; + seq->op->stop (seq, p); + goto done; + } + +while (1 ) { +loff_t pos = seq->index; + + offs = seq->count; + p = seq->op->next (seq, p, &seq->index); +if (pos == seq->index) { +pr_info_ratelimited ("buggy seq_file .next function %ps " +"did not updated position index\n" , + seq->op->next); + seq->index++; + } + +if (IS_ERR_OR_NULL (p)) { + err =PTR_ERR (p); +break ; + }kernel/bpf/bpf_iter.c static DEFINE_MUTEX(targets_mutex); · line 109
Quoted 10 lines
kernel/bpf/bpf_iter.c +10
-0+if (seq->count >= size) +break ; + + err = seq->op->show (seq, p); +if (err >0 ) { + seq->count = offs; + }else if (err <0 ||seq_has_overflowed (seq)) { + seq->count = offs; +if (!err) + err = -E2BIG ;nit: this
-E2BIGis set unconditionally even for 2nd+show(). This will work, because it will get ignored on next iteration, but I think it will be much more obvious if written as:
Quoted 2 lines
if (!err && offs = 0)
err = -E2BIG;
Yes, will make the change since it indeed makes code more readable.
Quoted 64 lines
It took me few re-readings of the code I'm pretty familiar with already to realize that this is ok.
I had to write the below piece to realize that this is fine :) Just leaving here just in case you find it useful:
else if (err < 0 || seq_has_overflowed(seq)) { if (!err && offs == 0) /* overflow in first show() output */ err = -E2BIG; if (err) { /* overflow in first show() or real error happened */ seq->count = 0; /* not strictly necessary, but shows that we are truncating output */ seq->op->stop(seq, p); goto done; /* done will return err */ } /* no error and overflow for 2nd+ show(), roll back output and stop */ seq->count = offs; break; }Quoted 43 lines
kernel/bpf/bpf_iter.c +37
-1+if (offs ==0 ) { + seq->op->stop (seq, p); + goto done; + } +break ; + } + } +stop: + offs = seq->count; +/* bpf program called if !p */ + seq->op->stop (seq, p); +if (!p &&seq_has_overflowed (seq)) { + seq->count = offs; +if (offs ==0 ) { + err = -E2BIG ; + goto done; + } + } + + n =min (seq->count, size); + err =copy_to_user (buf, seq->buf, n); +if (err) { + err = -EFAULT ; + goto done; + } + copied = n; + seq->count -= n; + seq->from = n; +done: +if (!copied) + copied = err; +else + *ppos += copied; +mutex_unlock (&seq->lock); +return copied; +} +int bpf_iter_reg_target (struct bpf_iter_reg *reg_info) {struct bpf_iter_target_info *tinfo; --2.24.1
PATCH v3 6/21 bpf: create anonymous bpf iterator
In reply to Yonghong SongA new bpf command BPF_ITER_CREATE is added.
The anonymous bpf iterator is seq_file based.
The seq_file private data are referenced by targets. The bpf_iter infrastructure allocated additional space at seq_file->private before the space used by targets to store some meta data, e.g.,
prog: prog to run
session_id: an unique id for each opened seq_file
seq_num: how many times bpf programs are queried in this session
done_stop: an internal state to decide whether bpf program
should be called in seq_ops->stop() or not
The seq_num will start from 0 for valid objects. The bpf program may see the same seq_num more than once if
- seq_file buffer overflow happens and the same object is retried by bpf_seq_read(), or
- the bpf program explicitly requests a retry of the same object
Since module is not supported for bpf_iter, all target registeration happens at __init time, so there is no need to change bpf_iter_unreg_target() as it is used mostly in error path of the init function at which time no bpf iterators have been created yet.
--- include/linux/bpf.h | 1 + include/uapi/linux/bpf.h | 6 ++ kernel/bpf/bpf_iter.c | 129 +++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 26 +++++++ tools/include/uapi/linux/bpf.h | 6 ++ 5 files changed, 168 insertions(+)
include/linux/bpf.h +1 -0
@@ -1144,6 +1144,7 @@ int bpf_iter_reg_target(struct bpf_iter_reg *reg_info);void bpf_iter_unreg_target (const char *target);bool bpf_iter_prog_supported (struct bpf_prog *prog);int bpf_iter_link_attach (const union bpf_attr *attr,struct bpf_prog *prog); +int bpf_iter_new_fd (struct bpf_link *link);int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value);int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value);
include/uapi/linux/bpf.h +6 -0
@@ -116,6 +116,7 @@ enum bpf_cmd {BPF_LINK_GET_FD_BY_ID ,BPF_LINK_GET_NEXT_ID ,BPF_ENABLE_STATS , + BPF_ITER_CREATE, }; enum bpf_map_type {
@@ -614,6 +615,11 @@ union bpf_attr {__u32 type; } enable_stats; +struct {/* struct used by BPF_ITER_CREATE command */ +__u32 link_fd; +__u32 flags; + } iter_create; + } __attribute__((aligned(8)));/* The description below is an attempt at providing documentation to eBPF
kernel/bpf/bpf_iter.c +129 -0
156 lines not on this page · raw message
kernel/bpf/syscall.c +26 -0
@@ -3941,6 +3941,29 @@ static int bpf_enable_stats(union bpf_attr *attr)return -EINVAL ; } +#define BPF_ITER_CREATE_LAST_FIELD iter_create.flags + +static int bpf_iter_create (union bpf_attr *attr) +{ +struct bpf_link *link; +int err; + +if (CHECK_ATTR (BPF_ITER_CREATE )) +return -EINVAL ; + +if (attr->iter_create.flags) +return -EINVAL ; + + link =bpf_link_get_from_fd (attr->iter_create.link_fd); +if (IS_ERR (link)) +return PTR_ERR (link); + + err =bpf_iter_new_fd (link); +bpf_link_put (link); + +return err; +} + SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size) {union bpf_attr attr;
@@ -4068,6 +4091,9 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, sizcase BPF_ENABLE_STATS : err =bpf_enable_stats (&attr);break ; +case BPF_ITER_CREATE : + err =bpf_iter_create (&attr); +break ;default : err = -EINVAL ;break ;
tools/include/uapi/linux/bpf.h +6 -0
@@ -116,6 +116,7 @@ enum bpf_cmd {BPF_LINK_GET_FD_BY_ID ,BPF_LINK_GET_NEXT_ID ,BPF_ENABLE_STATS , + BPF_ITER_CREATE, }; enum bpf_map_type {
@@ -614,6 +615,11 @@ union bpf_attr {__u32 type; } enable_stats; +struct {/* struct used by BPF_ITER_CREATE command */ +__u32 link_fd; +__u32 flags; + } iter_create; + } __attribute__((aligned(8)));/* The description below is an attempt at providing documentation to eBPF
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 06/21] bpf: create anonymous bpf iterator
Quoted 21 lines · Yonghong Song
A new bpf command
BPF_ITER_CREATEis added.The anonymous bpf iterator is
seq_filebased.
Theseq_fileprivate data are referenced by targets. Thebpf_iterinfrastructure allocated additional space atseq_file->privatebefore the space used by targets to store some meta data, e.g.,
prog: prog to run
session_id: an unique id for each openedseq_file
seq_num: how many times bpf programs are queried in this session
done_stop: an internal state to decide whether bpf programshould be called in seq_ops->stop() or notThe
seq_numwill start from 0 for valid objects. The bpf program may see the sameseq_nummore than once if
-seq_filebuffer overflow happens and the same object is retried bybpf_seq_read(), or
- the bpf program explicitly requests a retry of the same object
Quoted 8 lines
Since module is not supported for
bpf_iter, all target registeration happens at__inittime, so there is no need to changebpf_iter_unreg_target()as it is used mostly in error path of the init function at which time no bpf iterators have been created yet.Signed-off-by Yonghong Song <[email protected]>---
LGTM.
Quoted 7 lines
include/linux/bpf.h | 1 + include/uapi/linux/bpf.h | 6 ++ kernel/bpf/bpf_iter.c | 129 +++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 26 +++++++ tools/include/uapi/linux/bpf.h | 6 ++ 5 files changed, 168 insertions(+)
[...]
PATCH v3 7/21 bpf: create file bpf iterator
In reply to Yonghong SongTo produce a file bpf iterator, the fd must be corresponding to a link_fd assocciated with a trace/iter program. When the pinned file is opened, a seq_file will be generated.
--- include/linux/bpf.h | 2 ++ kernel/bpf/bpf_iter.c | 17 ++++++++++++++++- kernel/bpf/inode.c | 5 ++++- 3 files changed, 22 insertions(+), 2 deletions(-)
include/linux/bpf.h +2 -0
@@ -1022,6 +1022,7 @@ static inline void bpf_enable_instrumentation(void)extern const struct file_operations bpf_map_fops;extern const struct file_operations bpf_prog_fops; +extern const struct file_operations bpf_iter_fops;#define BPF_PROG_TYPE (_id, _name, prog_ctx_type, kern_ctx_type) \ extern const struct bpf_prog_ops _name ## _prog_ops; \
@@ -1145,6 +1146,7 @@ void bpf_iter_unreg_target(const char *target);bool bpf_iter_prog_supported (struct bpf_prog *prog);int bpf_iter_link_attach (const union bpf_attr *attr,struct bpf_prog *prog);int bpf_iter_new_fd (struct bpf_link *link); +bool bpf_link_is_iter (struct bpf_link *link);int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value);int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value);
kernel/bpf/bpf_iter.c +16 -1
@@ -39,6 +39,8 @@ static DEFINE_MUTEX(link_mutex);/* incremented on every opened seq_file */ static atomic64_t session_id; +static int prepare_seq_file (struct file *file,struct bpf_iter_link *link); +/* bpf_seq_read, a customized and simpler version for bpf iterator. * no_llseek is assumed for this file. * The following are differences from seq_read():
@@ -157,6 +159,13 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size,return copied; } +static int iter_open (struct inode *inode,struct file *file) +{ +struct bpf_iter_link *link = inode->i_private; + +return prepare_seq_file (file, link); +} +static int iter_release (struct inode *inode,struct file *file) {struct bpf_iter_priv_data *iter_priv;
@@ -178,7 +187,8 @@ static int iter_release(struct inode *inode, struct file *file)return seq_release_private (inode, file); } -static const struct file_operations bpf_iter_fops = { +const struct file_operations bpf_iter_fops = { + .open = iter_open, .llseek = no_llseek, .read = bpf_seq_read, .release = iter_release,
@@ -305,6 +315,11 @@ static const struct bpf_link_ops bpf_iter_link_lops = { .update_prog = bpf_iter_link_replace, }; +bool bpf_link_is_iter (struct bpf_link *link) +{ +return link->ops == &bpf_iter_link_lops; +} +int bpf_iter_link_attach (const union bpf_attr *attr,struct bpf_prog *prog) {struct bpf_link_primer link_primer;
kernel/bpf/inode.c +4 -1
@@ -358,8 +358,11 @@ static int bpf_mkmap(struct dentry *dentry, umode_t mode, void *arg)static int bpf_mklink (struct dentry *dentry,umode_t mode,void *arg) { +struct bpf_link *link = arg; +return bpf_mkobj_ops (dentry, mode, arg, &bpf_link_iops, - &bpffs_obj_fops); +bpf_link_is_iter (link) ? + &bpf_iter_fops : &bpffs_obj_fops); } static struct dentry *
Signature · 1 line
2.24.1
PATCH v3 8/21 bpf: implement common macros/helpers for target iterators
In reply to Yonghong SongMacro DEFINE_BPF_ITER_FUNC is implemented so target can define an init function to capture the BTF type which represents the target.
The bpf_iter_meta is a structure holding meta data, common to all targets in the bpf program.
Additional marker functions are called before or after bpf_seq_read() show()/next()/stop() callback functions to help calculate precise seq_num and whether call bpf_prog inside stop().
Two functions, bpf_iter_get_info() and bpf_iter_run_prog(), are implemented so target can get needed information from bpf_iter infrastructure and can run the program.
--- include/linux/bpf.h | 11 ++++++ kernel/bpf/bpf_iter.c | 86 ++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 92 insertions(+), 5 deletions(-)
include/linux/bpf.h +11 -0
@@ -1129,6 +1129,9 @@ int bpf_obj_pin_user(u32 ufd, const char __user *pathname);int bpf_obj_get_user (const char __user *pathname,int flags);#define BPF_ITER_FUNC_PREFIX "__bpf_iter__" +#define DEFINE_BPF_ITER_FUNC(target, args...) \ + extern int __bpf_iter__ ## target(args); \ + int __init __bpf_iter__ ## target(args) { return 0; }typedef int (*bpf_iter_init_seq_priv_t )(void *private_data);typedef void (*bpf_iter_fini_seq_priv_t )(void *private_data);
@@ -1141,12 +1144,20 @@ struct bpf_iter_reg {u32 seq_priv_size; }; +struct bpf_iter_meta { + __bpf_md_ptr(struct seq_file *, seq); +u64 session_id; +u64 seq_num; +}; +int bpf_iter_reg_target (struct bpf_iter_reg *reg_info);void bpf_iter_unreg_target (const char *target);bool bpf_iter_prog_supported (struct bpf_prog *prog);int bpf_iter_link_attach (const union bpf_attr *attr,struct bpf_prog *prog);int bpf_iter_new_fd (struct bpf_link *link);bool bpf_link_is_iter (struct bpf_link *link); +struct bpf_prog *bpf_iter_get_info (struct bpf_iter_meta *meta,bool in_stop); +int bpf_iter_run_prog (struct bpf_prog *prog,void *ctx);int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value);int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value);
kernel/bpf/bpf_iter.c +81 -5
118 lines not on this page · raw message
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 08/21] bpf: implement common macros/helpers for target iterators
Quoted 17 lines · Yonghong Song
Macro
DEFINE_BPF_ITER_FUNCis implemented so target can define an init function to capture the BTF type which represents the target.The
bpf_iter_metais a structure holding meta data, common to all targets in the bpf program.Additional marker functions are called before or after
bpf_seq_read()show()/next()/stop() callback functions to help calculate preciseseq_numand whether callbpf_proginsidestop().Two functions,
bpf_iter_get_info()andbpf_iter_run_prog(), are implemented so target can get needed information frombpf_iterinfrastructure and can run the program.
Quoted 6 lines
Signed-off-by Yonghong Song <[email protected]>--- include/linux/bpf.h | 11 ++++++ kernel/bpf/bpf_iter.c | 86 ++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 92 insertions(+), 5 deletions(-)
Looks good. I was worried about re-using seq_num when element is skipped, but this could already happen that same seq_num is associated with different objects: overflow + retry returns different object (because iteration is not a snapshot, so the element could be gone on retry). Both cases will have to be handled in about the same fashion, so it's fine.
Hm... Could this be a problem for start() implementation? E.g., if object is still there, but iterator wants to skip it permanently. Re-using seq_num will mean that start() will keep trying to fetch same to-be-skipped element? Not sure, please think about it, but we can fix it up later, if necessary.
[...]
kernel/bpf/bpf_iter.c static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, · line 147
Quoted 6 lines
kernel/bpf/bpf_iter.c +2
-0@@ -112,11 +143,16 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, err =PTR_ERR (p);break ; } + +/* get a valid next object, increase seq_num */
typo: get -> got
Quoted 11 lines
kernel/bpf/bpf_iter.c +3
-0+bpf_iter_inc_seq_num (seq); +if (seq->count >= size)break ; err = seq->op->show (seq, p);if (err >0 ) { +bpf_iter_dec_seq_num (seq); seq->count = offs; }else if (err <0 ||seq_has_overflowed (seq)) { seq->count = offs;
[...]
Re: [PATCH bpf-next v3 08/21] bpf: implement common macros/helpers for target iterators
Quoted 25 lines · Andrii Nakryiko
Quoted 17 lines · Yonghong Song
Macro
DEFINE_BPF_ITER_FUNCis implemented so target can define an init function to capture the BTF type which represents the target.The
bpf_iter_metais a structure holding meta data, common to all targets in the bpf program.Additional marker functions are called before or after
bpf_seq_read()show()/next()/stop() callback functions to help calculate preciseseq_numand whether callbpf_proginsidestop().Two functions,
bpf_iter_get_info()andbpf_iter_run_prog(), are implemented so target can get needed information frombpf_iterinfrastructure and can run the program.Quoted 6 lines
Signed-off-by Yonghong Song <[email protected]>--- include/linux/bpf.h | 11 ++++++ kernel/bpf/bpf_iter.c | 86 ++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 92 insertions(+), 5 deletions(-)
Quoted 6 lines
Looks good. I was worried about re-using
seq_numwhen element is skipped, but this could already happen that sameseq_numis associated with different objects: overflow + retry returns different object (because iteration is not a snapshot, so the element could be gone on retry). Both cases will have to be handled in about the same fashion, so it's fine.
This is what I thought as well.
Quoted 6 lines
Hm... Could this be a problem for
start()implementation? E.g., if object is still there, but iterator wants to skip it permanently. Re-usingseq_numwill mean thatstart()will keep trying to fetch same to-be-skipped element? Not sure, please think about it, but we can fix it up later, if necessary.
The seq_num is for bpf_program context. It does not affect how start() behaves. The start() MAY retry the same element over and over again if show() overflows or returns <0, but in which case, user space should check the return error code to decide to retry or give up.
Quoted 5 lines
Acked-by Andrii Nakryiko <[email protected]>[...]
Quoted 8 lines
kernel/bpf/bpf_iter.c static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, · line 147
Quoted 6 lines
kernel/bpf/bpf_iter.c +2
-0@@ -112,11 +143,16 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, err =PTR_ERR (p);break ; } + +/* get a valid next object, increase seq_num */ typo: get -> got
Ack.
Quoted 15 lines
Quoted 12 lines
kernel/bpf/bpf_iter.c +3
-0+bpf_iter_inc_seq_num (seq); +if (seq->count >= size)break ; err = seq->op->show (seq, p);if (err >0 ) { +bpf_iter_dec_seq_num (seq); seq->count = offs; }else if (err <0 ||seq_has_overflowed (seq)) { seq->count = offs;[...]
Re: [PATCH bpf-next v3 08/21] bpf: implement common macros/helpers for target iterators
Quoted 45 lines · Yonghong Song
Quoted 25 lines · Andrii Nakryiko
Quoted 17 lines · Yonghong Song
Macro
DEFINE_BPF_ITER_FUNCis implemented so target can define an init function to capture the BTF type which represents the target.The
bpf_iter_metais a structure holding meta data, common to all targets in the bpf program.Additional marker functions are called before or after
bpf_seq_read()show()/next()/stop() callback functions to help calculate preciseseq_numand whether callbpf_proginsidestop().Two functions,
bpf_iter_get_info()andbpf_iter_run_prog(), are implemented so target can get needed information frombpf_iterinfrastructure and can run the program.Quoted 6 lines
Signed-off-by Yonghong Song <[email protected]>--- include/linux/bpf.h | 11 ++++++ kernel/bpf/bpf_iter.c | 86 ++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 92 insertions(+), 5 deletions(-)Quoted 6 lines
Looks good. I was worried about re-using
seq_numwhen element is skipped, but this could already happen that sameseq_numis associated with different objects: overflow + retry returns different object (because iteration is not a snapshot, so the element could be gone on retry). Both cases will have to be handled in about the same fashion, so it's fine.This is what I thought as well.
Quoted 6 lines
Hm... Could this be a problem for
start()implementation? E.g., if object is still there, but iterator wants to skip it permanently. Re-usingseq_numwill mean thatstart()will keep trying to fetch same to-be-skipped element? Not sure, please think about it, but we can fix it up later, if necessary.
Quoted 4 lines
The
seq_numis forbpf_programcontext. It does not affect howstart()behaves. Thestart()MAY retry the same element over and over again ifshow()overflows or returns <0, but in which case, user space should check the return error code to decide to retry or give up.
ah, right seq_num vs internal id, makes sense, never mind :)
Quoted 32 lines
Quoted 6 lines
Acked-by Andrii Nakryiko <[email protected]>[...]
Quoted 8 lines
kernel/bpf/bpf_iter.c static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, · line 147
Quoted 6 lines
kernel/bpf/bpf_iter.c +2
-0@@ -112,11 +143,16 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, err =PTR_ERR (p);break ; } + +/* get a valid next object, increase seq_num */ typo: get -> got
Ack.
Quoted 15 lines
Quoted 12 lines
kernel/bpf/bpf_iter.c +3
-0+bpf_iter_inc_seq_num (seq); +if (seq->count >= size)break ; err = seq->op->show (seq, p);if (err >0 ) { +bpf_iter_dec_seq_num (seq); seq->count = offs; }else if (err <0 ||seq_has_overflowed (seq)) { seq->count = offs;[...]
PATCH v3 9/21 bpf: add bpf_map iterator
In reply to Yonghong SongImplement seq_file operations to traverse all maps.
--- include/linux/bpf.h | 1 + kernel/bpf/Makefile | 2 +- kernel/bpf/map_iter.c | 97 +++++++++++++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 19 +++++++++ 4 files changed, 118 insertions(+), 1 deletion(-) create mode 100644 kernel/bpf/map_iter.c
include/linux/bpf.h +1 -0
@@ -1082,6 +1082,7 @@ int generic_map_update_batch(struct bpf_map *map,int generic_map_delete_batch (struct bpf_map *map,const union bpf_attr *attr,union bpf_attr __user *uattr); +struct bpf_map *bpf_map_get_curr_or_next (u32 *id);extern int sysctl_unprivileged_bpf_disabled;
kernel/bpf/Makefile +1 -1
@@ -2,7 +2,7 @@ obj-y := core.o CFLAGS_core.o += $(call cc-disable-warning, override-init) -obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o +obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o
kernel/bpf/map_iter.c new file +97 -0
97 lines not on this page · raw message
kernel/bpf/syscall.c +19 -0
@@ -2934,6 +2934,25 @@ static int bpf_obj_get_next_id(const union bpf_attr *attr,return err; } +struct bpf_map *bpf_map_get_curr_or_next (u32 *id) +{ +struct bpf_map *map; + +spin_lock_bh (&map_idr_lock); +again: + map =idr_get_next (&map_idr, id); +if (map) { + map =__bpf_map_inc_not_zero (map, false); +if (IS_ERR (map)) { + (*id)++; + goto again; + } + } +spin_unlock_bh (&map_idr_lock); + +return map; +} +#define BPF_PROG_GET_FD_BY_ID_LAST_FIELD prog_idstruct bpf_prog *bpf_prog_by_id (u32 id)
Signature · 1 line
2.24.1
PATCH v3 10/21 net: bpf: add netlink and ipv6_route bpf_iter targets
In reply to Yonghong SongThis patch added netlink and ipv6_route targets, using the same seq_ops (except show() and minor changes for stop()) for /proc/net/{netlink,ipv6_route}.
The net namespace for these targets are the current net namespace at file open stage, similar to /proc/net/{netlink,ipv6_route} reference counting the net namespace at seq_file open stage.
Since module is not supported for now, ipv6_route is supported only if the IPV6 is built-in, i.e., not compiled as a module. The restriction can be lifted once module is properly supported for bpf_iter.
--- fs/proc/proc_net.c | 19 +++++++++ include/linux/proc_fs.h | 3 ++ net/ipv6/ip6_fib.c | 65 +++++++++++++++++++++++++++++- net/ipv6/route.c | 37 +++++++++++++++++ net/netlink/af_netlink.c | 87 +++++++++++++++++++++++++++++++++++++++- 5 files changed, 207 insertions(+), 4 deletions(-)
fs/proc/proc_net.c +19 -0
@@ -98,6 +98,25 @@ static const struct proc_ops proc_net_seq_ops = { .proc_release = seq_release_net, }; +int bpf_iter_init_seq_net (void *priv_data) +{ +#ifdef CONFIG_NET_NS +struct seq_net_private *p = priv_data; + + p->net =get_net (current->nsproxy->net_ns); +#endif +return 0 ; +} + +void bpf_iter_fini_seq_net (void *priv_data) +{ +#ifdef CONFIG_NET_NS +struct seq_net_private *p = priv_data; + +put_net (p->net); +#endif +} +struct proc_dir_entry *proc_create_net_data (const char *name,umode_t mode,struct proc_dir_entry *parent,const struct seq_operations *ops,unsigned int state_size,void *data)
include/linux/proc_fs.h +3 -0
@@ -105,6 +105,9 @@ struct proc_dir_entry *proc_create_net_single_write(const char *name, umode_t mo void *data);extern struct pid *tgid_pidfd_to_pid (const struct file *file); +extern int bpf_iter_init_seq_net (void *priv_data); +extern void bpf_iter_fini_seq_net (void *priv_data); +#ifdef CONFIG_PROC_PID_ARCH_STATUS /* * The architecture which selects CONFIG_PROC_PID_ARCH_STATUS must
net/ipv6/ip6_fib.c +63 -2
83 lines not on this page · raw message
net/ipv6/route.c +37 -0
@@ -6393,6 +6393,30 @@ void __init ip6_route_init_special_entries(void)#endif } +#if IS_BUILTIN (CONFIG_IPV6 ) +#if defined(CONFIG_BPF_SYSCALL ) && defined(CONFIG_PROC_FS ) +DEFINE_BPF_ITER_FUNC(ipv6_route, struct bpf_iter_meta *meta, struct fib6_info *rt) + +static int __init bpf_iter_register (void ) +{ +struct bpf_iter_reg reg_info = { + .target ="ipv6_route" , + .seq_ops = &ipv6_route_seq_ops, + .init_seq_private = bpf_iter_init_seq_net, + .fini_seq_private = bpf_iter_fini_seq_net, + .seq_priv_size =sizeof (struct ipv6_route_iter ), + }; + +return bpf_iter_reg_target (®_info); +} + +static void bpf_iter_unregister (void ) +{ +bpf_iter_unreg_target ("ipv6_route" ); +} +#endif +#endif +int __init ip6_route_init (void ) {int ret;
@@ -6455,6 +6479,14 @@ int __init ip6_route_init(void)if (ret) goto out_register_late_subsys; +#if IS_BUILTIN (CONFIG_IPV6 ) +#if defined(CONFIG_BPF_SYSCALL ) && defined(CONFIG_PROC_FS ) + ret =bpf_iter_register (); +if (ret) + goto out_register_late_subsys; +#endif +#endif +for_each_possible_cpu (cpu) {struct uncached_list *ul =per_cpu_ptr (&rt6_uncached_list, cpu);
@@ -6487,6 +6519,11 @@ int __init ip6_route_init(void)void ip6_route_cleanup (void ) { +#if IS_BUILTIN (CONFIG_IPV6 ) +#if defined(CONFIG_BPF_SYSCALL ) && defined(CONFIG_PROC_FS ) +bpf_iter_unregister (); +#endif +#endif unregister_netdevice_notifier (&ip6_route_dev_notifier);unregister_pernet_subsys (&ip6_route_net_late_ops);fib6_rules_cleanup ();
net/netlink/af_netlink.c +85 -2
117 lines not on this page · raw message
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 10/21] net: bpf: add netlink and ipv6_route bpf_iter targets
Quoted 10 lines · Yonghong Song
This patch added netlink and
ipv6_routetargets, using the sameseq_ops(exceptshow()and minor changes forstop()) for/proc/net/{netlink,ipv6_route}.The net namespace for these targets are the current net namespace at file open stage, similar to
/proc/net/{netlink,ipv6_route}reference counting the net namespace atseq_fileopen stage.
Quoted 7 lines
Since module is not supported for now,
ipv6_routeis supported only if the IPV6 is built-in, i.e., not compiled as a module. The restriction can be lifted once module is properly supported forbpf_iter.Signed-off-by Yonghong Song <[email protected]>---
Looks correct.
Quoted 7 lines
fs/proc/proc_net.c | 19 +++++++++ include/linux/proc_fs.h | 3 ++ net/ipv6/ip6_fib.c | 65 +++++++++++++++++++++++++++++- net/ipv6/route.c | 37 +++++++++++++++++ net/netlink/af_netlink.c | 87 +++++++++++++++++++++++++++++++++++++++- 5 files changed, 207 insertions(+), 4 deletions(-)
[...]
Quoted 18 lines
net/ipv6/route.c +10
-0@@ -6393,6 +6393,30 @@ void __init ip6_route_init_special_entries(void)#endif } +#if IS_BUILTIN(CONFIG_IPV6) +#if defined(CONFIG_BPF_SYSCALL) && defined(CONFIG_PROC_FS) +DEFINE_BPF_ITER_FUNC(ipv6_route, struct bpf_iter_meta *meta, struct fib6_info *rt) + +static int __init bpf_iter_register(void) +{ + struct bpf_iter_reg reg_info = { + .target = "ipv6_route", + .seq_ops = &ipv6_route_seq_ops, + .init_seq_private = bpf_iter_init_seq_net,
net/ipv6/route.c void __init ip6_route_init_special_entries(void) · line 6415
Quoted 10 lines
net/ipv6/route.c +10
-0+ .fini_seq_private = bpf_iter_fini_seq_net, + .seq_priv_size = sizeof(struct ipv6_route_iter), + }; + +return bpf_iter_reg_target (®_info); +} + +static void bpf_iter_unregister (void ) +{ +bpf_iter_unreg_target ("ipv6_route" );
Nit. This string duplication is unfortunate. If bpf_iter_unreg_target took same struct bpf_iter_ret * as bpf_iter_reg_target(), it would be symmetrical and not dependent on magic strings anymore. That reg_info struct would just be static const struct global variable passed to both register/unregister.
Quoted 4 lines
net/ipv6/route.c +4
-0+} +#endif +#endif +
[...]
PATCH v3 11/21 bpf: add task and task/file iterator targets
In reply to Yonghong SongOnly the tasks belonging to "current" pid namespace are enumerated.
For task/file target, the bpf program will have access to
struct task_struct *task
u32 fd
struct file *file
where fd/file is an open file for the task.
--- kernel/bpf/Makefile | 2 +- kernel/bpf/task_iter.c | 332 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 333 insertions(+), 1 deletion(-) create mode 100644 kernel/bpf/task_iter.c
kernel/bpf/Makefile +1 -1
@@ -2,7 +2,7 @@ obj-y := core.o CFLAGS_core.o += $(call cc-disable-warning, override-init) -obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o +obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o task_iter.o obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o
kernel/bpf/task_iter.c new file +332 -0
332 lines not on this page · raw message
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 11/21] bpf: add task and task/file iterator targets
Quoted 10 lines · Yonghong Song
Only the tasks belonging to "current" pid namespace are enumerated.
For task/file target, the bpf program will have access to
structtask_struct*task
u32 fd
struct file *file
where fd/file is an open file for the task.
Quoted 7 lines
Signed-off-by Yonghong Song <[email protected]>--- kernel/bpf/Makefile | 2 +- kernel/bpf/task_iter.c | 332 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 333 insertions(+), 1 deletion(-) create mode 100644 kernel/bpf/task_iter.c
Looks good, though I'd just use bpf_iter_seq_task_file_info directly, given it is a state of iterator by design. Probably would simplify code a bit as well. But either way I can't find any bug in the code, so:
[...]
kernel/bpf/task_iter.c line 18
Quoted 9 lines
kernel/bpf/task_iter.c new file +9
-0+ +struct bpf_iter_seq_task_common { +struct pid_namespace *ns; +}; + +struct bpf_iter_seq_task_info { +/* The first field must be struct bpf_iter_seq_task_common. +* this is assumed by {init, fini}_seq_pidns() callback functions. +*/
Awesome, thanks!
Quoted 4 lines
kernel/bpf/task_iter.c new file +4
-0+struct bpf_iter_seq_task_common common; +u32 tid; +}; +
[...]
Review
Quoted 2 lines
(quoted diff) +2
-0+ +static void *task_seq_next(struct seq_file *seq, void *v, loff_t *pos)
nit: v is very non-descriptive name, prev_task?
Quoted 12 lines
kernel/bpf/task_iter.c new file +12
-0+{ +struct bpf_iter_seq_task_info *info = seq->private; +struct task_struct *task; + + ++*pos; + ++info->tid; +put_task_struct ((struct task_struct *)v); + task =task_seq_get_next (info->common.ns, &info->tid); +if (!task) +return NULL ; + +return task;
kernel/bpf/task_iter.c line 73
Quoted 10 lines
kernel/bpf/task_iter.c new file +10
-0+} + +struct bpf_iter__task { + __bpf_md_ptr(struct bpf_iter_meta *, meta); + __bpf_md_ptr(struct task_struct *, task); +}; + +DEFINE_BPF_ITER_FUNC(task, struct bpf_iter_meta *meta, struct task_struct *task) + +static int __task_seq_show (struct seq_file *seq,void *v,bool in_stop)
same nit: v -> task? Can also specify struct task_struct * type, same for above.
[...]
Quoted 5 lines
kernel/bpf/task_iter.c new file +5
-0+ +struct bpf_iter_seq_task_file_info { +/* The first field must be struct bpf_iter_seq_task_common. +* this is assumed by {init, fini}_seq_pidns() callback functions. +*/
kernel/bpf/task_iter.c line 123
Quoted 10 lines
kernel/bpf/task_iter.c new file +10
-0+struct bpf_iter_seq_task_common common; +struct task_struct *task; +struct files_struct *files; +u32 tid; +u32 fd; +}; + +static struct file *task_file_seq_get_next (struct pid_namespace *ns,u32 *tid, +int *fd,struct task_struct **task, +struct files_struct **fstruct)
Why not just passing struct bpf_iter_seq_task_file_info* directly, instead of these 5 individual pointers?
Quoted 10 lines
kernel/bpf/task_iter.c new file +10
-0+{ +struct files_struct *curr_files; +struct task_struct *curr_task; +u32 curr_tid = *tid, max_fds; +int curr_fd = *fd; + +/* If this function returns a non-NULL file object, +* it held a reference to the task/files_struct/file. +* Otherwise, it does not hold any reference. +*/
[...]
PATCH v3 12/21 bpf: add PTR_TO_BTF_ID_OR_NULL support
In reply to Yonghong SongAdd bpf_reg_type PTR_TO_BTF_ID_OR_NULL support. For tracing/iter program, the bpf program context definition, e.g., for previous bpf_map target, looks like
struct bpf_iter__bpf_map {
struct bpf_iter_meta *meta;
struct bpf_map *map;};
The kernel guarantees that meta is not NULL, but map pointer maybe NULL. The NULL map indicates that all objects have been traversed, so bpf program can take proper action, e.g., do final aggregation and/or send final report to user space.
Add btf_id_or_null_non0_off to prog->aux structure, to indicate that if the context access offset is not 0, set to PTR_TO_BTF_ID_OR_NULL instead of PTR_TO_BTF_ID. This bit is set for tracing/iter program.
--- include/linux/bpf.h | 2 ++ kernel/bpf/btf.c | 5 ++++- kernel/bpf/verifier.c | 16 ++++++++++++---- 3 files changed, 18 insertions(+), 5 deletions(-)
include/linux/bpf.h +2 -0
@@ -320,6 +320,7 @@ enum bpf_reg_type {PTR_TO_TP_BUFFER ,/* reg points to a writable raw tp's buffer */ PTR_TO_XDP_SOCK ,/* reg points to struct xdp_sock */ PTR_TO_BTF_ID ,/* reg points to kernel struct */ + PTR_TO_BTF_ID_OR_NULL, /* reg points to kernel struct or NULL */ };/* The information passed from prog-specific *_is_valid_access
@@ -658,6 +659,7 @@ struct bpf_prog_aux {bool offload_requested;bool attach_btf_trace;/* true if attaching to BTF-enabled raw tp */ bool func_proto_unreliable; +bool btf_id_or_null_non0_off;enum bpf_tramp_prog_type trampoline_prog_type;struct bpf_trampoline *trampoline;struct hlist_node tramp_hlist;
kernel/bpf/btf.c +4 -1
@@ -3790,7 +3790,10 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,return true;/* this is a pointer to another type */ - info->reg_type =PTR_TO_BTF_ID ; +if (off !=0 && prog->aux->btf_id_or_null_non0_off) + info->reg_type =PTR_TO_BTF_ID_OR_NULL ; +else + info->reg_type =PTR_TO_BTF_ID ;if (tgt_prog) { ret =btf_translate_to_vmlinux (log, btf, t, tgt_prog->type, arg);
kernel/bpf/verifier.c +12 -4
@@ -398,7 +398,8 @@ static bool reg_type_may_be_null(enum bpf_reg_type type)return type ==PTR_TO_MAP_VALUE_OR_NULL || type ==PTR_TO_SOCKET_OR_NULL || type ==PTR_TO_SOCK_COMMON_OR_NULL || - type ==PTR_TO_TCP_SOCK_OR_NULL ; + type ==PTR_TO_TCP_SOCK_OR_NULL || + type ==PTR_TO_BTF_ID_OR_NULL ; }static bool reg_may_point_to_spin_lock (const struct bpf_reg_state *reg)
@@ -483,6 +484,7 @@ static const char * const reg_type_str[] = { [PTR_TO_TP_BUFFER] = "tp_buffer", [PTR_TO_XDP_SOCK] = "xdp_sock", [PTR_TO_BTF_ID] = "ptr_", + [PTR_TO_BTF_ID_OR_NULL] = "ptr_or_null_", };static char slot_type_char[] = {
@@ -543,7 +545,7 @@ static void print_verifier_state(struct bpf_verifier_env *env,/* reg->off should be 0 for SCALAR_VALUE */ verbose (env,"%lld" , reg->var_off.value + reg->off); }else { -if (t ==PTR_TO_BTF_ID ) +if (t ==PTR_TO_BTF_ID || t ==PTR_TO_BTF_ID_OR_NULL )verbose (env,"%s" ,kernel_type_name (reg->btf_id));verbose (env,"(id=%d" , reg->id);if (reg_type_may_be_refcounted_or_null (t))
@@ -2139,6 +2141,7 @@ static bool is_spillable_regtype(enum bpf_reg_type type)case PTR_TO_TCP_SOCK_OR_NULL :case PTR_TO_XDP_SOCK :case PTR_TO_BTF_ID : +case PTR_TO_BTF_ID_OR_NULL :return true;default :return false;
@@ -2659,7 +2662,7 @@ static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int off,*/ *reg_type = info.reg_type; -if (*reg_type ==PTR_TO_BTF_ID ) +if (*reg_type ==PTR_TO_BTF_ID || *reg_type ==PTR_TO_BTF_ID_OR_NULL ) *btf_id = info.btf_id;else env->insn_aux_data[insn_idx].ctx_field_size = info.ctx_field_size;
@@ -3243,7 +3246,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn* a sub-register. */ regs[value_regno].subreg_def =DEF_NOT_SUBREG ; -if (reg_type ==PTR_TO_BTF_ID ) +if (reg_type ==PTR_TO_BTF_ID || + reg_type ==PTR_TO_BTF_ID_OR_NULL ) regs[value_regno].btf_id = btf_id; } regs[value_regno].type = reg_type;
@@ -6572,6 +6576,8 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state, reg->type =PTR_TO_SOCK_COMMON ; }else if (reg->type ==PTR_TO_TCP_SOCK_OR_NULL ) { reg->type =PTR_TO_TCP_SOCK ; + }else if (reg->type ==PTR_TO_BTF_ID_OR_NULL ) { + reg->type =PTR_TO_BTF_ID ; }if (is_null) {/* We don't need id and ref_obj_id from this point
@@ -8429,6 +8435,7 @@ static bool reg_type_mismatch_ok(enum bpf_reg_type type)case PTR_TO_TCP_SOCK_OR_NULL :case PTR_TO_XDP_SOCK :case PTR_TO_BTF_ID : +case PTR_TO_BTF_ID_OR_NULL :return false;default :return true;
@@ -10640,6 +10647,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) prog->aux->attach_func_proto = t;if (!bpf_iter_prog_supported (prog))return -EINVAL ; + prog->aux->btf_id_or_null_non0_off = true; ret =btf_distill_func_proto (&env->log, btf, t, tname, &fmodel);return ret;
Signature · 1 line
2.24.1
PATCH v3 13/21 bpf: add bpf_seq_printf and bpf_seq_write helpers
In reply to Yonghong SongTwo helpers bpf_seq_printf and bpf_seq_write, are added for writing data to the seq_file buffer.
bpf_seq_printf supports common format string flag/width/type fields so at least I can get identical results for netlink and ipv6_route targets.
For bpf_seq_printf and bpf_seq_write, return value -EOVERFLOW specifically indicates a write failure due to overflow, which means the object will be repeated in the next bpf invocation if object collection stays the same. Note that if the object collection is changed, depending how collection traversal is done, even if the object still in the collection, it may not be visited.
bpf_seq_printf may return -EBUSY meaning that internal percpu buffer for memory copy of strings or other pointees is not available. Bpf program can return 1 to indicate it wants the same object to be repeated. Right now, this should not happen on no-RT kernels since migrate_disable(), which guards bpf prog call, calls preempt_disable().
--- include/uapi/linux/bpf.h | 32 +++++- kernel/trace/bpf_trace.c | 200 +++++++++++++++++++++++++++++++++ scripts/bpf_helpers_doc.py | 2 + tools/include/uapi/linux/bpf.h | 32 +++++- 4 files changed, 264 insertions(+), 2 deletions(-)
include/uapi/linux/bpf.h +31 -1
@@ -3076,6 +3076,34 @@ union bpf_attr {* See: clock_gettime(CLOCK_BOOTTIME) * Return * Current *ktime*. +* +* int bpf_seq_printf(struct seq_file *m, const char *fmt, u32 fmt_size, const void *data, u32 data_len) +* Description +* seq_printf uses seq_file seq_printf() to print out the format string. +* The *m* represents the seq_file. The *fmt* and *fmt_size* are for +* the format string itself. The *data* and *data_len* are format string +* arguments. The *data* are a u64 array and corresponding format string +* values are stored in the array. For strings and pointers where pointees +* are accessed, only the pointer values are stored in the *data* array. +* The *data_len* is the *data* size in term of bytes. +* Return +* 0 on success, or a negative errno in case of failure. +* +* * **-EBUSY** Percpu memory copy buffer is busy, can try again +* by returning 1 from bpf program. +* * **-EINVAL** Invalid arguments, or invalid/unsupported formats. +* * **-E2BIG** Too many format specifiers. +* * **-EOVERFLOW** Overflow happens, the same object will be tried again. +* +* int bpf_seq_write(struct seq_file *m, const void *data, u32 len) +* Description +* seq_write uses seq_file seq_write() to write the data. +* The *m* represents the seq_file. The *data* and *len* represent the +* data to write in bytes. +* Return +* 0 on success, or a negative errno in case of failure. +* +* * **-EOVERFLOW** Overflow happens, the same object will be tried again. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -3203,7 +3231,9 @@ union bpf_attr {FN (get_netns_cookie), \FN (get_current_ancestor_cgroup_id), \FN (sk_assign), \ -FN (ktime_get_boot_ns), +FN (ktime_get_boot_ns), \ +FN (seq_printf), \ +FN (seq_write),/* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call
kernel/trace/bpf_trace.c +200 -0
212 lines not on this page · raw message
scripts/bpf_helpers_doc.py +2 -0
@@ -414,6 +414,7 @@ class PrinterHelpers(Printer): 'struct sk_reuseport_md', 'struct sockaddr', 'struct tcphdr', + 'struct seq_file', 'struct __sk_buff', 'struct sk_msg_md',
@@ -450,6 +451,7 @@ class PrinterHelpers(Printer): 'struct sk_reuseport_md', 'struct sockaddr', 'struct tcphdr', + 'struct seq_file', } mapped_types = { 'u8': '__u8',
tools/include/uapi/linux/bpf.h +31 -1
@@ -3076,6 +3076,34 @@ union bpf_attr {* See: clock_gettime(CLOCK_BOOTTIME) * Return * Current *ktime*. +* +* int bpf_seq_printf(struct seq_file *m, const char *fmt, u32 fmt_size, const void *data, u32 data_len) +* Description +* seq_printf uses seq_file seq_printf() to print out the format string. +* The *m* represents the seq_file. The *fmt* and *fmt_size* are for +* the format string itself. The *data* and *data_len* are format string +* arguments. The *data* are a u64 array and corresponding format string +* values are stored in the array. For strings and pointers where pointees +* are accessed, only the pointer values are stored in the *data* array. +* The *data_len* is the *data* size in term of bytes. +* Return +* 0 on success, or a negative errno in case of failure. +* +* * **-EBUSY** Percpu memory copy buffer is busy, can try again +* by returning 1 from bpf program. +* * **-EINVAL** Invalid arguments, or invalid/unsupported formats. +* * **-E2BIG** Too many format specifiers. +* * **-EOVERFLOW** Overflow happens, the same object will be tried again. +* +* int bpf_seq_write(struct seq_file *m, const void *data, u32 len) +* Description +* seq_write uses seq_file seq_write() to write the data. +* The *m* represents the seq_file. The *data* and *len* represent the +* data to write in bytes. +* Return +* 0 on success, or a negative errno in case of failure. +* +* * **-EOVERFLOW** Overflow happens, the same object will be tried again. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -3203,7 +3231,9 @@ union bpf_attr {FN (get_netns_cookie), \FN (get_current_ancestor_cgroup_id), \FN (sk_assign), \ -FN (ktime_get_boot_ns), +FN (ktime_get_boot_ns), \ +FN (seq_printf), \ +FN (seq_write),/* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 13/21] bpf: add bpf_seq_printf and bpf_seq_write helpers
Quoted 23 lines · Yonghong Song
Two helpers
bpf_seq_printfandbpf_seq_write, are added for writing data to theseq_filebuffer.
bpf_seq_printfsupports common format string flag/width/type fields so at least I can get identical results for netlink andipv6_routetargets.For
bpf_seq_printfandbpf_seq_write, return value-EOVERFLOWspecifically indicates a write failure due to overflow, which means the object will be repeated in the next bpf invocation if object collection stays the same. Note that if the object collection is changed, depending how collection traversal is done, even if the object still in the collection, it may not be visited.
bpf_seq_printfmay return-EBUSYmeaning that internal percpu buffer for memory copy of strings or other pointees is not available. Bpf program can return 1 to indicate it wants the same object to be repeated. Right now, this should not happen on no-RT kernels sincemigrate_disable(), which guards bpf prog call, callspreempt_disable().
Quoted 8 lines
Signed-off-by Yonghong Song <[email protected]>--- include/uapi/linux/bpf.h | 32 +++++- kernel/trace/bpf_trace.c | 200 +++++++++++++++++++++++++++++++++ scripts/bpf_helpers_doc.py | 2 + tools/include/uapi/linux/bpf.h | 32 +++++- 4 files changed, 264 insertions(+), 2 deletions(-)
Was a bit surprised by behavior on failed memory read, I think it's important to emphasize and document this. But otherwise:
[...]
kernel/trace/bpf_trace.c const struct bpf_func_proto *bpf_get_trace_printk_proto(void) · line 554
Quoted 11 lines
kernel/trace/bpf_trace.c +11
-0+if (fmt[i] =='s' ) { +/* try our best to copy */ +if (memcpy_cnt >=MAX_SEQ_PRINTF_MAX_MEMCPY ) { + err = -E2BIG ; + goto out; + } + + bufs->buf[memcpy_cnt][0 ] =0 ; +strncpy_from_unsafe (bufs->buf[memcpy_cnt], + (void *) (long ) args[fmt_cnt], +MAX_SEQ_PRINTF_STR_LEN );
So the behavior is that we try to read string, but if it fails, we treat it as empty string? That needs to be documented, IMHO. My expectation was that entire printf would fail.
Same for pointers below, right?
Quoted 7 lines
kernel/trace/bpf_trace.c +7
-0+ params[fmt_cnt] = (u64 )(long )bufs->buf[memcpy_cnt]; + + fmt_cnt++; + memcpy_cnt++; +continue ; + } +
[...]
Re: [PATCH bpf-next v3 13/21] bpf: add bpf_seq_printf and bpf_seq_write helpers
Quoted 52 lines · Andrii Nakryiko
Quoted 23 lines · Yonghong Song
Two helpers
bpf_seq_printfandbpf_seq_write, are added for writing data to theseq_filebuffer.
bpf_seq_printfsupports common format string flag/width/type fields so at least I can get identical results for netlink andipv6_routetargets.For
bpf_seq_printfandbpf_seq_write, return value-EOVERFLOWspecifically indicates a write failure due to overflow, which means the object will be repeated in the next bpf invocation if object collection stays the same. Note that if the object collection is changed, depending how collection traversal is done, even if the object still in the collection, it may not be visited.
bpf_seq_printfmay return-EBUSYmeaning that internal percpu buffer for memory copy of strings or other pointees is not available. Bpf program can return 1 to indicate it wants the same object to be repeated. Right now, this should not happen on no-RT kernels sincemigrate_disable(), which guards bpf prog call, callspreempt_disable().Quoted 8 lines
Signed-off-by Yonghong Song <[email protected]>--- include/uapi/linux/bpf.h | 32 +++++- kernel/trace/bpf_trace.c | 200 +++++++++++++++++++++++++++++++++ scripts/bpf_helpers_doc.py | 2 + tools/include/uapi/linux/bpf.h | 32 +++++- 4 files changed, 264 insertions(+), 2 deletions(-)Was a bit surprised by behavior on failed memory read, I think it's important to emphasize and document this. But otherwise:
Acked-by Andrii Nakryiko <[email protected]>[...]
kernel/trace/bpf_trace.c const struct bpf_func_proto *bpf_get_trace_printk_proto(void) · line 554
Quoted 11 lines
kernel/trace/bpf_trace.c +11
-0+if (fmt[i] =='s' ) { +/* try our best to copy */ +if (memcpy_cnt >=MAX_SEQ_PRINTF_MAX_MEMCPY ) { + err = -E2BIG ; + goto out; + } + + bufs->buf[memcpy_cnt][0 ] =0 ; +strncpy_from_unsafe (bufs->buf[memcpy_cnt], + (void *) (long ) args[fmt_cnt], +MAX_SEQ_PRINTF_STR_LEN );
Quoted 3 lines
So the behavior is that we try to read string, but if it fails, we treat it as empty string? That needs to be documented, IMHO. My expectation was that entire printf would fail.
Let me return proper error. Currently, two possible errors may happen:
- user provide an invalid address, yes, an error should be returned and we should not do anything
- user provide a valid address, but it needs page fault happening to read the content. With current implementation, strncpy_from_unsafe will return fail. Future sleepable bpf program will help for this case, so an error means a real address error.
Quoted 13 lines
Same for pointers below, right?
Quoted 7 lines
kernel/trace/bpf_trace.c +7
-0+ params[fmt_cnt] = (u64 )(long )bufs->buf[memcpy_cnt]; + + fmt_cnt++; + memcpy_cnt++; +continue ; + } +[...]
Re: [PATCH bpf-next v3 13/21] bpf: add bpf_seq_printf and bpf_seq_write helpers
Quoted 60 lines · Yonghong Song
Quoted 53 lines · Andrii Nakryiko
Quoted 23 lines · Yonghong Song
Two helpers
bpf_seq_printfandbpf_seq_write, are added for writing data to theseq_filebuffer.
bpf_seq_printfsupports common format string flag/width/type fields so at least I can get identical results for netlink andipv6_routetargets.For
bpf_seq_printfandbpf_seq_write, return value-EOVERFLOWspecifically indicates a write failure due to overflow, which means the object will be repeated in the next bpf invocation if object collection stays the same. Note that if the object collection is changed, depending how collection traversal is done, even if the object still in the collection, it may not be visited.
bpf_seq_printfmay return-EBUSYmeaning that internal percpu buffer for memory copy of strings or other pointees is not available. Bpf program can return 1 to indicate it wants the same object to be repeated. Right now, this should not happen on no-RT kernels sincemigrate_disable(), which guards bpf prog call, callspreempt_disable().Quoted 8 lines
Signed-off-by Yonghong Song <[email protected]>---
include/uapi/linux/bpf.h | 32 +++++- kernel/trace/bpf_trace.c | 200 +++++++++++++++++++++++++++++++++ scripts/bpf_helpers_doc.py | 2 +tools/include/uapi/linux/bpf.h| 32 +++++-4 files changed, 264 insertions(+), 2 deletions(-)Was a bit surprised by behavior on failed memory read, I think it's important to emphasize and document this. But otherwise:
Acked-by Andrii Nakryiko <[email protected]>[...]
Quoted 4 lines
+ if (fmt[i] == 's') {
+ /* try our best to copy */
+ if (memcpy_cnt>=MAX_SEQ_PRINTF_MAX_MEMCPY) { + err =-E2BIG;Quoted 8 lines
+ goto out;
+ }
+
+ bufs->buf[memcpy_cnt][0] = 0;
+ strncpy_from_unsafe(bufs->buf[memcpy_cnt], + (void *) (long) args[fmt_cnt],
+MAX_SEQ_PRINTF_STR_LEN);Quoted 3 lines
So the behavior is that we try to read string, but if it fails, we treat it as empty string? That needs to be documented, IMHO. My expectation was that entire printf would fail.
Quoted 8 lines
Let me return proper error. Currently, two possible errors may happen:
- user provide an invalid address, yes, an error should be returned and we should not do anything
- user provide a valid address, but it needs page fault happening to read the content. With current implementation,strncpy_from_unsafewill return fail. Future sleepable bpf program will help for this case, so an error means a real address error.
It matches what bpf_trace_printk() is doing. I suggest to defer any improvements to later patches. Both should be consistent.
Re: [PATCH bpf-next v3 13/21] bpf: add bpf_seq_printf and bpf_seq_write helpers
Quoted 71 lines · Alexei Starovoitov
Quoted 61 lines · Yonghong Song
Quoted 54 lines · Andrii Nakryiko
Quoted 23 lines · Yonghong Song
Two helpers
bpf_seq_printfandbpf_seq_write, are added for writing data to theseq_filebuffer.
bpf_seq_printfsupports common format string flag/width/type fields so at least I can get identical results for netlink andipv6_routetargets.For
bpf_seq_printfandbpf_seq_write, return value-EOVERFLOWspecifically indicates a write failure due to overflow, which means the object will be repeated in the next bpf invocation if object collection stays the same. Note that if the object collection is changed, depending how collection traversal is done, even if the object still in the collection, it may not be visited.
bpf_seq_printfmay return-EBUSYmeaning that internal percpu buffer for memory copy of strings or other pointees is not available. Bpf program can return 1 to indicate it wants the same object to be repeated. Right now, this should not happen on no-RT kernels sincemigrate_disable(), which guards bpf prog call, callspreempt_disable().Quoted 9 lines
Signed-off-by Yonghong Song <[email protected]>---
include/uapi/linux/bpf.h | 32 +++++-
kernel/trace/bpf_trace.c | 200
+++++++++++++++++++++++++++++++++
scripts/bpf_helpers_doc.py | 2 +
tools/include/uapi/linux/bpf.h| 32 +++++-4 files changed, 264 insertions(+), 2 deletions(-)Was a bit surprised by behavior on failed memory read, I think it's important to emphasize and document this. But otherwise:
Acked-by Andrii Nakryiko <[email protected]>[...]
Quoted 4 lines
+ if (fmt[i] == 's') {
+ /* try our best to copy */
+ if (memcpy_cnt>=MAX_SEQ_PRINTF_MAX_MEMCPY) { + err =-E2BIG;Quoted 8 lines
+ goto out;
+ }
+
+ bufs->buf[memcpy_cnt][0] = 0;
+ strncpy_from_unsafe(bufs->buf[memcpy_cnt], + (void *) (long) args[fmt_cnt],
+MAX_SEQ_PRINTF_STR_LEN);Quoted 3 lines
So the behavior is that we try to read string, but if it fails, we treat it as empty string? That needs to be documented, IMHO. My expectation was that entire printf would fail.
Quoted 8 lines
Let me return proper error. Currently, two possible errors may happen:
- user provide an invalid address, yes, an error should be returned and we should not do anything
- user provide a valid address, but it needs page fault happening to read the content. With current implementation,strncpy_from_unsafewill return fail. Future sleepable bpf program will help for this case, so an error means a real address error.
Quoted 3 lines
It matches what
bpf_trace_printk()is doing. I suggest to defer any improvements to later patches. Both should be consistent.
Sure. We can do that.
PATCH v3 14/21 bpf: handle spilled PTR_TO_BTF_ID properly when checking stack_boundary
In reply to Yonghong SongThis specifically to handle the case like below:
// ptr below is a socket ptr identified by PTR_TO_BTF_ID u64 param[2] = { ptr, val };
bpf_seq_printf(seq, fmt, sizeof(fmt), param, sizeof(param));
In this case, the 16 bytes stack for "param" contains:
8 bytes for ptr with spilled PTR_TO_BTF_ID 8 bytes for val as STACK_MISC
The current verifier will complain the ptr should not be visible to the helper.
...
16: (7b) *(u64 *)(r10 -64) = r2
18: (7b) *(u64 *)(r10 -56) = r1
19: (bf) r4 = r10
;
20: (07) r4 += -64
; BPF_SEQ_PRINTF(seq, fmt1, (long)s, s->sk_protocol); 21: (bf) r1 = r6
22: (18) r2 = 0xffffa8d00018605a
24: (b4) w3 = 10
25: (b4) w5 = 16
26: (85) call bpf_seq_printf#125
R0=inv(id=0) R1_w=ptr_seq_file(id=0,off=0,imm=0)
R2_w=map_value(id=0,off=90,ks=4,vs=144,imm=0) R3_w=inv10
R4_w=fp-64 R5_w=inv16 R6=ptr_seq_file(id=0,off=0,imm=0)
R7=ptr_netlink_sock(id=0,off=0,imm=0) R10=fp0 fp-56_w=mmmmmmmm
fp-64_w=ptr_ last_idx 26 first_idx 13
regs=8 stack=0 before 25: (b4) w5 = 16 regs=8 stack=0 before 24: (b4) w3 = 10 invalid indirect read from stack off -64+0 size 16
--- kernel/bpf/verifier.c | 5 +++++ 1 file changed, 5 insertions(+)
kernel/bpf/verifier.c +5 -0
@@ -3494,6 +3494,11 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, *stype =STACK_MISC ; goto mark; } + +if (state->stack[spi].slot_type[0 ] ==STACK_SPILL && + state->stack[spi].spilled_ptr.type ==PTR_TO_BTF_ID ) + goto mark; +if (state->stack[spi].slot_type[0 ] ==STACK_SPILL && state->stack[spi].spilled_ptr.type ==SCALAR_VALUE ) {__mark_reg_unknown (env, &state->stack[spi].spilled_ptr);
Signature · 1 line
2.24.1
PATCH v3 15/21 bpf: support variable length array in tracing programs
In reply to Yonghong SongIn /proc/net/ipv6_route, we have
struct fib6_info {
struct fib6_table *fib6_table;
...
struct fib6_nh fib6_nh[0];
}
struct fib6_nh {
struct fib_nh_common nh_common;
struct rt6_info **rt6i_pcpu;
struct rt6_exception_bucket *rt6i_exception_bucket;
};
struct fib_nh_common {
...
u8 nhc_gw_family;
...}
The access:
struct fib6_nh *fib6_nh = &rt->fib6_nh;
... fib6_nh->nh_common.nhc_gw_family ...
This patch ensures such an access is handled properly.
--- kernel/bpf/btf.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
kernel/bpf/btf.c +37 -0
@@ -3833,6 +3833,7 @@ int btf_struct_access(struct bpf_verifier_log *log,const struct btf_type *mtype, *elem_type =NULL ;const struct btf_member *member;const char *tname, *mname; +u32 vlen; again: tname =__btf_name_by_offset (btf_vmlinux, t->name_off);
@@ -3841,7 +3842,43 @@ int btf_struct_access(struct bpf_verifier_log *log,return -EINVAL ; } + vlen =btf_type_vlen (t);if (off + size > t->size) { +/* If the last element is a variable size array, we may +* need to relax the rule. +*/ +struct btf_array *array_elem; + +if (vlen ==0 ) + goto error; + + member =btf_type_member (t) + vlen -1 ; + mtype =btf_type_skip_modifiers (btf_vmlinux, member->type, +NULL ); +if (!btf_type_is_array (mtype)) + goto error; + + array_elem = (struct btf_array *)(mtype +1 ); +if (array_elem->nelems !=0 ) + goto error; + + moff =btf_member_bit_offset (t, member) /8 ; +if (off < moff) + goto error; + +/* Only allow structure for now, can be relaxed for +* other types later. +*/ + elem_type =btf_type_skip_modifiers (btf_vmlinux, + array_elem->type,NULL ); +if (!btf_type_is_struct (elem_type)) + goto error; + + off = (off - moff) % elem_type->size; +return btf_struct_access (log, elem_type, off, size, atype, + next_btf_id); + +error:bpf_log (log,"access beyond struct %s at off %u size %u\n" , tname, off, size);return -EACCES ;
Signature · 1 line
2.24.1
PATCH v3 16/21 tools/libbpf: add bpf_iter support
In reply to Yonghong SongTwo new libbpf APIs are added to support bpf_iter:
- bpf_program__attach_iter
Given a bpf program and additional parameters, which is none now, returns a bpf_link.
- bpf_iter_create
syscall level API to create a bpf iterator.
The macro BPF_SEQ_PRINTF are also introduced. The format looks like:
BPF_SEQ_PRINTF(seq, "task id %d\n", pid);
This macro can help bpf program writers with nicer bpf_seq_printf syntax similar to the kernel one.
--- tools/lib/bpf/bpf.c | 10 +++++++ tools/lib/bpf/bpf.h | 2 ++ tools/lib/bpf/bpf_tracing.h | 16 ++++++++++++ tools/lib/bpf/libbpf.c | 52 +++++++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf.h | 9 +++++++ tools/lib/bpf/libbpf.map | 2 ++ 6 files changed, 91 insertions(+)
tools/lib/bpf/bpf.c +10 -0
@@ -619,6 +619,16 @@ int bpf_link_update(int link_fd, int new_prog_fd,return sys_bpf (BPF_LINK_UPDATE , &attr,sizeof (attr)); } +int bpf_iter_create (int link_fd) +{ +union bpf_attr attr; + +memset (&attr,0 ,sizeof (attr)); + attr.iter_create.link_fd = link_fd; + +return sys_bpf (BPF_ITER_CREATE , &attr,sizeof (attr)); +} +int bpf_prog_query (int target_fd,enum bpf_attach_type type,__u32 query_flags,__u32 *attach_flags,__u32 *prog_ids,__u32 *prog_cnt) {
tools/lib/bpf/bpf.h +2 -0
@@ -187,6 +187,8 @@ struct bpf_link_update_opts { LIBBPF_API int bpf_link_update(int link_fd, int new_prog_fd,const struct bpf_link_update_opts *opts); +LIBBPF_API int bpf_iter_create(int link_fd); +struct bpf_prog_test_run_attr {int prog_fd;int repeat;
tools/lib/bpf/bpf_tracing.h +16 -0
@@ -413,4 +413,20 @@ typeof(name(0)) name(struct pt_regs *ctx) \ } \ static __always_inline typeof(name(0)) ____##name(struct pt_regs *ctx, ##args) +/* +* BPF_SEQ_PRINTF to wrap bpf_seq_printf to-be-printed values +* in a structure. +*/ +#define BPF_SEQ_PRINTF(seq, fmt, args...) \ + ({ \ + _Pragma("GCC diagnostic push") \ + _Pragma("GCC diagnostic ignored \"-Wint-conversion\"") \ + static const char ___fmt[] = fmt; \ + unsigned long long ___param[] = { args }; \ + _Pragma("GCC diagnostic pop") \ + int ___ret = bpf_seq_printf(seq, ___fmt, sizeof(___fmt), \ + ___param, sizeof(___param)); \ + ___ret; \ + }) +#endif
tools/lib/bpf/libbpf.c +52 -0
88 lines not on this page · raw message
tools/lib/bpf/libbpf.h +9 -0
@@ -258,6 +258,15 @@ struct bpf_map; LIBBPF_API struct bpf_link *bpf_map__attach_struct_ops(struct bpf_map *map); +struct bpf_iter_attach_opts { +size_t sz;/* size of this struct for forward/backward compatibility */ +}; +#define bpf_iter_attach_opts__last_field sz + +LIBBPF_API struct bpf_link * +bpf_program__attach_iter (struct bpf_program *prog, +const struct bpf_iter_attach_opts *opts); +struct bpf_insn ;/*
tools/lib/bpf/libbpf.map +2 -0
@@ -258,6 +258,8 @@ LIBBPF_0.0.8 { LIBBPF_0.0.9 { global: bpf_enable_stats; + bpf_iter_create; bpf_link_get_fd_by_id; bpf_link_get_next_id; + bpf_program__attach_iter; } LIBBPF_0.0.8;
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 16/21] tools/libbpf: add bpf_iter support
Quoted 12 lines · Yonghong Song
Two new libbpf APIs are added to support
bpf_iter:
-bpf_program__attach_iter
Given a bpf program and additional parameters, which is none now, returns abpf_link.
-bpf_iter_create
syscall level API to create a bpf iterator.The macro
BPF_SEQ_PRINTFare also introduced. The format looks like:
BPF_SEQ_PRINTF(seq, "task id %d\n", pid);
Quoted 5 lines
This macro can help bpf program writers with nicer
bpf_seq_printfsyntax similar to the kernel one.Signed-off-by Yonghong Song <[email protected]>---
Looks great!
Quoted 8 lines
tools/lib/bpf/bpf.c | 10 +++++++ tools/lib/bpf/bpf.h | 2 ++ tools/lib/bpf/bpf_tracing.h | 16 ++++++++++++ tools/lib/bpf/libbpf.c | 52 +++++++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf.h | 9 +++++++ tools/lib/bpf/libbpf.map | 2 ++ 6 files changed, 91 insertions(+)
[...]
PATCH v3 17/21 tools/libpf: add offsetof/container_of macro in bpf_helpers.h
In reply to Yonghong SongThese two helpers will be used later in bpf_iter bpf program bpf_iter_netlink.c. Put them in bpf_helpers.h since they could be useful in other cases.
--- tools/lib/bpf/bpf_helpers.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
tools/lib/bpf/bpf_helpers.h +14 -0
@@ -36,6 +36,20 @@#define __weak __attribute__((weak))#endif +/* +* Helper macro to manipulate data structures +*/ +#ifndef offsetof +#define offsetof (TYPE ,MEMBER ) ((size_t)&((TYPE *)0)->MEMBER) +#endif +#ifndef container_of +#define container_of (ptr, type, member) \ + ({ \ + void *__mptr = (void *)(ptr); \ + ((type *)(__mptr - offsetof(type, member))); \ + }) +#endif +/* * Helper structure used by eBPF C program * to describe BPF map attributes to libbpf loader
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 17/21] tools/libpf: add offsetof/container_of macro in bpf_helpers.h
Quoted 7 lines · Yonghong Song
These two helpers will be used later in
bpf_iterbpf programbpf_iter_netlink.c. Put them inbpf_helpers.hsince they could be useful in other cases.Signed-off-by Yonghong Song <[email protected]>---
Not sure #ifndef/#endif guards are needed, but won't hurt either.
Quoted 3 lines
tools/lib/bpf/bpf_helpers.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
[...]
PATCH v3 18/21 tools/bpftool: add bpf_iter support for bptool
In reply to Yonghong SongCurrently, only one command is supported
bpftool iter pin <bpf_prog.o> <path>
It will pin the trace/iter bpf program in
the object file <bpf_prog.o> to the <path>
where <path> should be on a bpffs mount.
For example,
$ bpftool iter pin ./bpf_iter_ipv6_route.o \
/sys/fs/bpf/my_route
User can then do a `cat` to print out the results:
$ cat /sys/fs/bpf/my_route
fe800000000000000000000000000000 40 00000000000000000000000000000000 ...
00000000000000000000000000000000 00 00000000000000000000000000000000 ...
00000000000000000000000000000001 80 00000000000000000000000000000000 ...
fe800000000000008c0162fffebdfd57 80 00000000000000000000000000000000 ...
ff000000000000000000000000000000 08 00000000000000000000000000000000 ...
00000000000000000000000000000000 00 00000000000000000000000000000000 ...The implementation for ipv6_route iterator is in one of subsequent patches.
This patch also added BPF_LINK_TYPE_ITER to link query.
In the future, we may add additional parameters to pin command by parameterizing the bpf iterator. For example, a map_id or pid may be added to let bpf program only traverses a single map or task, similar to kernel seq_file single_open().
We may also add introspection command for targets/iterators by leveraging the bpf_iter itself.
--- .../bpftool/Documentation/bpftool-iter.rst | 83 ++++++++++++++++++ tools/bpf/bpftool/bash-completion/bpftool | 13 +++ tools/bpf/bpftool/iter.c | 84 +++++++++++++++++++ tools/bpf/bpftool/link.c | 1 + tools/bpf/bpftool/main.c | 3 +- tools/bpf/bpftool/main.h | 1 + 6 files changed, 184 insertions(+), 1 deletion(-) create mode 100644 tools/bpf/bpftool/Documentation/bpftool-iter.rst create mode 100644 tools/bpf/bpftool/iter.c
tools/bpf/bpftool/Documentation/bpftool-iter.rst new file +83 -0
@@ -0,0 +1,83 @@ +============ +bpftool-iter +============ +------------------------------------------------------------------------------- +tool to create BPF iterators +------------------------------------------------------------------------------- + +:Manual section: 8 + +SYNOPSIS +======== + + **bpftool** [*OPTIONS*] **iter** *COMMAND* + + *COMMANDS* := { **pin** | **help** } + +ITER COMMANDS +=================== + +| **bpftool** **iter pin** *OBJ* *PATH* +| **bpftool** **iter help** +| +| *OBJ* := /a/file/of/bpf_iter_target.o + + +DESCRIPTION +=========== + **bpftool iter pin** *OBJ* *PATH* + A bpf iterator combines a kernel iterating of + particular kernel data (e.g., tasks, bpf_maps, etc.) + and a bpf program called for each kernel data object + (e.g., one task, one bpf_map, etc.). User space can + *read* kernel iterator output through *read()* syscall. + + The *pin* command creates a bpf iterator from *OBJ*, + and pin it to *PATH*. The *PATH* should be located + in *bpffs* mount. It must not contain a dot + character ('.'), which is reserved for future extensions + of *bpffs*. + + User can then *cat PATH* to see the bpf iterator output. + + **bpftool iter help** + Print short help message. + +OPTIONS +======= + -h, --help + Print short generic help message (similar to **bpftool help**). + + -V, --version + Print version number (similar to **bpftool version**). + + -d, --debug + Print all logs available, even debug-level information. This + includes logs from libbpf as well as from the verifier, when + attempting to load programs. + +EXAMPLES +======== +**# bpftool iter pin bpf_iter_netlink.o /sys/fs/bpf/my_netlink** + +:: + + Create a file-based bpf iterator from bpf_iter_netlink.o and pin it + to /sys/fs/bpf/my_netlink + + +SEE ALSO +======== + **bpf**\ (2), + **bpf-helpers**\ (7), + **bpftool**\ (8), + **bpftool-prog**\ (8), + **bpftool-map**\ (8), + **bpftool-link**\ (8), + **bpftool-cgroup**\ (8), + **bpftool-feature**\ (8), + **bpftool-net**\ (8), + **bpftool-perf**\ (8), + **bpftool-btf**\ (8) + **bpftool-gen**\ (8) + **bpftool-struct_ops**\ (8)
tools/bpf/bpftool/bash-completion/bpftool +13 -0
@@ -610,6 +610,19 @@ _bpftool() ;; esac ;; + iter) + case $command in + pin) + _filedir + return 0 + ;; + *) + [[ $prev == $object ]] && \ + COMPREPLY=( $( compgen -W 'pin help' \ + -- "$cur" ) ) + ;; + esac + ;; map) local MAP_TYPE='id pinned name' case $command in
tools/bpf/bpftool/iter.c new file +84 -0
84 lines not on this page · raw message
tools/bpf/bpftool/link.c +1 -0
@@ -16,6 +16,7 @@ static const char * const link_type_name[] = { [BPF_LINK_TYPE_RAW_TRACEPOINT] = "raw_tracepoint", [BPF_LINK_TYPE_TRACING] = "tracing", [BPF_LINK_TYPE_CGROUP] = "cgroup", + [BPF_LINK_TYPE_ITER] = "iter", };static int link_parse_fd (int *argc,char ***argv)
tools/bpf/bpftool/main.c +2 -1
@@ -59,7 +59,7 @@ static int do_help(int argc, char **argv)" %s batch file FILE\n" " %s version\n" "\n" -" OBJECT := { prog | map | link | cgroup | perf | net | feature | btf | gen | struct_ops }\n" +" OBJECT := { prog | map | link | cgroup | perf | net | feature | btf | gen | struct_ops | iter }\n" " " HELP_SPEC_OPTIONS "\n" "" , bin_name, bin_name, bin_name);
@@ -224,6 +224,7 @@ static const struct cmd cmds[] = { { "btf", do_btf }, { "gen", do_gen }, { "struct_ops", do_struct_ops }, + { "iter", do_iter }, { "version", do_version }, { 0 } };
tools/bpf/bpftool/main.h +1 -0
@@ -199,6 +199,7 @@ int do_feature(int argc, char **argv);int do_btf (int argc,char **argv);int do_gen (int argc,char **argv);int do_struct_ops (int argc,char **argv); +int do_iter (int argc,char **argv);int parse_u32_arg (int *argc,char ***argv,__u32 *val,const char *what);int prog_parse_fd (int *argc,char ***argv);
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 18/21] tools/bpftool: add bpf_iter support for bptool
Quoted 45 lines · Yonghong Song
Currently, only one command is supported
bpftool iter pin <bpf_prog.o> <path>It will pin the trace/iter bpf program in
the object file <bpf_prog.o> to the <path>
where <path> should be on a bpffs mount.For example,
$ bpftool iter pin ./bpf_iter_ipv6_route.o \ /sys/fs/bpf/my_route User can then do a `cat` to print out the results: $ cat /sys/fs/bpf/my_route fe800000000000000000000000000000 40 00000000000000000000000000000000 ... 00000000000000000000000000000000 00 00000000000000000000000000000000 ... 00000000000000000000000000000001 80 00000000000000000000000000000000 ... fe800000000000008c0162fffebdfd57 80 00000000000000000000000000000000 ... ff000000000000000000000000000000 08 00000000000000000000000000000000 ... 00000000000000000000000000000000 00 00000000000000000000000000000000 ...The implementation for
ipv6_routeiterator is in one of subsequent patches.This patch also added
BPF_LINK_TYPE_ITERto link query.In the future, we may add additional parameters to pin command by parameterizing the bpf iterator. For example, a
map_idor pid may be added to let bpf program only traverses a single map or task, similar to kernelseq_filesingle_open().We may also add introspection command for targets/iterators by leveraging the
bpf_iteritself.Signed-off-by Yonghong Song <[email protected]>--- .../bpftool/Documentation/bpftool-iter.rst | 83 ++++++++++++++++++ tools/bpf/bpftool/bash-completion/bpftool | 13 +++ tools/bpf/bpftool/iter.c | 84 +++++++++++++++++++ tools/bpf/bpftool/link.c | 1 + tools/bpf/bpftool/main.c | 3 +- tools/bpf/bpftool/main.h | 1 + 6 files changed, 184 insertions(+), 1 deletion(-) create mode 100644 tools/bpf/bpftool/Documentation/bpftool-iter.rst create mode 100644 tools/bpf/bpftool/iter.c
[...]
Quoted 21 lines
tools/bpf/bpftool/iter.c new file +15
-0@@ -0,0 +1,84 @@ +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +// Copyright (C) 2020 Facebook + +#define _GNU_SOURCE +#include <linux/err.h> +#include <bpf/libbpf.h> + +#include "main.h" + +static int do_pin (int argc,char **argv) +{ +const char *objfile, *path; +struct bpf_program *prog; +struct bpf_object *obj; +struct bpf_link *link;
tools/bpf/bpftool/iter.c line 25
Quoted 10 lines
tools/bpf/bpftool/iter.c new file +10
-0+int err; + +if (!REQ_ARGS (2 )) +usage (); + + objfile =GET_ARG (); + path =GET_ARG (); + + obj =bpf_object__open (objfile); +if (IS_ERR_OR_NULL (obj)) {
nit: can't be NULL
tools/bpf/bpftool/iter.c line 36
Quoted 11 lines
tools/bpf/bpftool/iter.c new file +11
-0+p_err ("can't open objfile %s" , objfile); +return -1 ; + } + + err =bpf_object__load (obj); +if (err) { +p_err ("can't load objfile %s" , objfile); + goto close_obj; + } + + prog =bpf_program__next (NULL , obj);
check for null and printf error? Crashing is not good.
Quoted 11 lines
tools/bpf/bpftool/iter.c new file +11
-0+ link =bpf_program__attach_iter (prog,NULL ); +if (IS_ERR (link)) { + err =PTR_ERR (link); +p_err ("attach_iter failed for program %s" , +bpf_program__name (prog)); + goto close_obj; + } + + err =mount_bpffs_for_pin (path); +if (err) + goto close_link;
tools/bpf/bpftool/iter.c line 57
Quoted 10 lines
tools/bpf/bpftool/iter.c new file +10
-0+ + err =bpf_link__pin (link, path); +if (err) { +p_err ("pin_iter failed for program %s to path %s" , +bpf_program__name (prog), path); + goto close_link; + } + +close_link: +bpf_link__disconnect (link);
this is wrong, just destroy()
Quoted 6 lines
tools/bpf/bpftool/iter.c new file +6
-0+bpf_link__destroy (link); +close_obj: +bpf_object__close (obj); +return err; +} +
[...]
Re: [PATCH bpf-next v3 18/21] tools/bpftool: add bpf_iter support for bptool
Quoted 74 lines · Andrii Nakryiko
Quoted 45 lines · Yonghong Song
Currently, only one command is supported
bpftool iter pin <bpf_prog.o> <path>It will pin the trace/iter bpf program in
the object file <bpf_prog.o> to the <path>
where <path> should be on a bpffs mount.For example,
$ bpftool iter pin ./bpf_iter_ipv6_route.o \ /sys/fs/bpf/my_route User can then do a `cat` to print out the results: $ cat /sys/fs/bpf/my_route fe800000000000000000000000000000 40 00000000000000000000000000000000 ... 00000000000000000000000000000000 00 00000000000000000000000000000000 ... 00000000000000000000000000000001 80 00000000000000000000000000000000 ... fe800000000000008c0162fffebdfd57 80 00000000000000000000000000000000 ... ff000000000000000000000000000000 08 00000000000000000000000000000000 ... 00000000000000000000000000000000 00 00000000000000000000000000000000 ...The implementation for
ipv6_routeiterator is in one of subsequent patches.This patch also added
BPF_LINK_TYPE_ITERto link query.In the future, we may add additional parameters to pin command by parameterizing the bpf iterator. For example, a
map_idor pid may be added to let bpf program only traverses a single map or task, similar to kernelseq_filesingle_open().We may also add introspection command for targets/iterators by leveraging the
bpf_iteritself.Signed-off-by Yonghong Song <[email protected]>--- .../bpftool/Documentation/bpftool-iter.rst | 83 ++++++++++++++++++ tools/bpf/bpftool/bash-completion/bpftool | 13 +++ tools/bpf/bpftool/iter.c | 84 +++++++++++++++++++ tools/bpf/bpftool/link.c | 1 + tools/bpf/bpftool/main.c | 3 +- tools/bpf/bpftool/main.h | 1 + 6 files changed, 184 insertions(+), 1 deletion(-)create mode 100644
tools/bpf/bpftool/Documentation/bpftool-iter.rstcreate mode 100644tools/bpf/bpftool/iter.c[...]
Quoted 25 lines
tools/bpf/bpftool/iter.c new file +19
-0@@ -0,0 +1,84 @@ +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +// Copyright (C) 2020 Facebook + +#define _GNU_SOURCE +#include <linux/err.h> +#include <bpf/libbpf.h> + +#include "main.h" + +static int do_pin (int argc,char **argv) +{ +const char *objfile, *path; +struct bpf_program *prog; +struct bpf_object *obj; +struct bpf_link *link; +int err; + +if (!REQ_ARGS (2 )) +usage ();
Quoted 8 lines
tools/bpf/bpftool/iter.c line 25
Quoted 6 lines
tools/bpf/bpftool/iter.c new file +6
-0+ + objfile =GET_ARG (); + path =GET_ARG (); + + obj =bpf_object__open (objfile); +if (IS_ERR_OR_NULL (obj)) {nit: can't be NULL
Ack. Will change.
Quoted 14 lines
tools/bpf/bpftool/iter.c line 36
Quoted 12 lines
tools/bpf/bpftool/iter.c new file +11
-0+p_err ("can't open objfile %s" , objfile); +return -1 ; + } + + err =bpf_object__load (obj); +if (err) { +p_err ("can't load objfile %s" , objfile); + goto close_obj; + } + + prog =bpf_program__next (NULL , obj);check for null and printf error? Crashing is not good.
Make sense. Will change.
Quoted 16 lines
Quoted 16 lines
tools/bpf/bpftool/iter.c new file +15
-0+ link =bpf_program__attach_iter (prog,NULL ); +if (IS_ERR (link)) { + err =PTR_ERR (link); +p_err ("attach_iter failed for program %s" , +bpf_program__name (prog)); + goto close_obj; + } + + err =mount_bpffs_for_pin (path); +if (err) + goto close_link; + + err =bpf_link__pin (link, path); +if (err) { + p_err("pin_iter failed for program %s to path %s",
Quoted 8 lines
tools/bpf/bpftool/iter.c line 57
Quoted 6 lines
tools/bpf/bpftool/iter.c new file +6
-0+ bpf_program__name(prog), path); + goto close_link; + } + +close_link: +bpf_link__disconnect (link);this is wrong, just
destroy()
Will change.
Quoted 10 lines
Quoted 7 lines
tools/bpf/bpftool/iter.c new file +6
-0+bpf_link__destroy (link); +close_obj: +bpf_object__close (obj); +return err; +} +[...]
PATCH v3 19/21 tools/bpf: selftests: add iterator programs for ipv6_route and netlink
In reply to Yonghong SongTwo bpf programs are added in this patch for netlink and ipv6_route target. On my VM, I am able to achieve identical
results compared to /proc/net/netlink and /proc/net/ipv6_route.
$ cat /proc/net/netlink sk Eth Pid Groups Rmem Wmem Dump Locks Drops Inode 000000002c42d58b 0 0 00000000 0 0 0 2 0 7 00000000a4e8b5e1 0 1 00000551 0 0 0 2 0 18719 00000000e1b1c195 4 0 00000000 0 0 0 2 0 16422 000000007e6b29f9 6 0 00000000 0 0 0 2 0 16424 .... 00000000159a170d 15 1862 00000002 0 0 0 2 0 1886 000000009aca4bc9 15 3918224839 00000002 0 0 0 2 0 19076 00000000d0ab31d2 15 1 00000002 0 0 0 2 0 18683 000000008398fb08 16 0 00000000 0 0 0 2 0 27 $ cat /sys/fs/bpf/my_netlink sk Eth Pid Groups Rmem Wmem Dump Locks Drops Inode 000000002c42d58b 0 0 00000000 0 0 0 2 0 7 00000000a4e8b5e1 0 1 00000551 0 0 0 2 0 18719 00000000e1b1c195 4 0 00000000 0 0 0 2 0 16422 000000007e6b29f9 6 0 00000000 0 0 0 2 0 16424 .... 00000000159a170d 15 1862 00000002 0 0 0 2 0 1886 000000009aca4bc9 15 3918224839 00000002 0 0 0 2 0 19076 00000000d0ab31d2 15 1 00000002 0 0 0 2 0 18683 000000008398fb08 16 0 00000000 0 0 0 2 0 27
$ cat /proc/net/ipv6_route fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000001 00000000 00000001 eth0 00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo 00000000000000000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000003 00000000 80200001 lo fe80000000000000c04b03fffe7827ce 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000002 00000000 80200001 eth0 ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000003 00000000 00000001 eth0 00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo $ cat /sys/fs/bpf/my_ipv6_route fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000001 00000000 00000001 eth0 00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo 00000000000000000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000003 00000000 80200001 lo fe80000000000000c04b03fffe7827ce 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000002 00000000 80200001 eth0 ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000003 00000000 00000001 eth0 00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 00000000 00200200 lo
--- .../selftests/bpf/progs/bpf_iter_ipv6_route.c | 62 +++++++++++++++++ .../selftests/bpf/progs/bpf_iter_netlink.c | 66 +++++++++++++++++++ 2 files changed, 128 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_ipv6_route.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_netlink.c
tools/testing/selftests/bpf/progs/bpf_iter_ipv6_route.c new file +62 -0
@@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +char _license[] SEC("license") = "GPL"; + +extern bool CONFIG_IPV6_SUBTREES __kconfig __weak; + +#define RTF_GATEWAY 0x0002 +#define IFNAMSIZ 16 +#define fib_nh_gw_family nh_common.nhc_gw_family +#define fib_nh_gw6 nh_common.nhc_gw.ipv6 +#define fib_nh_dev nh_common.nhc_dev + +SEC("iter/ipv6_route") +int dump_ipv6_route (struct bpf_iter__ipv6_route *ctx) +{ +struct seq_file *seq = ctx->meta->seq; +struct fib6_info *rt = ctx->rt; +const struct net_device *dev; +struct fib6_nh *fib6_nh; +unsigned int flags; +struct nexthop *nh; + +if (rt == (void *)0 ) +return 0 ; + + fib6_nh = &rt->fib6_nh[0 ]; + flags = rt->fib6_flags; + +/* FIXME: nexthop_is_multipath is not handled here. */ + nh = rt->nh; +if (rt->nh) + fib6_nh = &nh->nh_info->fib6_nh; + +BPF_SEQ_PRINTF (seq,"%pi6 %02x " , &rt->fib6_dst.addr, rt->fib6_dst.plen); + +if (CONFIG_IPV6_SUBTREES ) +BPF_SEQ_PRINTF (seq,"%pi6 %02x " , &rt->fib6_src.addr, + rt->fib6_src.plen); +else +BPF_SEQ_PRINTF (seq,"00000000000000000000000000000000 00 " ); + +if (fib6_nh->fib_nh_gw_family) { + flags |=RTF_GATEWAY ; +BPF_SEQ_PRINTF (seq,"%pi6 " , &fib6_nh->fib_nh_gw6); + }else { +BPF_SEQ_PRINTF (seq,"00000000000000000000000000000000 " ); + } + + dev = fib6_nh->fib_nh_dev; +if (dev) +BPF_SEQ_PRINTF (seq,"%08x %08x %08x %08x %8s\n" , rt->fib6_metric, + rt->fib6_ref.refs.counter,0 , flags, dev->name); +else +BPF_SEQ_PRINTF (seq,"%08x %08x %08x %08x\n" , rt->fib6_metric, + rt->fib6_ref.refs.counter,0 , flags); + +return 0 ; +}
tools/testing/selftests/bpf/progs/bpf_iter_netlink.c new file +66 -0
66 lines not on this page · raw message
Signature · 1 line
2.24.1
PATCH v3 20/21 tools/bpf: selftests: add iter progs for bpf_map/task/task_file
In reply to Yonghong SongThe implementation is arbitrary, just to show how the bpf programs can be written for bpf_map/task/task_file. They can be costomized for specific needs.
For example, for bpf_map, the iterator prints out:
$ cat /sys/fs/bpf/my_bpf_map
id refcnt usercnt locked_vm
3 2 0 20
6 2 0 20
9 2 0 20
12 2 0 20
13 2 0 20
16 2 0 20
19 2 0 20
%%% END %%%For task, the iterator prints out:
$ cat /sys/fs/bpf/my_task
tgid gid
1 1
2 2
....
1944 1944
1948 1948
1949 1949
1953 1953
=== END ===For task/file, the iterator prints out:
$ cat /sys/fs/bpf/my_task_file
tgid gid fd file
1 1 0 ffffffff95c97600
1 1 1 ffffffff95c97600
1 1 2 ffffffff95c97600
....
1895 1895 255 ffffffff95c8fe00
1932 1932 0 ffffffff95c8fe00
1932 1932 1 ffffffff95c8fe00
1932 1932 2 ffffffff95c8fe00
1932 1932 3 ffffffff95c185c0This is able to print out all open files (fd and file->f_op), so user can compare f_op against a particular kernel file operations to find what it is. For example, from /proc/kallsyms, we can find
ffffffff95c185c0 r eventfd_fops
so we will know tgid 1932 fd 3 is an eventfd file descriptor.
--- .../selftests/bpf/progs/bpf_iter_bpf_map.c | 28 +++++++++++++++++++ .../selftests/bpf/progs/bpf_iter_task.c | 25 +++++++++++++++++ .../selftests/bpf/progs/bpf_iter_task_file.c | 26 +++++++++++++++++ 3 files changed, 79 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_bpf_map.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_task.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_task_file.c
tools/testing/selftests/bpf/progs/bpf_iter_bpf_map.c new file +28 -0
@@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +char _license[] SEC("license") = "GPL"; + +SEC("iter/bpf_map") +int dump_bpf_map (struct bpf_iter__bpf_map *ctx) +{ +struct seq_file *seq = ctx->meta->seq; +__u64 seq_num = ctx->meta->seq_num; +struct bpf_map *map = ctx->map; + +if (map == (void *)0 ) { +BPF_SEQ_PRINTF (seq," %%%%%% END %%%%%%\n" ); +return 0 ; + } + +if (seq_num ==0 ) +BPF_SEQ_PRINTF (seq," id refcnt usercnt locked_vm\n" ); + +BPF_SEQ_PRINTF (seq,"%8u %8ld %8ld %10lu\n" , map->id, map->refcnt.counter, + map->usercnt.counter, + map->memory.user->locked_vm.counter); +return 0 ; +}
tools/testing/selftests/bpf/progs/bpf_iter_task.c new file +25 -0
@@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +char _license[] SEC("license") = "GPL"; + +SEC("iter/task") +int dump_task (struct bpf_iter__task *ctx) +{ +struct seq_file *seq = ctx->meta->seq; +struct task_struct *task = ctx->task; + +if (task == (void *)0 ) { +BPF_SEQ_PRINTF (seq," === END ===\n" ); +return 0 ; + } + +if (ctx->meta->seq_num ==0 ) +BPF_SEQ_PRINTF (seq," tgid gid\n" ); + +BPF_SEQ_PRINTF (seq,"%8d %8d\n" , task->tgid, task->pid); +return 0 ; +}
tools/testing/selftests/bpf/progs/bpf_iter_task_file.c new file +26 -0
@@ -0,0 +1,26 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +char _license[] SEC("license") = "GPL"; + +SEC("iter/task_file") +int dump_task_file (struct bpf_iter__task_file *ctx) +{ +struct seq_file *seq = ctx->meta->seq; +struct task_struct *task = ctx->task; +__u32 fd = ctx->fd; +struct file *file = ctx->file; + +if (task == (void *)0 || file == (void *)0 ) +return 0 ; + +if (ctx->meta->seq_num ==0 ) +BPF_SEQ_PRINTF (seq," tgid gid fd file\n" ); + +BPF_SEQ_PRINTF (seq,"%8d %8d %8d %lx\n" , task->tgid, task->pid, fd, + (long )file->f_op); +return 0 ; +}
Signature · 1 line
2.24.1
PATCH v3 21/21 tools/bpf: selftests: add bpf_iter selftests
In reply to Yonghong SongThe added test includes the following subtests:
- test verifier change for btf_id_or_null
- test load/create_iter/read for ipv6_route/netlink/bpf_map/task/task_file
- test anon bpf iterator
- test anon bpf iterator reading one char at a time
- test file bpf iterator
- test overflow (single bpf program output not overflow)
- test overflow (single bpf program output overflows)
- test bpf prog returning 1
The ipv6_route tests the following verifier change
- access fields in the variable length array of the structure.
The netlink load tests the following verifier change
- put a btf_id ptr value in a stack and accessible to tracing/iter programs.
The anon bpf iterator also tests link auto attach through skeleton.
$ test_progs -n 2 #2/1 btf_id_or_null:OK #2/2 ipv6_route:OK #2/3 netlink:OK #2/4 bpf_map:OK #2/5 task:OK #2/6 task_file:OK #2/7 anon:OK #2/8 anon-read-one-char:OK #2/9 file:OK #2/10 overflow:OK #2/11 overflow-e2big:OK #2/12 prog-ret-1:OK #2 bpf_iter:OK Summary: 1/12 PASSED, 0 SKIPPED, 0 FAILED
--- .../selftests/bpf/prog_tests/bpf_iter.c | 408 ++++++++++++++++++ .../selftests/bpf/progs/bpf_iter_test_kern1.c | 4 + .../selftests/bpf/progs/bpf_iter_test_kern2.c | 4 + .../selftests/bpf/progs/bpf_iter_test_kern3.c | 18 + .../selftests/bpf/progs/bpf_iter_test_kern4.c | 52 +++ .../bpf/progs/bpf_iter_test_kern_common.h | 22 + 6 files changed, 508 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_iter.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern1.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern2.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern3.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern4.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern_common.h
tools/testing/selftests/bpf/prog_tests/bpf_iter.c new file +408 -0
@@ -0,0 +1,408 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ +#include <test_progs.h> +#include "bpf_iter_ipv6_route.skel.h" +#include "bpf_iter_netlink.skel.h" +#include "bpf_iter_bpf_map.skel.h" +#include "bpf_iter_task.skel.h" +#include "bpf_iter_task_file.skel.h" +#include "bpf_iter_test_kern1.skel.h" +#include "bpf_iter_test_kern2.skel.h" +#include "bpf_iter_test_kern3.skel.h" +#include "bpf_iter_test_kern4.skel.h" + +static int duration; + +static void test_btf_id_or_null (void ) +{ +struct bpf_iter_test_kern3 *skel; + + skel =bpf_iter_test_kern3__open_and_load (); +if (CHECK (skel,"bpf_iter_test_kern3__open_and_load" , +"skeleton open_and_load unexpectedly succeeded\n" )) { +bpf_iter_test_kern3__destroy (skel); +return ; + } +} + +static void do_dummy_read (struct bpf_program *prog) +{ +struct bpf_link *link; +char buf[16 ] = {}; +int iter_fd, len; + + link =bpf_program__attach_iter (prog,NULL ); +if (CHECK (IS_ERR (link),"attach_iter" ,"attach_iter failed\n" )) +return ; + + iter_fd =bpf_iter_create (bpf_link__fd (link)); +if (CHECK (iter_fd <0 ,"create_iter" ,"create_iter failed\n" )) + goto free_link; + +/* not check contents, but ensure read() ends without error */ +while ((len =read (iter_fd, buf,sizeof (buf))) >0 ) + ; +CHECK (len <0 ,"read" ,"read failed: %s\n" ,strerror (errno)); + +close (iter_fd); + +free_link: +bpf_link__destroy (link); +} + +static void test_ipv6_route (void ) +{ +struct bpf_iter_ipv6_route *skel; + + skel =bpf_iter_ipv6_route__open_and_load (); +if (CHECK (!skel,"bpf_iter_ipv6_route__open_and_load" , +"skeleton open_and_load failed\n" )) +return ; + +do_dummy_read (skel->progs.dump_ipv6_route); + +bpf_iter_ipv6_route__destroy (skel); +} + +static void test_netlink (void ) +{ +struct bpf_iter_netlink *skel; + + skel =bpf_iter_netlink__open_and_load (); +if (CHECK (!skel,"bpf_iter_netlink__open_and_load" , +"skeleton open_and_load failed\n" )) +return ; + +do_dummy_read (skel->progs.dump_netlink); + +bpf_iter_netlink__destroy (skel); +} + +static void test_bpf_map (void ) +{ +struct bpf_iter_bpf_map *skel; + + skel =bpf_iter_bpf_map__open_and_load (); +if (CHECK (!skel,"bpf_iter_bpf_map__open_and_load" , +"skeleton open_and_load failed\n" )) +return ; + +do_dummy_read (skel->progs.dump_bpf_map); + +bpf_iter_bpf_map__destroy (skel); +} + +static void test_task (void ) +{ +struct bpf_iter_task *skel; + + skel =bpf_iter_task__open_and_load (); +if (CHECK (!skel,"bpf_iter_task__open_and_load" , +"skeleton open_and_load failed\n" )) +return ; + +do_dummy_read (skel->progs.dump_task); + +bpf_iter_task__destroy (skel); +} + +static void test_task_file (void ) +{ +struct bpf_iter_task_file *skel; + + skel =bpf_iter_task_file__open_and_load (); +if (CHECK (!skel,"bpf_iter_task_file__open_and_load" , +"skeleton open_and_load failed\n" )) +return ; + +do_dummy_read (skel->progs.dump_task_file); + +bpf_iter_task_file__destroy (skel); +} + +/* The expected string is less than 16 bytes */ +static int do_read_with_fd (int iter_fd,const char *expected, +bool read_one_char) +{ +int err =-1 , len, read_buf_len, start; +char buf[16 ] = {}; + + read_buf_len = read_one_char ?1 :16 ; + start =0 ; +while ((len =read (iter_fd, buf + start, read_buf_len)) >0 ) { + start += len; +if (CHECK (start >=16 ,"read" ,"read len %d\n" , len)) +return -1 ; + read_buf_len = read_one_char ?1 :16 - start; + } +if (CHECK (len <0 ,"read" ,"read failed: %s\n" ,strerror (errno))) +return -1 ; + + err =strcmp (buf, expected); +if (CHECK (err,"read" ,"incorrect read result: buf %s, expected %s\n" , + buf, expected)) +return -1 ; + +return 0 ; +} + +static void test_anon_iter (bool read_one_char) +{ +struct bpf_iter_test_kern1 *skel; +struct bpf_link *link; +int iter_fd, err; + + skel =bpf_iter_test_kern1__open_and_load (); +if (CHECK (!skel,"bpf_iter_test_kern1__open_and_load" , +"skeleton open_and_load failed\n" )) +return ; + + err =bpf_iter_test_kern1__attach (skel); +if (CHECK (err,"bpf_iter_test_kern1__attach" , +"skeleton attach failed\n" )) { + goto out; + } + + link = skel->links.dump_task; + iter_fd =bpf_iter_create (bpf_link__fd (link)); +if (CHECK (iter_fd <0 ,"create_iter" ,"create_iter failed\n" )) + goto out; + +do_read_with_fd (iter_fd,"abcd" , read_one_char); +close (iter_fd); + +out: +bpf_iter_test_kern1__destroy (skel); +} + +static int do_read (const char *path,const char *expected) +{ +int err, iter_fd; + + iter_fd =open (path,O_RDONLY ); +if (CHECK (iter_fd <0 ,"open" ,"open %s failed: %s\n" , + path,strerror (errno))) +return -1 ; + + err =do_read_with_fd (iter_fd, expected, false); +close (iter_fd); +return err; +} + +static void test_file_iter (void ) +{ +const char *path ="/sys/fs/bpf/bpf_iter_test1" ; +struct bpf_iter_test_kern1 *skel1; +struct bpf_iter_test_kern2 *skel2; +struct bpf_link *link; +int err; + + skel1 =bpf_iter_test_kern1__open_and_load (); +if (CHECK (!skel1,"bpf_iter_test_kern1__open_and_load" , +"skeleton open_and_load failed\n" )) +return ; + + link =bpf_program__attach_iter (skel1->progs.dump_task,NULL ); +if (CHECK (IS_ERR (link),"attach_iter" ,"attach_iter failed\n" )) + goto out; + +/* unlink this path if it exists. */ +unlink (path); + + err =bpf_link__pin (link, path); +if (CHECK (err,"pin_iter" ,"pin_iter to %s failed: %s\n" , path, +strerror (errno))) + goto free_link; + + err =do_read (path,"abcd" ); +if (err) + goto free_link; + +/* file based iterator seems working fine. Let us a link update +* of the underlying link and `cat` the iterator again, its content +* should change. +*/ + skel2 =bpf_iter_test_kern2__open_and_load (); +if (CHECK (!skel2,"bpf_iter_test_kern2__open_and_load" , +"skeleton open_and_load failed\n" )) + goto free_link; + + err =bpf_link__update_program (link, skel2->progs.dump_task); +if (CHECK (err,"update_prog" ,"update_prog failed\n" )) + goto destroy_skel2; + +do_read (path,"ABCD" ); + +destroy_skel2: +bpf_iter_test_kern2__destroy (skel2); +free_link: +bpf_link__destroy (link); +out: +bpf_iter_test_kern1__destroy (skel1); +} + +static void test_overflow (bool test_e2big_overflow,bool ret1) +{ +__u32 map_info_len, total_read_len, expected_read_len; +int err, iter_fd, map1_fd, map2_fd, len; +struct bpf_map_info map_info = {}; +struct bpf_iter_test_kern4 *skel; +struct bpf_link *link; +__u32 page_size; +char *buf; + + skel =bpf_iter_test_kern4__open (); +if (CHECK (!skel,"bpf_iter_test_kern4__open" , +"skeleton open failed\n" )) +return ; + +/* create two maps: bpf program will only do bpf_seq_write +* for these two maps. The goal is one map output almost +* fills seq_file buffer and then the other will trigger +* overflow and needs restart. +*/ + map1_fd =bpf_create_map (BPF_MAP_TYPE_ARRAY ,4 ,8 ,1 ,0 ); +if (CHECK (map1_fd <0 ,"bpf_create_map" , +"map_creation failed: %s\n" ,strerror (errno))) + goto out; + map2_fd =bpf_create_map (BPF_MAP_TYPE_ARRAY ,4 ,8 ,1 ,0 ); +if (CHECK (map2_fd <0 ,"bpf_create_map" , +"map_creation failed: %s\n" ,strerror (errno))) + goto free_map1; + +/* bpf_seq_printf kernel buffer is one page, so one map +* bpf_seq_write will mostly fill it, and the other map +* will partially fill and then trigger overflow and need +* bpf_seq_read restart. +*/ + page_size =sysconf (_SC_PAGE_SIZE); + +if (test_e2big_overflow) { + skel->rodata->print_len = (page_size +8 ) /8 ; + expected_read_len =2 * (page_size +8 ); + }else if (!ret1) { + skel->rodata->print_len = (page_size -8 ) /8 ; + expected_read_len =2 * (page_size -8 ); + }else { + skel->rodata->print_len =1 ; + expected_read_len =2 *8 ; + } + skel->rodata->ret1 = ret1; + +if (CHECK (bpf_iter_test_kern4__load (skel), +"bpf_iter_test_kern4__load" ,"skeleton load failed\n" )) + goto free_map2; + +/* setup filtering map_id in bpf program */ + map_info_len =sizeof (map_info); + err =bpf_obj_get_info_by_fd (map1_fd, &map_info, &map_info_len); +if (CHECK (err,"get_map_info" ,"get map info failed: %s\n" , +strerror (errno))) + goto free_map2; + skel->bss->map1_id = map_info.id; + + err =bpf_obj_get_info_by_fd (map2_fd, &map_info, &map_info_len); +if (CHECK (err,"get_map_info" ,"get map info failed: %s\n" , +strerror (errno))) + goto free_map2; + skel->bss->map2_id = map_info.id; + + link =bpf_program__attach_iter (skel->progs.dump_bpf_map,NULL ); +if (CHECK (IS_ERR (link),"attach_iter" ,"attach_iter failed\n" )) + goto free_map2; + + iter_fd =bpf_iter_create (bpf_link__fd (link)); +if (CHECK (iter_fd <0 ,"create_iter" ,"create_iter failed\n" )) + goto free_link; + + buf =malloc (expected_read_len); +if (!buf) + goto close_iter; + +/* do read */ + total_read_len =0 ; +if (test_e2big_overflow) { +while ((len =read (iter_fd, buf, expected_read_len)) >0 ) + total_read_len += len; + +CHECK (len !=-1 || errno !=E2BIG ,"read" , +"expected ret -1, errno E2BIG, but get ret %d, error %s\n" , + len,strerror (errno)); + goto free_buf; + }else if (!ret1) { +while ((len =read (iter_fd, buf, expected_read_len)) >0 ) + total_read_len += len; + +if (CHECK (len <0 ,"read" ,"read failed: %s\n" , +strerror (errno))) + goto free_buf; + }else { +do { + len =read (iter_fd, buf, expected_read_len); +if (len >0 ) + total_read_len += len; + }while (len >0 || len == -EAGAIN ); + +if (CHECK (len <0 ,"read" ,"read failed: %s\n" , +strerror (errno))) + goto free_buf; + } + +if (CHECK (total_read_len != expected_read_len,"read" , +"total len %u, expected len %u\n" , total_read_len, + expected_read_len)) + goto free_buf; + +if (CHECK (skel->bss->map1_accessed !=1 ,"map1_accessed" , +"expected 1 actual %d\n" , skel->bss->map1_accessed)) + goto free_buf; + +if (CHECK (skel->bss->map2_accessed !=2 ,"map2_accessed" , +"expected 2 actual %d\n" , skel->bss->map2_accessed)) + goto free_buf; + +CHECK (skel->bss->map2_seqnum1 != skel->bss->map2_seqnum2, +"map2_seqnum" ,"two different seqnum %lld %lld\n" , + skel->bss->map2_seqnum1, skel->bss->map2_seqnum2); + +free_buf: +free (buf); +close_iter: +close (iter_fd); +free_link: +bpf_link__destroy (link); +free_map2: +close (map2_fd); +free_map1: +close (map1_fd); +out: +bpf_iter_test_kern4__destroy (skel); +} + +void test_bpf_iter (void ) +{ +if (test__start_subtest ("btf_id_or_null" )) +test_btf_id_or_null (); +if (test__start_subtest ("ipv6_route" )) +test_ipv6_route (); +if (test__start_subtest ("netlink" )) +test_netlink (); +if (test__start_subtest ("bpf_map" )) +test_bpf_map (); +if (test__start_subtest ("task" )) +test_task (); +if (test__start_subtest ("task_file" )) +test_task_file (); +if (test__start_subtest ("anon" )) +test_anon_iter (false); +if (test__start_subtest ("anon-read-one-char" )) +test_anon_iter (true); +if (test__start_subtest ("file" )) +test_file_iter (); +if (test__start_subtest ("overflow" )) +test_overflow (false, false); +if (test__start_subtest ("overflow-e2big" )) +test_overflow (true, false); +if (test__start_subtest ("prog-ret-1" )) +test_overflow (false, true); +}
tools/testing/selftests/bpf/progs/bpf_iter_test_kern1.c new file +4 -0
@@ -0,0 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ +#define START_CHAR 'a' +#include "bpf_iter_test_kern_common.h"
tools/testing/selftests/bpf/progs/bpf_iter_test_kern2.c new file +4 -0
@@ -0,0 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ +#define START_CHAR 'A' +#include "bpf_iter_test_kern_common.h"
tools/testing/selftests/bpf/progs/bpf_iter_test_kern3.c new file +18 -0
@@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> + +char _license[] SEC("license") = "GPL"; + +SEC("iter/task") +int dump_task (struct bpf_iter__task *ctx) +{ +struct seq_file *seq = ctx->meta->seq; +struct task_struct *task = ctx->task; +int tgid; + + tgid = task->tgid; +bpf_seq_write (seq, &tgid,sizeof (tgid)); +return 0 ; +}
tools/testing/selftests/bpf/progs/bpf_iter_test_kern4.c new file +52 -0
52 lines not on this page · raw message
tools/testing/selftests/bpf/progs/bpf_iter_test_kern_common.h new file +22 -0
@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2020 Facebook */ +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> + +char _license[] SEC("license") = "GPL"; +int count =0 ; + +SEC("iter/task") +int dump_task (struct bpf_iter__task *ctx) +{ +struct seq_file *seq = ctx->meta->seq; +char c; + +if (count <4 ) { + c =START_CHAR + count; +bpf_seq_write (seq, &c,sizeof (c)); + count++; + } + +return 0 ; +}
Signature · 1 line
2.24.1
Re: [PATCH bpf-next v3 21/21] tools/bpf: selftests: add bpf_iter selftests
Quoted 37 lines · Yonghong Song
The added test includes the following subtests:
- test verifier change forbtf_id_or_null
- test load/create_iter/read for ipv6_route/netlink/bpf_map/task/task_file
- test anon bpf iterator
- test anon bpf iterator reading one char at a time
- test file bpf iterator
- test overflow (single bpf program output not overflow)
- test overflow (single bpf program output overflows)
- test bpf prog returning 1The
ipv6_routetests the following verifier change
- access fields in the variable length array of the structure.The netlink load tests the following verifier change
- put abtf_idptr value in a stack and accessible to tracing/iter programs.The anon bpf iterator also tests link auto attach through skeleton.
$ test_progs -n 2 #2/1 btf_id_or_null:OK #2/2 ipv6_route:OK #2/3 netlink:OK #2/4 bpf_map:OK #2/5 task:OK #2/6 task_file:OK #2/7 anon:OK #2/8 anon-read-one-char:OK #2/9 file:OK #2/10 overflow:OK #2/11 overflow-e2big:OK #2/12 prog-ret-1:OK #2 bpf_iter:OK Summary: 1/12 PASSED, 0 SKIPPED, 0 FAILED
Quoted 2 lines
Signed-off-by Yonghong Song <[email protected]>---
I'm personally not a big fan of bpf_iter_test_kern_common.h approach (including it and parameterizing with #define), I'd rather just copy/paste BPF program code (it's just a few lines) and maybe even put them in the same file/skeleton, less files to jump between. But that's just personal preferences, so:
Quoted 14 lines
.../selftests/bpf/prog_tests/bpf_iter.c | 408 ++++++++++++++++++ .../selftests/bpf/progs/bpf_iter_test_kern1.c | 4 + .../selftests/bpf/progs/bpf_iter_test_kern2.c | 4 + .../selftests/bpf/progs/bpf_iter_test_kern3.c | 18 + .../selftests/bpf/progs/bpf_iter_test_kern4.c | 52 +++ .../bpf/progs/bpf_iter_test_kern_common.h | 22 + 6 files changed, 508 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_iter.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern1.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern2.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern3.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern4.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_test_kern_common.h
[...]