Open-coded task_vma iter
Subsystems (4)
v5 cover Open-coded task_vma iter
At Meta we have a profiling daemon which periodically collects information on many hosts. This collection usually involves grabbing stacks (user and kernel) using perf_event BPF progs and later symbolicating them. For user stacks we try to use BPF_F_USER_BUILD_ID and rely on remote symbolication, but BPF_F_USER_BUILD_ID doesn't always succeed. In those cases we must fall back to digging around in /proc/PID/maps to map virtual address to (binary, offset). The /proc/PID/maps digging does not occur synchronously with stack collection, so the process might already be gone, in which case it won't have /proc/PID/maps and we will fail to symbolicate.
This 'exited process problem' doesn't occur very often as most of the prod services we care to profile are long-lived daemons, but there are enough usecases to warrant a workaround: a BPF program which can be optionally loaded at data collection time and essentially walks /proc/PID/maps. Currently this is done by walking the vma list:
struct vm_area_struct* mmap = BPF_CORE_READ(mm, mmap); mmap_next = BPF_CORE_READ(rmap, vm_next); /* in a loop */
Since commit 763ecb035029 ("mm: remove the vma linked list") there's no longer a vma linked list to walk. Walking the vma maple tree is not as simple as hopping struct vm_area_struct->vm_next. Luckily,
commit f39af05949a4 ("mm: add VMA iterator"), another commit in that series, added struct vma_iterator and for_each_vma macro for easy vma iteration. If similar functionality was exposed to BPF programs, it would be perfect for our usecase.
This series adds such functionality, specifically a BPF equivalent of for_each_vma using the open-coded iterator style.
Notes:
* This approach was chosen after discussion on a previous series [0] which attempted to solve the same problem by adding a BPF_F_VMA_NEXT flag to bpf_find_vma.
* Unlike the task_vma bpf_iter, the open-coded iterator kfuncs here do not drop the vma read lock between iterations. See Alexei's response in [0].
* The [vsyscall] page isn't really part of task->mm's vmas, but /proc/PID/maps returns information about it anyways. The vma iter added here does not do the same. See comment on selftest in patch 3.
* bpf_iter_task_vma allocates a _data struct which contains - among other things - struct vma_iterator, using BPF allocator and keeps a pointer to the bpf_iter_task_vma_data. This is done in order to prevent changes to struct ma_state - which is wrapped by struct vma_iterator - from necessitating changes to uapi struct bpf_iter_task_vma.
Changelog:
v4 -> v5: https://lore.kernel.org/bpf/[email protected]/
Patch numbers correspond to their position in v4
New Patch ("selftests/bpf: Rename bpf_iter_task_vma.c to bpf_iter_task_vmas.c")
* Patch 2's renaming of this selftest, and associated changes in the userspace runner, are split out into this separate commit (Andrii)
Patch 2 ("bpf: Introduce task_vma open-coded iterator kfuncs")
* Remove bpf_iter_task_vma kfuncs from libbpf's bpf_helpers.h, they'll be added to selftests' bpf_experimental.h in selftests patch below (Andrii)
* Split bpf_iter_task_vma.c renaming into separate commit (Andrii)
Patch 3 ("selftests/bpf: Add tests for open-coded task_vma iter")
* Add bpf_iter_task_vma kfuncs to bpf_experimental.h (Andrii)
* Remove '?' from prog SEC, open_and_load the skel in one operation (Andrii)
* Ensure that fclose() always happens in test runner (Andrii)
* Use global var w/ 1000 (vm_start, vm_end) structs instead of two MAP_TYPE_ARRAY's w/ 1k u64s each (Andrii)
v3 -> v4: https://lore.kernel.org/bpf/[email protected]/
Patch 1 ("bpf: Don't explicitly emit BTF for struct btf_iter_num")
* Add Andrii ack
Patch 2 ("bpf: Introduce task_vma open-coded iterator kfuncs")
* Mark bpf_iter_task_vma_new args KF_RCU and remove now-unnecessary !task check (Yonghong)
* Although KF_RCU is a function-level flag, in reality it only applies to the task_struct *task parameter, as the other two params are a scalar int and a specially-handled KF_ARG_PTR_TO_ITER
* Remove struct bpf_iter_task_vma definition from uapi headers, define in kernel/bpf/task_iter.c instead (Andrii)
Patch 3 ("selftests/bpf: Add tests for open-coded task_vma iter")
* Use a local var when looping over vmas to track map idx. Update vmas_seen global after done iterating. Don't start iterating or update vmas_seen if vmas_seen global is nonzero. (Andrii)
* Move getpgid() call to correct spot - above skel detach. (Andrii)
v2 -> v3: https://lore.kernel.org/bpf/[email protected]/
Patch 1 ("bpf: Don't explicitly emit BTF for struct btf_iter_num")
* Add Yonghong ack
Patch 2 ("bpf: Introduce task_vma open-coded iterator kfuncs")
* UAPI bpf header and tools/ version should match
* Add bpf_iter_task_vma_kern_data which bpf_iter_task_vma_kern points to, bpf_mem_alloc/free it instead of just vma_iterator. (Alexei)
* Inner data ptr == NULL implies initialization failed
v1 -> v2: https://lore.kernel.org/bpf/[email protected]/
* Patch 1
* Now removes the unnecessary BTF_TYPE_EMIT instead of changing the type (Yonghong)
* Patch 2
* Don't do unnecessary BTF_TYPE_EMIT (Yonghong)
* Bump task refcount to prevent ->mm reuse (Yonghong)
* Keep a pointer to vma_iterator in bpf_iter_task_vma, alloc/free via BPF mem allocator (Yonghong, Stanislav)
* Patch 3
Patch summary:
* Patch 1 is a tiny fix I ran into while implementing the vma iter in this series. It can be applied independently.
* Patch 2 is the meat of the implementation
* Patch 3 adds tests for the new functionality
* Existing iter tests exercise failure cases (e.g. prog that doesn't call _destroy()). I didn't replicate them in this series, but am happy to add them in v2 if folks feel that it would be worthwhile.
[0]: https://lore.kernel.org/bpf/[email protected]/
Dave Marchevsky (4):
bpf: Don't explicitly emit BTF for struct btf_iter_num selftests/bpf: Rename bpf_iter_task_vma.c to bpf_iter_task_vmas.c
bpf: Introduce task_vma open-coded iterator kfuncs selftests/bpf: Add tests for open-coded task_vma iter
kernel/bpf/bpf_iter.c | 2 -
kernel/bpf/helpers.c | 3 +
kernel/bpf/task_iter.c | 85 +++++++++++++++++++
.../testing/selftests/bpf/bpf_experimental.h | 8 ++
.../selftests/bpf/prog_tests/bpf_iter.c | 26 +++---
.../testing/selftests/bpf/prog_tests/iters.c | 59 +++++++++++++
...f_iter_task_vma.c => bpf_iter_task_vmas.c} | 0
.../selftests/bpf/progs/iters_task_vma.c | 46 ++++++++++
8 files changed, 214 insertions(+), 15 deletions(-)
rename tools/testing/selftests/bpf/progs/{bpf_iter_task_vma.c => bpf_iter_task_vmas.c} (100%)
create mode 100644 tools/testing/selftests/bpf/progs/iters_task_vma.cSignature · 1 line
2.34.1
PATCH v5 1/4 bpf: Don't explicitly emit BTF for struct btf_iter_num
Commit 6018e1f407cc ("bpf: implement numbers iterator") added the BTF_TYPE_EMIT line that this patch is modifying. The struct btf_iter_num doesn't exist, so only a forward declaration is emitted in BTF:
FWD 'btf_iter_num' fwd_kind=struct
That commit was probably hoping to ensure that struct bpf_iter_num is emitted in vmlinux BTF. A previous version of this patch changed the line to emit the correct type, but Yonghong confirmed that it would definitely be emitted regardless in [0], so this patch simply removes the line.
This isn't marked "Fixes" because the extraneous btf_iter_num FWD wasn't causing any issues that I noticed, aside from mild confusion when I looked through the code.
[0]: https://lore.kernel.org/bpf/[email protected]/
--- kernel/bpf/bpf_iter.c | 2 -- 1 file changed, 2 deletions(-)
kernel/bpf/bpf_iter.c +0 -2
@@ -793,8 +793,6 @@ __bpf_kfunc int bpf_iter_num_new(struct bpf_iter_num *it, int start, int end)BUILD_BUG_ON (sizeof (struct bpf_iter_num_kern ) !=sizeof (struct bpf_iter_num ));BUILD_BUG_ON (__alignof__(struct bpf_iter_num_kern ) != __alignof__(struct bpf_iter_num )); -BTF_TYPE_EMIT (struct btf_iter_num ); -/* start == end is legit, it's an empty range and we'll just get NULL * on first (and any subsequent) bpf_iter_num_next() call */
Signature · 1 line
2.34.1
PATCH v5 2/4 selftests/bpf: Rename bpf_iter_task_vma.c to bpf_iter_task_vmas.c
In reply to Dave MarchevskyFurther patches in this series will add a struct bpf_iter_task_vma, which will result in a name collision with the selftest prog renamed in this patch. Rename the selftest to avoid the collision.
---
.../selftests/bpf/prog_tests/bpf_iter.c | 26 +++++++++----------
...f_iter_task_vma.c => bpf_iter_task_vmas.c} | 0
2 files changed, 13 insertions(+), 13 deletions(-)
rename tools/testing/selftests/bpf/progs/{bpf_iter_task_vma.c => bpf_iter_task_vmas.c} (100%)tools/testing/selftests/bpf/prog_tests/bpf_iter.c +13 -13
@@ -10,7 +10,7 @@#include "bpf_iter_task.skel.h" #include "bpf_iter_task_stack.skel.h" #include "bpf_iter_task_file.skel.h" -#include "bpf_iter_task_vma.skel.h" +#include "bpf_iter_task_vmas.skel.h" #include "bpf_iter_task_btf.skel.h" #include "bpf_iter_tcp4.skel.h" #include "bpf_iter_tcp6.skel.h"
@@ -1399,19 +1399,19 @@ static void str_strip_first_line(char *str)static void test_task_vma_common (struct bpf_iter_attach_opts *opts) {int err, iter_fd =-1 , proc_maps_fd =-1 ; -struct bpf_iter_task_vma *skel; +struct bpf_iter_task_vmas *skel;int len, read_size =4 ;char maps_path[64 ]; - skel =bpf_iter_task_vma__open (); -if (!ASSERT_OK_PTR (skel,"bpf_iter_task_vma__open" )) + skel =bpf_iter_task_vmas__open (); +if (!ASSERT_OK_PTR (skel,"bpf_iter_task_vmas__open" ))return ; skel->bss->pid =getpid (); skel->bss->one_task = opts ?1 :0 ; - err =bpf_iter_task_vma__load (skel); -if (!ASSERT_OK (err,"bpf_iter_task_vma__load" )) + err =bpf_iter_task_vmas__load (skel); +if (!ASSERT_OK (err,"bpf_iter_task_vmas__load" )) goto out; skel->links.proc_maps = bpf_program__attach_iter(
@@ -1462,25 +1462,25 @@ static void test_task_vma_common(struct bpf_iter_attach_opts *opts) out:close (proc_maps_fd);close (iter_fd); -bpf_iter_task_vma__destroy (skel); +bpf_iter_task_vmas__destroy (skel); }static void test_task_vma_dead_task (void ) { -struct bpf_iter_task_vma *skel; +struct bpf_iter_task_vmas *skel;int wstatus, child_pid =-1 ;time_t start_tm, cur_tm;int err, iter_fd =-1 ;int wait_sec =3 ; - skel =bpf_iter_task_vma__open (); -if (!ASSERT_OK_PTR (skel,"bpf_iter_task_vma__open" )) + skel =bpf_iter_task_vmas__open (); +if (!ASSERT_OK_PTR (skel,"bpf_iter_task_vmas__open" ))return ; skel->bss->pid =getpid (); - err =bpf_iter_task_vma__load (skel); -if (!ASSERT_OK (err,"bpf_iter_task_vma__load" )) + err =bpf_iter_task_vmas__load (skel); +if (!ASSERT_OK (err,"bpf_iter_task_vmas__load" )) goto out; skel->links.proc_maps = bpf_program__attach_iter(
@@ -1533,7 +1533,7 @@ static void test_task_vma_dead_task(void) out:waitpid (child_pid, &wstatus,0 );close (iter_fd); -bpf_iter_task_vma__destroy (skel); +bpf_iter_task_vmas__destroy (skel); }void test_bpf_sockmap_map_iter_fd (void )
tools/testing/selftests/bpf/progs/bpf_iter_task_vmas.c renamed from tools/testing/selftests/bpf/progs/bpf_iter_task_vma.c +0 -0
Signature · 1 line
2.34.1
PATCH v5 3/4 bpf: Introduce task_vma open-coded iterator kfuncs
In reply to Dave MarchevskyThis patch adds kfuncs bpf_iter_task_vma_{new,next,destroy} which allow creation and manipulation of struct bpf_iter_task_vma in open-coded iterator style. BPF programs can use these kfuncs directly or through bpf_for_each macro for natural-looking iteration of all task vmas.
The implementation borrows heavily from bpf_find_vma helper's locking - differing only in that it holds the mmap_read lock for all iterations while the helper only executes its provided callback on a maximum of 1 vma. Aside from locking, struct vma_iterator and vma_next do all the heavy lifting.
A pointer to an inner data struct, struct bpf_iter_task_vma_data, is the only field in struct bpf_iter_task_vma. This is because the inner data struct contains a struct vma_iterator (not ptr), whose size is likely to change under us. If bpf_iter_task_vma_kern contained vma_iterator directly such a change would require change in opaque bpf_iter_task_vma struct's size. So better to allocate vma_iterator using BPF allocator, and since that alloc must already succeed, might as well allocate all iter fields, thereby freezing struct bpf_iter_task_vma size.
--- kernel/bpf/helpers.c | 3 ++ kernel/bpf/task_iter.c | 85 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+)
kernel/bpf/helpers.c +3 -0
@@ -2552,6 +2552,9 @@ BTF_ID_FLAGS(func, bpf_dynptr_slice_rdwr, KF_RET_NULL) BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW) BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY) +BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY) BTF_ID_FLAGS(func, bpf_dynptr_adjust) BTF_ID_FLAGS(func, bpf_dynptr_is_null)BTF_ID_FLAGS (func, bpf_dynptr_is_rdonly)
kernel/bpf/task_iter.c +85 -0
98 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v5 4/4 selftests/bpf: Add tests for open-coded task_vma iter
In reply to Dave MarchevskyThe open-coded task_vma iter added earlier in this series allows for natural iteration over a task's vmas using existing open-coded iter infrastructure, specifically bpf_for_each.
This patch adds a test demonstrating this pattern and validating correctness. The vma->vm_start and vma->vm_end addresses of the first 1000 vmas are recorded and compared to /proc/PID/maps output. As expected, both see the same vmas and addresses - with the exception of the [vsyscall] vma - which is explained in a comment in the prog_tests program.
--- .../testing/selftests/bpf/bpf_experimental.h | 8 +++ .../testing/selftests/bpf/prog_tests/iters.c | 59 +++++++++++++++++++ .../selftests/bpf/progs/iters_task_vma.c | 46 +++++++++++++++ 3 files changed, 113 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/iters_task_vma.c
tools/testing/selftests/bpf/bpf_experimental.h +8 -0
@@ -159,6 +159,14 @@ extern void *bpf_percpu_obj_new_impl(__u64 local_type_id, void *meta) __ksym;*/ extern void bpf_percpu_obj_drop_impl(void *kptr, void *meta) __ksym; +struct bpf_iter_task_vma ; + +extern int bpf_iter_task_vma_new(struct bpf_iter_task_vma *it, + struct task_struct *task, + unsigned long addr) __ksym; +extern struct vm_area_struct *bpf_iter_task_vma_next (struct bpf_iter_task_vma *it) __ksym; +extern void bpf_iter_task_vma_destroy(struct bpf_iter_task_vma *it) __ksym; +/* Convenience macro to wrap over bpf_obj_drop_impl */ #define bpf_percpu_obj_drop (kptr) bpf_percpu_obj_drop_impl(kptr, NULL)
tools/testing/selftests/bpf/prog_tests/iters.c +58 -0
74 lines not on this page · raw message
tools/testing/selftests/bpf/progs/iters_task_vma.c new file +46 -0
46 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v5 bpf-next 4/4] selftests/bpf: Add tests for open-coded task_vma iter
Quoted 11 lines · Dave Marchevsky
The open-coded
task_vmaiter added earlier in this series allows for natural iteration over a task's vmas using existing open-coded iter infrastructure, specificallybpf_for_each.This patch adds a test demonstrating this pattern and validating correctness. The
vma->vm_startandvma->vm_endaddresses of the first 1000 vmas are recorded and compared to/proc/PID/mapsoutput. As expected, both see the same vmas and addresses - with the exception of the [vsyscall] vma - which is explained in a comment in theprog_testsprogram.
Quoted 2 lines
Signed-off-by Dave Marchevsky <[email protected]>---
BPF CI is unhappy as it's unable to apply this patch. Likely because I manually deleted an extraneous empty line in the .patch file.
Will respin as v6 shortly.