bpf: Add bpf_iter_cpumask
Subsystems (4)
v5 cover bpf: Add bpf_iter_cpumask
Three new kfuncs, namely bpf_iter_cpumask_{new,next,destroy}, have been added for the new bpf_iter_cpumask functionality. These kfuncs enable the iteration of percpu data, such as runqueues, system_group_pcpu, and more.
In our specific use case, we leverage the cgroup iterator to traverse percpu data, subsequently exposing it to userspace through a seq file. Refer to example in patch #2 for the usage.
Changes:
- v4 -> v5:
- Use cpumask_size() instead of sizeof(struct cpumask) (David)
- Refactor the selftests as suggsted by David
- Improve the doc as suggested by David
- v3 -> v4:
- Use a copy of cpumask to avoid potential use-after-free (Yonghong)
- Various code improvement in selftests (Yonghong)
- v2 -> v3:
- Define KF_RCU_PROTECTED for bpf_iter_cpumask_new (Alexei)
- Code improvement in selftests
- Fix build error in selftest due to CONFIG_PSI=n reported by kernel test robot <[email protected]>
- v1 -> v2:
- Avoid changing cgroup subsystem (Tejun)
- Remove bpf_cpumask_set_from_pid(), and use bpf_cpumask_copy() instead (Tejun)
- Use int cpu; field in bpf_iter_cpumask_kern (Andrii)
- bpf: Add new bpf helper bpf_for_each_cpu
https://lwn.net/ml/bpf/[email protected]/
Yafang Shao (4):
bpf: Add bpf_iter_cpumask kfuncs
bpf, docs: Add document for cpumask iter selftests/bpf: Fix error checking for cpumask_success__load() selftests/bpf: Add selftests for cpumask iter
Documentation/bpf/cpumasks.rst | 60 +++++++ kernel/bpf/cpumask.c | 82 +++++++++ tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cpumask.c | 158 +++++++++++++++++- .../selftests/bpf/progs/cpumask_common.h | 3 + .../bpf/progs/cpumask_iter_failure.c | 99 +++++++++++ .../bpf/progs/cpumask_iter_success.c | 126 ++++++++++++++ 7 files changed, 526 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/cpumask_iter_failure.c create mode 100644 tools/testing/selftests/bpf/progs/cpumask_iter_success.c
Signature · 1 line
2.39.1
PATCH v5 1/4 bpf: Add bpf_iter_cpumask kfuncs
Add three new kfuncs for bpf_iter_cpumask.
- bpf_iter_cpumask_new
KF_RCU is defined because the cpumask must be a RCU trusted pointer such as task->cpus_ptr.
- bpf_iter_cpumask_next
- bpf_iter_cpumask_destroy
These new kfuncs facilitate the iteration of percpu data, such as runqueues, psi_cgroup_cpu, and more.
--- kernel/bpf/cpumask.c | 82 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+)
kernel/bpf/cpumask.c +82 -0
@@ -422,6 +422,85 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)return cpumask_weight (cpumask); } +struct bpf_iter_cpumask { +__u64 __opaque[2 ]; +}__aligned(8) ; + +struct bpf_iter_cpumask_kern { +struct cpumask *mask; +int cpu; +}__aligned(8) ; + +/** +* bpf_iter_cpumask_new() - Create a new bpf_iter_cpumask for a specified cpumask +* @it: The new bpf_iter_cpumask to be created. +* @mask: The cpumask to be iterated over. +* +* This function initializes a new bpf_iter_cpumask structure for iterating over +* the specified CPU mask. It assigns the provided cpumask to the newly created +* bpf_iter_cpumask @it for subsequent iteration operations. +* +* On success, 0 is returen. On failure, ERR is returned. +*/ +__bpf_kfunc int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; + +BUILD_BUG_ON (sizeof (struct bpf_iter_cpumask_kern ) >sizeof (struct bpf_iter_cpumask )); +BUILD_BUG_ON (__alignof__(struct bpf_iter_cpumask_kern ) != + __alignof__(struct bpf_iter_cpumask )); + + kit->mask =bpf_mem_alloc (&bpf_global_ma,cpumask_size ()); +if (!kit->mask) +return -ENOMEM ; + +cpumask_copy (kit->mask, mask); + kit->cpu =-1 ; +return 0 ; +} + +/** +* bpf_iter_cpumask_next() - Get the next CPU in a bpf_iter_cpumask +* @it: The bpf_iter_cpumask +* +* This function retrieves a pointer to the number of the next CPU within the +* specified bpf_iter_cpumask. It allows sequential access to CPUs within the +* cpumask. If there are no further CPUs available, it returns NULL. +* +* Returns a pointer to the number of the next CPU in the cpumask or NULL if no +* further CPUs. +*/ +__bpf_kfunc int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; +const struct cpumask *mask = kit->mask; +int cpu; + +if (!mask) +return NULL ; + cpu =cpumask_next (kit->cpu, mask); +if (cpu >= nr_cpu_ids) +return NULL ; + + kit->cpu = cpu; +return &kit->cpu; +} + +/** +* bpf_iter_cpumask_destroy() - Destroy a bpf_iter_cpumask +* @it: The bpf_iter_cpumask to be destroyed. +* +* Destroy the resource assiciated with the bpf_iter_cpumask. +*/ +__bpf_kfunc void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; + +if (!kit->mask) +return ; +bpf_mem_free (&bpf_global_ma, kit->mask); +} +__bpf_kfunc_end_defs ();BTF_SET8_START (cpumask_kfunc_btf_ids)
@@ -450,6 +529,9 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_new, KF_ITER_NEW | KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_cpumask_destroy, KF_ITER_DESTROY)BTF_SET8_END (cpumask_kfunc_btf_ids) static const struct btf_kfunc_id_set cpumask_kfunc_set = {
Signature · 1 line
2.39.1
Re: [PATCH v5 bpf-next 1/4] bpf: Add bpf_iter_cpumask kfuncs
Quoted 25 lines · Yafang Shao
Add three new kfuncs for
bpf_iter_cpumask.
-bpf_iter_cpumask_new
KF_RCUis defined because the cpumask must be a RCU trusted pointer such astask->cpus_ptr.
-bpf_iter_cpumask_next
-bpf_iter_cpumask_destroyThese new kfuncs facilitate the iteration of percpu data, such as runqueues,
psi_cgroup_cpu, and more.Signed-off-by Yafang Shao <[email protected]>--- kernel/bpf/cpumask.c | 82 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+)kernel/bpf/cpumask.c +1
-0@@ -422,6 +422,85 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)return cpumask_weight (cpumask); } +struct bpf_iter_cpumask {
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 435
Quoted 10 lines
kernel/bpf/cpumask.c +10
-0+__u64 __opaque[2 ]; +}__aligned(8) ; + +struct bpf_iter_cpumask_kern { +struct cpumask *mask; +int cpu; +}__aligned(8) ; + +/** +* bpf_iter_cpumask_new() - Create a new bpf_iter_cpumask for a specified cpumask
I'd say "Initialize a new CPU mask iterator for a given CPU mask"? "new bpf_iter_cpumask" is both confusing and misleading (we don't create it, we fill provided struct)
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 441
Quoted 6 lines
kernel/bpf/cpumask.c +6
-0+* @it: The new bpf_iter_cpumask to be created. +* @mask: The cpumask to be iterated over. +* +* This function initializes a new bpf_iter_cpumask structure for iterating over +* the specified CPU mask. It assigns the provided cpumask to the newly created +* bpf_iter_cpumask @it for subsequent iteration operations.
The description lgtm.
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 443
Quoted 2 lines
kernel/bpf/cpumask.c +2
-0+* +* On success, 0 is returen. On failure, ERR is returned.
typo: returned
Quoted 13 lines
kernel/bpf/cpumask.c +13
-0+*/ +__bpf_kfunc int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; + +BUILD_BUG_ON (sizeof (struct bpf_iter_cpumask_kern ) >sizeof (struct bpf_iter_cpumask )); +BUILD_BUG_ON (__alignof__(struct bpf_iter_cpumask_kern ) != + __alignof__(struct bpf_iter_cpumask )); + + kit->mask =bpf_mem_alloc (&bpf_global_ma,cpumask_size ()); +if (!kit->mask) +return -ENOMEM ; +
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 466
Quoted 10 lines
kernel/bpf/cpumask.c +10
-0+cpumask_copy (kit->mask, mask); + kit->cpu =-1 ; +return 0 ; +} + +/** +* bpf_iter_cpumask_next() - Get the next CPU in a bpf_iter_cpumask +* @it: The bpf_iter_cpumask +* +* This function retrieves a pointer to the number of the next CPU within the
"function returns a pointer to a number representing the ID of the next CPU in CPU mask" ?
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 471
Quoted 5 lines
kernel/bpf/cpumask.c +5
-0+* specified bpf_iter_cpumask. It allows sequential access to CPUs within the +* cpumask. If there are no further CPUs available, it returns NULL. +* +* Returns a pointer to the number of the next CPU in the cpumask or NULL if no +* further CPUs.
this and last sentence before this basically repeat the same twice, let's keep just one?
Quoted 12 lines
kernel/bpf/cpumask.c +12
-0+*/ +__bpf_kfunc int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; +const struct cpumask *mask = kit->mask; +int cpu; + +if (!mask) +return NULL ; + cpu =cpumask_next (kit->cpu, mask); +if (cpu >= nr_cpu_ids) +return NULL ;
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 493
Quoted 10 lines
kernel/bpf/cpumask.c +10
-0+ + kit->cpu = cpu; +return &kit->cpu; +} + +/** +* bpf_iter_cpumask_destroy() - Destroy a bpf_iter_cpumask +* @it: The bpf_iter_cpumask to be destroyed. +* +* Destroy the resource assiciated with the bpf_iter_cpumask.
typo: associated
Quoted 14 lines
kernel/bpf/cpumask.c +10
-0+*/ +__bpf_kfunc void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; + +if (!kit->mask) +return ; +bpf_mem_free (&bpf_global_ma, kit->mask); +} +__bpf_kfunc_end_defs ();BTF_SET8_START (cpumask_kfunc_btf_ids)
kernel/bpf/cpumask.c BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) · line 535
Quoted 7 lines
kernel/bpf/cpumask.c +3
-0@@ -450,6 +529,9 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_new, KF_ITER_NEW | KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_cpumask_destroy, KF_ITER_DESTROY)BTF_SET8_END (cpumask_kfunc_btf_ids)
Seems like you'll have to rebase, there is a merge conflict with recently landed changes.
Quoted 5 lines
kernel/bpf/cpumask.c +0
-1static const struct btf_kfunc_id_set cpumask_kfunc_set = { --2.39.1
Re: [PATCH v5 bpf-next 1/4] bpf: Add bpf_iter_cpumask kfuncs
Quoted 26 lines · Andrii Nakryiko
Quoted 24 lines · Yafang Shao
Add three new kfuncs for
bpf_iter_cpumask.
-bpf_iter_cpumask_new
KF_RCUis defined because the cpumask must be a RCU trusted pointer such astask->cpus_ptr.
-bpf_iter_cpumask_next
-bpf_iter_cpumask_destroyThese new kfuncs facilitate the iteration of percpu data, such as runqueues,
psi_cgroup_cpu, and more.Signed-off-by Yafang Shao <[email protected]>--- kernel/bpf/cpumask.c | 82 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+)kernel/bpf/cpumask.c +0
-0@@ -422,6 +422,85 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)return cpumask_weight (cpumask); }
Quoted 15 lines
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 435
Quoted 11 lines
kernel/bpf/cpumask.c +11
-0+struct bpf_iter_cpumask { +__u64 __opaque[2 ]; +}__aligned(8) ; + +struct bpf_iter_cpumask_kern { +struct cpumask *mask; +int cpu; +}__aligned(8) ; + +/** +* bpf_iter_cpumask_new() - Create a new bpf_iter_cpumask for a specified cpumask I'd say "Initialize a new CPU mask iterator for a given CPU mask"? "new
bpf_iter_cpumask" is both confusing and misleading (we don't create it, we fill provided struct)
will change it.
Quoted 8 lines
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 441
Quoted 7 lines
kernel/bpf/cpumask.c +6
-0+* @it: The new bpf_iter_cpumask to be created. +* @mask: The cpumask to be iterated over. +* +* This function initializes a new bpf_iter_cpumask structure for iterating over +* the specified CPU mask. It assigns the provided cpumask to the newly created +* bpf_iter_cpumask @it for subsequent iteration operations.
Quoted 6 lines
The description lgtm.
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 443
Quoted 2 lines
kernel/bpf/cpumask.c +2
-0+* +* On success, 0 is returen. On failure, ERR is returned. typo: returned
will fix it.
Quoted 19 lines
Quoted 19 lines
kernel/bpf/cpumask.c +18
-0+*/ +__bpf_kfunc int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; + +BUILD_BUG_ON (sizeof (struct bpf_iter_cpumask_kern ) >sizeof (struct bpf_iter_cpumask )); +BUILD_BUG_ON (__alignof__(struct bpf_iter_cpumask_kern ) != + __alignof__(struct bpf_iter_cpumask )); + + kit->mask =bpf_mem_alloc (&bpf_global_ma,cpumask_size ()); +if (!kit->mask) +return -ENOMEM ; + +cpumask_copy (kit->mask, mask); + kit->cpu =-1 ; +return 0 ; +} +
Quoted 8 lines
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 466
Quoted 5 lines
kernel/bpf/cpumask.c +5
-0+/** +* bpf_iter_cpumask_next() - Get the next CPU in a bpf_iter_cpumask +* @it: The bpf_iter_cpumask +* +* This function retrieves a pointer to the number of the next CPU within the "function returns a pointer to a number representing the ID of the next CPU in CPU mask" ?
will change it.
Quoted 9 lines
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 471
Quoted 6 lines
kernel/bpf/cpumask.c +5
-0+* specified bpf_iter_cpumask. It allows sequential access to CPUs within the +* cpumask. If there are no further CPUs available, it returns NULL. +* +* Returns a pointer to the number of the next CPU in the cpumask or NULL if no +* further CPUs. this and last sentence before this basically repeat the same twice, let's keep just one?
will change it.
Quoted 18 lines
Quoted 18 lines
kernel/bpf/cpumask.c +16
-0+*/ +__bpf_kfunc int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; +const struct cpumask *mask = kit->mask; +int cpu; + +if (!mask) +return NULL ; + cpu =cpumask_next (kit->cpu, mask); +if (cpu >= nr_cpu_ids) +return NULL ; + + kit->cpu = cpu; +return &kit->cpu; +}
Quoted 8 lines
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 493
Quoted 6 lines
kernel/bpf/cpumask.c +6
-0+ +/** +* bpf_iter_cpumask_destroy() - Destroy a bpf_iter_cpumask +* @it: The bpf_iter_cpumask to be destroyed. +* +* Destroy the resource assiciated with the bpf_iter_cpumask. typo: associated
will fix it.
Quoted 13 lines
Quoted 13 lines
kernel/bpf/cpumask.c +10
-0+*/ +__bpf_kfunc void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; + +if (!kit->mask) +return ; +bpf_mem_free (&bpf_global_ma, kit->mask); +} +__bpf_kfunc_end_defs ();
Quoted 12 lines
kernel/bpf/cpumask.c BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) · line 535
Quoted 9 lines
kernel/bpf/cpumask.c +3
-0BTF_SET8_START (cpumask_kfunc_btf_ids)@@ -450,6 +529,9 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_new, KF_ITER_NEW | KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_cpumask_destroy, KF_ITER_DESTROY)BTF_SET8_END (cpumask_kfunc_btf_ids)Seems like you'll have to rebase, there is a merge conflict with recently landed changes.
will do it.
Quoted 7 lines
Quoted 7 lines
kernel/bpf/cpumask.c +0
-1static const struct btf_kfunc_id_set cpumask_kfunc_set = { --2.39.1
Signature · 2 lines
Regards Yafang
PATCH v5 2/4 bpf, docs: Add document for cpumask iter
In reply to Yafang ShaoThis patch adds the document for the newly added cpumask iterator kfuncs.
--- Documentation/bpf/cpumasks.rst | 60 ++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+)
Documentation/bpf/cpumasks.rst +60 -0
@@ -372,6 +372,66 @@ used. .. _tools/testing/selftests/bpf/progs/cpumask_success.c: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/progs/cpumask_success.c +3.3 cpumask iterator +-------------------- + +The cpumask iterator facilitates the iteration of per-CPU data, including +runqueues, system_group_pcpu, and other such structures. To leverage the cpumask +iterator, one can employ the bpf_for_each() macro. + +Here's an example illustrating how to determine the number of running tasks on +each CPU. + +.. code-block:: c + + /** + * Here's an example demonstrating the functionality of the cpumask iterator. + * We retrieve the cpumask associated with the specified pid, iterate through + * its elements, and ultimately expose per-CPU data to userspace through a + * seq file. + */ + const struct rq runqueues __ksym __weak; + u32 target_pid; + + SEC("iter/cgroup") + int BPF_PROG(cpu_cgroup, struct bpf_iter_meta *meta, struct cgroup *cgrp) + { + u32 nr_running = 0, nr_cpus = 0, nr_null_rq = 0; + struct task_struct *p; + struct rq *rq; + int *cpu; + + /* epilogue */ + if (cgrp == NULL) + return 0; + + p = bpf_task_from_pid(target_pid); + if (!p) + return 1; + + BPF_SEQ_PRINTF(meta->seq, "%4s %s\n", "CPU", "nr_running"); + bpf_for_each(cpumask, cpu, p->cpus_ptr) { + rq = (struct rq *)bpf_per_cpu_ptr(&runqueues, *cpu); + if (!rq) { + nr_null_rq += 1; + continue; + } + nr_cpus += 1; + + if (!rq->nr_running) + continue; + + nr_running += rq->nr_running; + BPF_SEQ_PRINTF(meta->seq, "%4u %u\n", *cpu, rq->nr_running); + } + BPF_SEQ_PRINTF(meta->seq, "Summary: nr_cpus %u, nr_running %u, nr_null_rq %u\n", + nr_cpus, nr_running, nr_null_rq); + + bpf_task_release(p); + return 0; + } + +---- 4. Adding BPF cpumask kfuncs ============================
Signature · 1 line
2.39.1
Re: [PATCH v5 bpf-next 2/4] bpf, docs: Add document for cpumask iter
Quoted 6 lines · Yafang Shao
This patch adds the document for the newly added cpumask iterator kfuncs.
Signed-off-by Yafang Shao <[email protected]>---
LGTM.
Quoted 77 lines
Documentation/bpf/cpumasks.rst | 60 ++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+)Documentation/bpf/cpumasks.rst +60
-1@@ -372,6 +372,66 @@ used. .. _tools/testing/selftests/bpf/progs/cpumask_success.c: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/progs/cpumask_success.c +3.3 cpumask iterator +-------------------- + +The cpumask iterator facilitates the iteration of per-CPU data, including +runqueues, system_group_pcpu, and other such structures. To leverage the cpumask +iterator, one can employ the bpf_for_each() macro. + +Here's an example illustrating how to determine the number of running tasks on +each CPU. + +.. code-block:: c + + /** + * Here's an example demonstrating the functionality of the cpumask iterator. + * We retrieve the cpumask associated with the specified pid, iterate through + * its elements, and ultimately expose per-CPU data to userspace through a + * seq file. + */ + const struct rq runqueues __ksym __weak; + u32 target_pid; + + SEC("iter/cgroup") + int BPF_PROG(cpu_cgroup, struct bpf_iter_meta *meta, struct cgroup *cgrp) + { + u32 nr_running = 0, nr_cpus = 0, nr_null_rq = 0; + struct task_struct *p; + struct rq *rq; + int *cpu; + + /* epilogue */ + if (cgrp == NULL) + return 0; + + p = bpf_task_from_pid(target_pid); + if (!p) + return 1; + + BPF_SEQ_PRINTF(meta->seq, "%4s %s\n", "CPU", "nr_running"); + bpf_for_each(cpumask, cpu, p->cpus_ptr) { + rq = (struct rq *)bpf_per_cpu_ptr(&runqueues, *cpu); + if (!rq) { + nr_null_rq += 1; + continue; + } + nr_cpus += 1; + + if (!rq->nr_running) + continue; + + nr_running += rq->nr_running; + BPF_SEQ_PRINTF(meta->seq, "%4u %u\n", *cpu, rq->nr_running); + } + BPF_SEQ_PRINTF(meta->seq, "Summary: nr_cpus %u, nr_running %u, nr_null_rq %u\n", + nr_cpus, nr_running, nr_null_rq); + + bpf_task_release(p); + return 0; + } + +---- 4. Adding BPF cpumask kfuncs ============================ --2.39.1
PATCH v5 3/4 selftests/bpf: Fix error checking for cpumask_success__load()
In reply to Yafang ShaoWe should verify the return value of cpumask_success__load().
--- tools/testing/selftests/bpf/prog_tests/cpumask.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tools/testing/selftests/bpf/prog_tests/cpumask.c +3 -3
@@ -27,7 +27,7 @@ static void verify_success(const char *prog_name)struct bpf_program *prog;struct bpf_link *link =NULL ;pid_t child_pid; -int status; +int status, err; skel =cpumask_success__open ();if (!ASSERT_OK_PTR (skel,"cpumask_success__open" ))
@@ -36,8 +36,8 @@ static void verify_success(const char *prog_name) skel->bss->pid =getpid (); skel->bss->nr_cpus =libbpf_num_possible_cpus (); -cpumask_success__load (skel); -if (!ASSERT_OK_PTR (skel,"cpumask_success__load" )) + err =cpumask_success__load (skel); +if (!ASSERT_OK (err,"cpumask_success__load" )) goto cleanup; prog =bpf_object__find_program_by_name (skel->obj, prog_name);
Signature · 1 line
2.39.1
Re: [PATCH v5 bpf-next 3/4] selftests/bpf: Fix error checking for cpumask_success__load()
Quoted 6 lines · Yafang Shao
We should verify the return value of
cpumask_success__load().Signed-off-by Yafang Shao <[email protected]>Cc David Vernet <[email protected]>---
Quoted 30 lines
tools/testing/selftests/bpf/prog_tests/cpumask.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)tools/testing/selftests/bpf/prog_tests/cpumask.c +3
-4@@ -27,7 +27,7 @@ static void verify_success(const char *prog_name)struct bpf_program *prog;struct bpf_link *link =NULL ;pid_t child_pid; -int status; +int status, err; skel =cpumask_success__open ();if (!ASSERT_OK_PTR (skel,"cpumask_success__open" ))@@ -36,8 +36,8 @@ static void verify_success(const char *prog_name) skel->bss->pid =getpid (); skel->bss->nr_cpus =libbpf_num_possible_cpus (); -cpumask_success__load (skel); -if (!ASSERT_OK_PTR (skel,"cpumask_success__load" )) + err =cpumask_success__load (skel); +if (!ASSERT_OK (err,"cpumask_success__load" )) goto cleanup; prog =bpf_object__find_program_by_name (skel->obj, prog_name); --2.39.1
PATCH v5 4/4 selftests/bpf: Add selftests for cpumask iter
In reply to Yafang ShaoAdd selftests for the newly added cpumask iter.
- cpumask_iter_success
- The number of CPUs should be expected when iterating over the cpumask
- percpu data extracted from the percpu struct should be expected
- It can work in both non-sleepable and sleepable prog
- RCU lock is only required by bpf_iter_cpumask_new()
- It is fine without calling bpf_iter_cpumask_next()
- cpumask_iter_failure
- RCU lock is required in sleepable prog
- The cpumask to be iterated over can't be NULL
- bpf_iter_cpumask_destroy() is required after calling bpf_iter_cpumask_new()
- bpf_iter_cpumask_destroy() can only destroy an initilialized iter
- bpf_iter_cpumask_next() must use an initilialized iter
The result as follows,
#64/37 cpumask/test_cpumask_iter:OK #64/38 cpumask/test_cpumask_iter_sleepable:OK #64/39 cpumask/test_cpumask_iter_sleepable:OK #64/40 cpumask/test_cpumask_iter_next_no_rcu:OK #64/41 cpumask/test_cpumask_iter_no_next:OK #64/42 cpumask/test_cpumask_iter:OK #64/43 cpumask/test_cpumask_iter_no_rcu:OK #64/44 cpumask/test_cpumask_iter_no_destroy:OK #64/45 cpumask/test_cpumask_iter_null_pointer:OK #64/46 cpumask/test_cpumask_iter_next_uninit:OK #64/47 cpumask/test_cpumask_iter_destroy_uninit:OK #64 cpumask:OK
--- tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cpumask.c | 152 ++++++++++++++++++ .../selftests/bpf/progs/cpumask_common.h | 3 + .../bpf/progs/cpumask_iter_failure.c | 99 ++++++++++++ .../bpf/progs/cpumask_iter_success.c | 126 +++++++++++++++ 5 files changed, 381 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/cpumask_iter_failure.c create mode 100644 tools/testing/selftests/bpf/progs/cpumask_iter_success.c
tools/testing/selftests/bpf/config +1 -0
@@ -78,6 +78,7 @@ CONFIG_NF_CONNTRACK_MARK=y CONFIG_NF_DEFRAG_IPV4=y CONFIG_NF_DEFRAG_IPV6=y CONFIG_NF_NAT=y +CONFIG_PSI=y CONFIG_RC_CORE=y CONFIG_SECURITY=y CONFIG_SECURITYFS=y
tools/testing/selftests/bpf/prog_tests/cpumask.c +152 -0
171 lines not on this page · raw message
tools/testing/selftests/bpf/progs/cpumask_common.h +3 -0
@@ -55,6 +55,9 @@ void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym u32 bpf_cpumask_any_distribute(const struct cpumask *src) __ksym; u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1, const struct cpumask *src2) __ksym; u32 bpf_cpumask_weight(const struct cpumask *cpumask) __ksym; +int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) __ksym; +int *bpf_iter_cpumask_next (struct bpf_iter_cpumask *it) __ksym; +void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) __ksym; void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym;
tools/testing/selftests/bpf/progs/cpumask_iter_failure.c new file +99 -0
99 lines not on this page · raw message
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +126 -0
126 lines not on this page · raw message
Signature · 1 line
2.39.1
Re: [PATCH v5 bpf-next 4/4] selftests/bpf: Add selftests for cpumask iter
Quoted 9 lines · Yafang Shao
Add selftests for the newly added cpumask iter.
-cpumask_iter_success
- The number of CPUs should be expected when iterating over the cpumask
- percpu data extracted from the percpu struct should be expected
- It can work in both non-sleepable and sleepable prog
- RCU lock is only required bybpf_iter_cpumask_new()
- It is fine without callingbpf_iter_cpumask_next()
Quoted 7 lines
-
cpumask_iter_failure
- RCU lock is required in sleepable prog
- The cpumask to be iterated over can't be NULL
-bpf_iter_cpumask_destroy()is required after callingbpf_iter_cpumask_new()
-bpf_iter_cpumask_destroy()can only destroy an initilialized iter
-bpf_iter_cpumask_next()must use an initilialized iter
typos: initialized
Quoted 17 lines
The result as follows,
#64/37 cpumask/test_cpumask_iter:OK #64/38 cpumask/test_cpumask_iter_sleepable:OK #64/39 cpumask/test_cpumask_iter_sleepable:OK #64/40 cpumask/test_cpumask_iter_next_no_rcu:OK #64/41 cpumask/test_cpumask_iter_no_next:OK #64/42 cpumask/test_cpumask_iter:OK #64/43 cpumask/test_cpumask_iter_no_rcu:OK #64/44 cpumask/test_cpumask_iter_no_destroy:OK #64/45 cpumask/test_cpumask_iter_null_pointer:OK #64/46 cpumask/test_cpumask_iter_next_uninit:OK #64/47 cpumask/test_cpumask_iter_destroy_uninit:OK #64 cpumask:OKSigned-off-by Yafang Shao <[email protected]>
Quoted 10 lines
--- tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cpumask.c | 152 ++++++++++++++++++ .../selftests/bpf/progs/cpumask_common.h | 3 + .../bpf/progs/cpumask_iter_failure.c | 99 ++++++++++++ .../bpf/progs/cpumask_iter_success.c | 126 +++++++++++++++ 5 files changed, 381 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/cpumask_iter_failure.c create mode 100644 tools/testing/selftests/bpf/progs/cpumask_iter_success.c
LGTM overall, except for seemingly unnecessary use of a big macro
tools/testing/selftests/bpf/progs/cpumask_common.h void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym · line 60
Quoted 11 lines
tools/testing/selftests/bpf/progs/cpumask_common.h +3
-0@@ -55,6 +55,9 @@ void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym u32 bpf_cpumask_any_distribute(const struct cpumask *src) __ksym; u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1, const struct cpumask *src2) __ksym; u32 bpf_cpumask_weight(const struct cpumask *cpumask) __ksym; +int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) __ksym; +int *bpf_iter_cpumask_next (struct bpf_iter_cpumask *it) __ksym; +void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) __ksym;
let's mark them __weak so they don't conflict with definitions that will eventually come from vmlinux.h (that applies to all the kfunc definitions we currently have and we'll need to clean all that up, but let's not add non-weak kfuncs going forward)
Quoted 3 lines
tools/testing/selftests/bpf/progs/cpumask_common.h +0
-0void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym;
[...]
Quoted 40 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +34
-0@@ -0,0 +1,126 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2024 Yafang Shao <[email protected]> */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +#include "task_kfunc_common.h" +#include "cpumask_common.h" + +char _license[] SEC("license") = "GPL"; + +extern const struct psi_group_cpu system_group_pcpu __ksym __weak; +extern const struct rq runqueues __ksym __weak; + +int pid; + +#define READ_PERCPU_DATA (meta, cgrp, mask) \ +{ \ + u32 nr_running = 0, psi_nr_running = 0, nr_cpus = 0; \ + struct psi_group_cpu *groupc; \ + struct rq *rq; \ + int *cpu; \ + \ + bpf_for_each(cpumask, cpu, mask) { \ + rq = (struct rq *)bpf_per_cpu_ptr(&runqueues, *cpu); \ + if (!rq) { \ + err += 1; \ + continue; \ + } \ + nr_running += rq->nr_running; \ + nr_cpus += 1; \ + \ + groupc = (struct psi_group_cpu *)bpf_per_cpu_ptr(&system_group_pcpu, *cpu); \
tools/testing/selftests/bpf/progs/cpumask_iter_success.c line 44
Quoted 10 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +10
-0+if (!groupc) { \ + err +=1 ; \ +continue ; \ + } \ + psi_nr_running += groupc->tasks[NR_RUNNING ]; \ + } \ +BPF_SEQ_PRINTF (meta->seq,"nr_running %u nr_cpus %u psi_running %u\n" , \ + nr_running, nr_cpus, psi_nr_running); \ +} +
Does this have to be a gigantic macro? Why can't it be just a function?
Quoted 22 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +22
-0+SEC("iter.s/cgroup") +int BPF_PROG (test_cpumask_iter_sleepable ,struct bpf_iter_meta *meta,struct cgroup *cgrp) +{ +struct task_struct *p; + +/* epilogue */ +if (!cgrp) +return 0 ; + +bpf_rcu_read_lock (); + p =bpf_task_from_pid (pid); +if (!p) { +bpf_rcu_read_unlock (); +return 1 ; + } + +READ_PERCPU_DATA (meta, cgrp, p->cpus_ptr); +bpf_task_release (p); +bpf_rcu_read_unlock (); +return 0 ; +} +
[...]
Re: [PATCH v5 bpf-next 4/4] selftests/bpf: Add selftests for cpumask iter
Quoted 19 lines · Andrii Nakryiko
Quoted 9 lines · Yafang Shao
Add selftests for the newly added cpumask iter.
-cpumask_iter_success
- The number of CPUs should be expected when iterating over the cpumask
- percpu data extracted from the percpu struct should be expected
- It can work in both non-sleepable and sleepable prog
- RCU lock is only required bybpf_iter_cpumask_new()
- It is fine without callingbpf_iter_cpumask_next()Quoted 7 lines
-
cpumask_iter_failure
- RCU lock is required in sleepable prog
- The cpumask to be iterated over can't be NULL
-bpf_iter_cpumask_destroy()is required after callingbpf_iter_cpumask_new()
-bpf_iter_cpumask_destroy()can only destroy an initilialized iter
-bpf_iter_cpumask_next()must use an initilialized iter
Quoted 1 line
typos: initialized
will fix it.
Quoted 31 lines
Quoted 18 lines
The result as follows,
#64/37 cpumask/test_cpumask_iter:OK #64/38 cpumask/test_cpumask_iter_sleepable:OK #64/39 cpumask/test_cpumask_iter_sleepable:OK #64/40 cpumask/test_cpumask_iter_next_no_rcu:OK #64/41 cpumask/test_cpumask_iter_no_next:OK #64/42 cpumask/test_cpumask_iter:OK #64/43 cpumask/test_cpumask_iter_no_rcu:OK #64/44 cpumask/test_cpumask_iter_no_destroy:OK #64/45 cpumask/test_cpumask_iter_null_pointer:OK #64/46 cpumask/test_cpumask_iter_next_uninit:OK #64/47 cpumask/test_cpumask_iter_destroy_uninit:OK #64 cpumask:OKSigned-off-by Yafang Shao <[email protected]>Quoted 10 lines
--- tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cpumask.c | 152 ++++++++++++++++++ .../selftests/bpf/progs/cpumask_common.h | 3 + .../bpf/progs/cpumask_iter_failure.c | 99 ++++++++++++ .../bpf/progs/cpumask_iter_success.c | 126 +++++++++++++++ 5 files changed, 381 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/cpumask_iter_failure.c create mode 100644 tools/testing/selftests/bpf/progs/cpumask_iter_success.cLGTM overall, except for seemingly unnecessary use of a big macro
Quoted 16 lines
tools/testing/selftests/bpf/progs/cpumask_common.h void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym · line 60
Quoted 11 lines
tools/testing/selftests/bpf/progs/cpumask_common.h +3
-0@@ -55,6 +55,9 @@ void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym u32 bpf_cpumask_any_distribute(const struct cpumask *src) __ksym; u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1, const struct cpumask *src2) __ksym; u32 bpf_cpumask_weight(const struct cpumask *cpumask) __ksym; +int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) __ksym; +int *bpf_iter_cpumask_next (struct bpf_iter_cpumask *it) __ksym; +void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) __ksym;let's mark them
__weakso they don't conflict with definitions that will eventually come fromvmlinux.h(that applies to all the kfunc definitions we currently have and we'll need to clean all that up, but let's not add non-weak kfuncs going forward)
will change it.
Quoted 51 lines
Quoted 4 lines
tools/testing/selftests/bpf/progs/cpumask_common.h +0
-0void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym;[...]
Quoted 44 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +38
-0@@ -0,0 +1,126 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2024 Yafang Shao <[email protected]> */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +#include "task_kfunc_common.h" +#include "cpumask_common.h" + +char _license[] SEC("license") = "GPL"; + +extern const struct psi_group_cpu system_group_pcpu __ksym __weak; +extern const struct rq runqueues __ksym __weak; + +int pid; + +#define READ_PERCPU_DATA (meta, cgrp, mask) \ +{ \ + u32 nr_running = 0, psi_nr_running = 0, nr_cpus = 0; \ + struct psi_group_cpu *groupc; \ + struct rq *rq; \ + int *cpu; \ + \ + bpf_for_each(cpumask, cpu, mask) { \ + rq = (struct rq *)bpf_per_cpu_ptr(&runqueues, *cpu); \ + if (!rq) { \ + err += 1; \ + continue; \ + } \ + nr_running += rq->nr_running; \ + nr_cpus += 1; \ + \ + groupc = (struct psi_group_cpu *)bpf_per_cpu_ptr(&system_group_pcpu, *cpu); \ + if (!groupc) { \ + err += 1; \ + continue; \ + } \
Quoted 8 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c line 44
Quoted 6 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +6
-0+ psi_nr_running += groupc->tasks[NR_RUNNING ]; \ + } \ +BPF_SEQ_PRINTF (meta->seq,"nr_running %u nr_cpus %u psi_running %u\n" , \ + nr_running, nr_cpus, psi_nr_running); \ +} +Does this have to be a gigantic macro? Why can't it be just a function?
It seems that the verifier can't identify a function call between bpf_rcu_read_lock() and bpf_rcu_read_unlock().
That said, if there's a function call between them, the verifier will fail. Below is the full verifier log if I define it as :
static inline void read_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)
VERIFIER LOG:
=============
0: R1=ctx() R10=fp0
; int BPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
0: (b4) w7 = 0 ; R7_w=0
; int BPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
1: (79) r2 = *(u64 *)(r1 +8) ; R1=ctx()
R2_w=trusted_ptr_or_null_cgroup(id=1)
; if (!cgrp)
2: (15) if r2 == 0x0 goto pc+16 ; R2_w=trusted_ptr_cgroup()
; int BPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
3: (79) r6 = *(u64 *)(r1 +0)
func 'bpf_iter_cgroup' arg0 has btf_id 10966 type STRUCT 'bpf_iter_meta' 4: R1=ctx() R6_w=trusted_ptr_bpf_iter_meta()
; bpf_rcu_read_lock();
4: (85) call bpf_rcu_read_lock#84184 ;
; p = bpf_task_from_pid(pid);
5: (18) r1 = 0xffffbc1ad3f72004 ;
R1_w=map_value(map=cpumask_.bss,ks=4,vs=8,off=4)
7: (61) r1 = *(u32 *)(r1 +0) ;
R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) ; p = bpf_task_from_pid(pid);
8: (85) call bpf_task_from_pid#84204 ;
R0=ptr_or_null_task_struct(id=3,ref_obj_id=3) refs=3
9: (bf) r8 = r0 ;
R0=ptr_or_null_task_struct(id=3,ref_obj_id=3) R8_w=ptr_or_null_task_struct(id=3,ref_obj_id=3) refs=3
10: (b4) w7 = 1 ; R7_w=1 refs=3
; if (!p) {
11: (15) if r8 == 0x0 goto pc+6 ;
R8_w=ptr_task_struct(ref_obj_id=3) refs=3
; read_percpu_data(meta, cgrp, p->cpus_ptr);
12: (79) r2 = *(u64 *)(r8 +984) ; R2_w=rcu_ptr_cpumask()
R8_w=ptr_task_struct(ref_obj_id=3) refs=3
; read_percpu_data(meta, cgrp, p->cpus_ptr);
13: (bf) r1 = r6 ;
R1_w=trusted_ptr_bpf_iter_meta() R6=trusted_ptr_bpf_iter_meta() refs=3 14: (85) call pc+6
caller:
R6=trusted_ptr_bpf_iter_meta() R7_w=1
R8_w=ptr_task_struct(ref_obj_id=3) R10=fp0 refs=3
callee:
frame1: R1_w=trusted_ptr_bpf_iter_meta() R2_w=rcu_ptr_cpumask() R10=fp0 refs=3
21: frame1: R1_w=trusted_ptr_bpf_iter_meta() R2_w=rcu_ptr_cpumask() R10=fp0 refs=3
; static inline void read_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)
21: (bf) r8 = r1 ; frame1:
R1_w=trusted_ptr_bpf_iter_meta() R8_w=trusted_ptr_bpf_iter_meta() refs=3
22: (bf) r7 = r10 ; frame1: R7_w=fp0 R10=fp0 refs=3
;
23: (07) r7 += -24 ; frame1: R7_w=fp-24 refs=3
; bpf_for_each(cpumask, cpu, mask) {
24: (bf) r1 = r7 ; frame1: R1_w=fp-24 R7_w=fp-24 refs=3
25: (85) call bpf_iter_cpumask_new#77163 ; frame1: R0=scalar()
fp-24=iter_cpumask(ref_id=4,state=active,depth=0) refs=3,4
; bpf_for_each(cpumask, cpu, mask) {
26: (bf) r1 = r7 ; frame1: R1=fp-24 R7=fp-24 refs=3,4
27: (85) call bpf_iter_cpumask_next#77165 ; frame1: R0_w=0
fp-24=iter_cpumask(ref_id=4,state=drained,depth=0) refs=3,4
28: (bf) r7 = r0 ; frame1: R0_w=0 R7_w=0 refs=3,4 29: (b4) w1 = 0 ; frame1: R1_w=0 refs=3,4 30: (63) *(u32 *)(r10 -40) = r1 ; frame1: R1_w=0 R10=fp0
fp-40=????0 refs=3,4
31: (b4) w1 = 0 ; frame1: R1_w=0 refs=3,4
32: (7b) *(u64 *)(r10 -32) = r1 ; frame1: R1_w=0 R10=fp0
fp-32_w=0 refs=3,4
33: (b4) w9 = 0 ; frame1: R9_w=0 refs=3,4
; bpf_for_each(cpumask, cpu, mask) {
34: (15) if r7 == 0x0 goto pc+57 ; frame1: R7_w=0 refs=3,4
; bpf_for_each(cpumask, cpu, mask) {
92: (bf) r1 = r10 ; frame1: R1_w=fp0 R10=fp0 refs=3,4
; bpf_for_each(cpumask, cpu, mask) {
93: (07) r1 += -24 ; frame1: R1_w=fp-24 refs=3,4
94: (85) call bpf_iter_cpumask_destroy#77161 ; frame1: refs=3
; BPF_SEQ_PRINTF(meta->seq, "nr_running %u nr_cpus %u psi_running %u\n",
95: (61) r1 = *(u32 *)(r10 -40) ; frame1: R1_w=0 R10=fp0
fp-40=????0 refs=3
96: (bc) w1 = w1 ; frame1: R1_w=0 refs=3 97: (7b) *(u64 *)(r10 -8) = r1 ; frame1: R1_w=0 R10=fp0 fp-8_w=0 refs=3 98: (79) r1 = *(u64 *)(r10 -32) ; frame1: R1_w=0 R10=fp0 fp-32=0 refs=3 99: (7b) *(u64 *)(r10 -16) = r1 ; frame1: R1_w=0 R10=fp0 fp-16_w=0 refs=3 100: (7b) *(u64 *)(r10 -24) = r9 ; frame1: R9=0 R10=fp0 fp-24_w=0 refs=3 101: (79) r1 = *(u64 *)(r8 +0) ; frame1:
R1_w=trusted_ptr_seq_file() R8=trusted_ptr_bpf_iter_meta() refs=3
102: (bf) r4 = r10 ; frame1: R4_w=fp0 R10=fp0 refs=3
; bpf_for_each(cpumask, cpu, mask) {
103: (07) r4 += -24 ; frame1: R4_w=fp-24 refs=3
; BPF_SEQ_PRINTF(meta->seq, "nr_running %u nr_cpus %u psi_running %u\n",
104: (18) r2 = 0xffff9bce47e0e210 ; frame1:
R2_w=map_value(map=cpumask_.rodata,ks=4,vs=41) refs=3
106: (b4) w3 = 41 ; frame1: R3_w=41 refs=3 107: (b4) w5 = 24 ; frame1: R5_w=24 refs=3 108: (85) call bpf_seq_printf#126 ; frame1: R0=scalar() refs=3
; }
109: (95) exitbpf_rcu_read_unlock is missing
processed 45 insns (limit 1000000) max_states_per_insn 0 total_states 5 peak_states 5 mark_read 3
=============
Another workaround is using the __always_inline :
static __always_inline void read_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)
Quoted 25 lines
Quoted 23 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +22
-0+SEC("iter.s/cgroup") +int BPF_PROG (test_cpumask_iter_sleepable ,struct bpf_iter_meta *meta,struct cgroup *cgrp) +{ +struct task_struct *p; + +/* epilogue */ +if (!cgrp) +return 0 ; + +bpf_rcu_read_lock (); + p =bpf_task_from_pid (pid); +if (!p) { +bpf_rcu_read_unlock (); +return 1 ; + } + +READ_PERCPU_DATA (meta, cgrp, p->cpus_ptr); +bpf_task_release (p); +bpf_rcu_read_unlock (); +return 0 ; +} +[...]
Signature · 2 lines
Regards Yafang
Yonghong Song
reviewerRe: [PATCH v5 bpf-next 4/4] selftests/bpf: Add selftests for cpumask iter
Quoted 226 lines · Yafang Shao
Quoted 9 lines · Andrii Nakryiko
Quoted 8 lines · Yafang Shao
Add selftests for the newly added cpumask iter.
-cpumask_iter_success
- The number of CPUs should be expected when iterating over the cpumask
- percpu data extracted from the percpu struct should be expected
- It can work in both non-sleepable and sleepable prog
- RCU lock is only required bybpf_iter_cpumask_new()
- It is fine without callingbpf_iter_cpumask_next()Quoted 8 lines
Quoted 7 lines
-
cpumask_iter_failure
- RCU lock is required in sleepable prog
- The cpumask to be iterated over can't be NULL
-bpf_iter_cpumask_destroy()is required after callingbpf_iter_cpumask_new()
-bpf_iter_cpumask_destroy()can only destroy an initilialized iter
-bpf_iter_cpumask_next()must use an initilialized itertypos: initialized
will fix it.
Quoted 28 lines
Quoted 23 lines
The result as follows,
#64/37 cpumask/test_cpumask_iter:OK #64/38 cpumask/test_cpumask_iter_sleepable:OK #64/39 cpumask/test_cpumask_iter_sleepable:OK #64/40 cpumask/test_cpumask_iter_next_no_rcu:OK #64/41 cpumask/test_cpumask_iter_no_next:OK #64/42 cpumask/test_cpumask_iter:OK #64/43 cpumask/test_cpumask_iter_no_rcu:OK #64/44 cpumask/test_cpumask_iter_no_destroy:OK #64/45 cpumask/test_cpumask_iter_null_pointer:OK #64/46 cpumask/test_cpumask_iter_next_uninit:OK #64/47 cpumask/test_cpumask_iter_destroy_uninit:OK #64 cpumask:OKSigned-off-by Yafang Shao <[email protected]>--- tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cpumask.c | 152 ++++++++++++++++++ .../selftests/bpf/progs/cpumask_common.h | 3 + .../bpf/progs/cpumask_iter_failure.c | 99 ++++++++++++ .../bpf/progs/cpumask_iter_success.c | 126 +++++++++++++++ 5 files changed, 381 insertions(+)Quoted 3 lines
create mode 100644
tools/testing/selftests/bpf/progs/cpumask_iter_failure.ccreate mode 100644tools/testing/selftests/bpf/progs/cpumask_iter_success.cLGTM overall, except for seemingly unnecessary use of a big macro
Quoted 15 lines
tools/testing/selftests/bpf/progs/cpumask_common.h void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym · line 60
Quoted 11 lines
tools/testing/selftests/bpf/progs/cpumask_common.h +3
-0@@ -55,6 +55,9 @@ void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym u32 bpf_cpumask_any_distribute(const struct cpumask *src) __ksym; u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1, const struct cpumask *src2) __ksym; u32 bpf_cpumask_weight(const struct cpumask *cpumask) __ksym; +int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) __ksym; +int *bpf_iter_cpumask_next (struct bpf_iter_cpumask *it) __ksym; +void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) __ksym;let's mark them
__weakso they don't conflict with definitions that will eventually come fromvmlinux.h(that applies to all the kfunc definitions we currently have and we'll need to clean all that up, but let's not add non-weak kfuncs going forward)will change it.
Quoted 47 lines
Quoted 2 lines
tools/testing/selftests/bpf/progs/cpumask_common.h +0
-0void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym;[...]
Quoted 43 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +37
-0@@ -0,0 +1,126 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2024 Yafang Shao <[email protected]> */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +#include "task_kfunc_common.h" +#include "cpumask_common.h" + +char _license[] SEC("license") = "GPL"; + +extern const struct psi_group_cpu system_group_pcpu __ksym __weak; +extern const struct rq runqueues __ksym __weak; + +int pid; + +#define READ_PERCPU_DATA (meta, cgrp, mask) \ +{ \ + u32 nr_running = 0, psi_nr_running = 0, nr_cpus = 0; \ + struct psi_group_cpu *groupc; \ + struct rq *rq; \ + int *cpu; \ + \ + bpf_for_each(cpumask, cpu, mask) { \ + rq = (struct rq *)bpf_per_cpu_ptr(&runqueues, *cpu); \ + if (!rq) { \ + err += 1; \ + continue; \ + } \ + nr_running += rq->nr_running; \ + nr_cpus += 1; \ + \ + groupc = (struct psi_group_cpu *)bpf_per_cpu_ptr(&system_group_pcpu, *cpu); \ + if (!groupc) { \ + err += 1; \ + continue; \Quoted 8 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c line 44
Quoted 7 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +7
-0+ } \ + psi_nr_running += groupc->tasks[NR_RUNNING ]; \ + } \ +BPF_SEQ_PRINTF (meta->seq,"nr_running %u nr_cpus %u psi_running %u\n" , \ + nr_running, nr_cpus, psi_nr_running); \ +} +Does this have to be a gigantic macro? Why can't it be just a function?
It seems that the verifier can't identify a function call between
bpf_rcu_read_lock()andbpf_rcu_read_unlock().
That said, if there's a function call between them, the verifier will fail. Below is the full verifier log if I define it as :
static inline voidread_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)VERIFIER LOG:
=============
0: R1=ctx() R10=fp0
; intBPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
0: (b4) w7 = 0 ; R7_w=0
; intBPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
1: (79) r2 = *(u64 *)(r1 +8) ; R1=ctx()
R2_w=trusted_ptr_or_null_cgroup(id=1)
; if (!cgrp)
2: (15) if r2 ==0x0goto pc+16 ; R2_w=trusted_ptr_cgroup()
; intBPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
3: (79) r6 = *(u64 *)(r1 +0)
func 'bpf_iter_cgroup' arg0 hasbtf_id10966 type STRUCT 'bpf_iter_meta' 4: R1=ctx() R6_w=trusted_ptr_bpf_iter_meta()
;bpf_rcu_read_lock();
4: (85) call bpf_rcu_read_lock#84184 ;
; p =bpf_task_from_pid(pid);
5: (18) r1 =0xffffbc1ad3f72004;
R1_w=map_value(map=cpumask_.bss,ks=4,vs=8,off=4)
7: (61) r1 = *(u32 *)(r1 +0) ;
R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0;0xffffffff)) ; p =bpf_task_from_pid(pid);
8: (85) call bpf_task_from_pid#84204 ;
R0=ptr_or_null_task_struct(id=3,ref_obj_id=3) refs=3
9: (bf) r8 = r0 ;
R0=ptr_or_null_task_struct(id=3,ref_obj_id=3) R8_w=ptr_or_null_task_struct(id=3,ref_obj_id=3) refs=3
10: (b4) w7 = 1 ; R7_w=1 refs=3
; if (!p) {
11: (15) if r8 ==0x0goto pc+6 ;
R8_w=ptr_task_struct(ref_obj_id=3) refs=3
;read_percpu_data(meta, cgrp, p->cpus_ptr);
12: (79) r2 = *(u64 *)(r8 +984) ; R2_w=rcu_ptr_cpumask()
R8_w=ptr_task_struct(ref_obj_id=3) refs=3
;read_percpu_data(meta, cgrp, p->cpus_ptr);
13: (bf) r1 = r6 ;
R1_w=trusted_ptr_bpf_iter_meta() R6=trusted_ptr_bpf_iter_meta() refs=3 14: (85) call pc+6
caller:
R6=trusted_ptr_bpf_iter_meta() R7_w=1
R8_w=ptr_task_struct(ref_obj_id=3) R10=fp0 refs=3
callee:
frame1: R1_w=trusted_ptr_bpf_iter_meta() R2_w=rcu_ptr_cpumask() R10=fp0 refs=3
21: frame1: R1_w=trusted_ptr_bpf_iter_meta() R2_w=rcu_ptr_cpumask() R10=fp0 refs=3
; static inline voidread_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)
21: (bf) r8 = r1 ; frame1:
R1_w=trusted_ptr_bpf_iter_meta() R8_w=trusted_ptr_bpf_iter_meta() refs=3
22: (bf) r7 = r10 ; frame1: R7_w=fp0 R10=fp0 refs=3
;
23: (07) r7 += -24 ; frame1: R7_w=fp-24 refs=3
;bpf_for_each(cpumask, cpu, mask){
24: (bf) r1 = r7 ; frame1: R1_w=fp-24 R7_w=fp-24 refs=3
25: (85) call bpf_iter_cpumask_new#77163 ; frame1: R0=scalar()
fp-24=iter_cpumask(ref_id=4,state=active,depth=0) refs=3,4
;bpf_for_each(cpumask, cpu, mask){
26: (bf) r1 = r7 ; frame1: R1=fp-24 R7=fp-24 refs=3,4
27: (85) call bpf_iter_cpumask_next#77165 ; frame1: R0_w=0
fp-24=iter_cpumask(ref_id=4,state=drained,depth=0) refs=3,428: (bf) r7 = r0 ; frame1: R0_w=0 R7_w=0 refs=3,4 29: (b4) w1 = 0 ; frame1: R1_w=0 refs=3,4 30: (63) *(u32 *)(r10 -40) = r1 ; frame1: R1_w=0 R10=fp0fp-40=????0 refs=3,4
31: (b4) w1 = 0 ; frame1: R1_w=0 refs=3,4
32: (7b) *(u64 *)(r10 -32) = r1 ; frame1: R1_w=0 R10=fp0
fp-32_w=0 refs=3,4
33: (b4) w9 = 0 ; frame1: R9_w=0 refs=3,4
;bpf_for_each(cpumask, cpu, mask){
34: (15) if r7 ==0x0goto pc+57 ; frame1: R7_w=0 refs=3,4
;bpf_for_each(cpumask, cpu, mask){
92: (bf) r1 = r10 ; frame1: R1_w=fp0 R10=fp0 refs=3,4
;bpf_for_each(cpumask, cpu, mask){
93: (07) r1 += -24 ; frame1: R1_w=fp-24 refs=3,4
94: (85) call bpf_iter_cpumask_destroy#77161 ; frame1: refs=3
; BPF_SEQ_PRINTF(meta->seq, "nr_running%unr_cpus%upsi_running%u\n",
95: (61) r1 = *(u32 *)(r10 -40) ; frame1: R1_w=0 R10=fp0
fp-40=????0 refs=396: (bc) w1 = w1 ; frame1: R1_w=0 refs=3 97: (7b) *(u64 *)(r10 -8) = r1 ; frame1: R1_w=0 R10=fp0 fp-8_w=0 refs=3 98: (79) r1 = *(u64 *)(r10 -32) ; frame1: R1_w=0 R10=fp0 fp-32=0 refs=3 99: (7b) *(u64 *)(r10 -16) = r1 ; frame1: R1_w=0 R10=fp0 fp-16_w=0 refs=3 100: (7b) *(u64 *)(r10 -24) = r9 ; frame1: R9=0 R10=fp0 fp-24_w=0 refs=3 101: (79) r1 = *(u64 *)(r8 +0) ; frame1:R1_w=trusted_ptr_seq_file() R8=trusted_ptr_bpf_iter_meta() refs=3
102: (bf) r4 = r10 ; frame1: R4_w=fp0 R10=fp0 refs=3
;bpf_for_each(cpumask, cpu, mask){
103: (07) r4 += -24 ; frame1: R4_w=fp-24 refs=3
; BPF_SEQ_PRINTF(meta->seq, "nr_running%unr_cpus%upsi_running%u\n",
104: (18) r2 =0xffff9bce47e0e210; frame1:
R2_w=map_value(map=cpumask_.rodata,ks=4,vs=41) refs=3106: (b4) w3 = 41 ; frame1: R3_w=41 refs=3 107: (b4) w5 = 24 ; frame1: R5_w=24 refs=3 108: (85) call bpf_seq_printf#126 ; frame1: R0=scalar() refs=3
Quoted 6 lines
; }
109: (95) exitbpf_rcu_read_unlockis missing
processed 45 insns (limit 1000000)max_states_per_insn0total_states5peak_states5mark_read3
=============
The error is due to the following in verifier:
} else if (opcode == BPF_EXIT) {
...
if (env->cur_state->active_rcu_lock &&
!in_rbtree_lock_required_cb(env)) {
verbose(env, "bpf_rcu_read_unlock is missing\n");
return -EINVAL;
}I guess, we could relax the condition not to return -EINVAL if it is a static function.
Quoted 5 lines
Another workaround is using the
__always_inline:
static__always_inlinevoidread_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)
__always_inline is also work. But let us improve verifier so we can avoid such workarounds in the future. Note that Kumar just submitted a patch set to relax spin_lock for static functions:
https://lore.kernel.org/bpf/[email protected]/
Quoted 26 lines
Quoted 24 lines
Quoted 23 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +22
-0+SEC("iter.s/cgroup") +int BPF_PROG (test_cpumask_iter_sleepable ,struct bpf_iter_meta *meta,struct cgroup *cgrp) +{ +struct task_struct *p; + +/* epilogue */ +if (!cgrp) +return 0 ; + +bpf_rcu_read_lock (); + p =bpf_task_from_pid (pid); +if (!p) { +bpf_rcu_read_unlock (); +return 1 ; + } + +READ_PERCPU_DATA (meta, cgrp, p->cpus_ptr); +bpf_task_release (p); +bpf_rcu_read_unlock (); +return 0 ; +} +[...]
Re: [PATCH v5 bpf-next 4/4] selftests/bpf: Add selftests for cpumask iter
Quoted 247 lines · Yonghong Song
Quoted 226 lines · Yafang Shao
Quoted 9 lines · Andrii Nakryiko
Quoted 8 lines · Yafang Shao
Add selftests for the newly added cpumask iter.
-cpumask_iter_success
- The number of CPUs should be expected when iterating over the cpumask
- percpu data extracted from the percpu struct should be expected
- It can work in both non-sleepable and sleepable prog
- RCU lock is only required bybpf_iter_cpumask_new()
- It is fine without callingbpf_iter_cpumask_next()Quoted 8 lines
Quoted 7 lines
-
cpumask_iter_failure
- RCU lock is required in sleepable prog
- The cpumask to be iterated over can't be NULL
-bpf_iter_cpumask_destroy()is required after callingbpf_iter_cpumask_new()
-bpf_iter_cpumask_destroy()can only destroy an initilialized iter
-bpf_iter_cpumask_next()must use an initilialized itertypos: initialized
will fix it.
Quoted 28 lines
Quoted 23 lines
The result as follows,
#64/37 cpumask/test_cpumask_iter:OK #64/38 cpumask/test_cpumask_iter_sleepable:OK #64/39 cpumask/test_cpumask_iter_sleepable:OK #64/40 cpumask/test_cpumask_iter_next_no_rcu:OK #64/41 cpumask/test_cpumask_iter_no_next:OK #64/42 cpumask/test_cpumask_iter:OK #64/43 cpumask/test_cpumask_iter_no_rcu:OK #64/44 cpumask/test_cpumask_iter_no_destroy:OK #64/45 cpumask/test_cpumask_iter_null_pointer:OK #64/46 cpumask/test_cpumask_iter_next_uninit:OK #64/47 cpumask/test_cpumask_iter_destroy_uninit:OK #64 cpumask:OKSigned-off-by Yafang Shao <[email protected]>--- tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cpumask.c | 152 ++++++++++++++++++ .../selftests/bpf/progs/cpumask_common.h | 3 + .../bpf/progs/cpumask_iter_failure.c | 99 ++++++++++++ .../bpf/progs/cpumask_iter_success.c | 126 +++++++++++++++ 5 files changed, 381 insertions(+)Quoted 3 lines
create mode 100644
tools/testing/selftests/bpf/progs/cpumask_iter_failure.ccreate mode 100644tools/testing/selftests/bpf/progs/cpumask_iter_success.cLGTM overall, except for seemingly unnecessary use of a big macro
Quoted 15 lines
tools/testing/selftests/bpf/progs/cpumask_common.h void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym · line 60
Quoted 11 lines
tools/testing/selftests/bpf/progs/cpumask_common.h +3
-0@@ -55,6 +55,9 @@ void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym u32 bpf_cpumask_any_distribute(const struct cpumask *src) __ksym; u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1, const struct cpumask *src2) __ksym; u32 bpf_cpumask_weight(const struct cpumask *cpumask) __ksym; +int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) __ksym; +int *bpf_iter_cpumask_next (struct bpf_iter_cpumask *it) __ksym; +void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) __ksym;let's mark them
__weakso they don't conflict with definitions that will eventually come fromvmlinux.h(that applies to all the kfunc definitions we currently have and we'll need to clean all that up, but let's not add non-weak kfuncs going forward)will change it.
Quoted 47 lines
Quoted 2 lines
tools/testing/selftests/bpf/progs/cpumask_common.h +0
-0void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym;[...]
Quoted 43 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +37
-0@@ -0,0 +1,126 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2024 Yafang Shao <[email protected]> */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +#include "task_kfunc_common.h" +#include "cpumask_common.h" + +char _license[] SEC("license") = "GPL"; + +extern const struct psi_group_cpu system_group_pcpu __ksym __weak; +extern const struct rq runqueues __ksym __weak; + +int pid; + +#define READ_PERCPU_DATA (meta, cgrp, mask) \ +{ \ + u32 nr_running = 0, psi_nr_running = 0, nr_cpus = 0; \ + struct psi_group_cpu *groupc; \ + struct rq *rq; \ + int *cpu; \ + \ + bpf_for_each(cpumask, cpu, mask) { \ + rq = (struct rq *)bpf_per_cpu_ptr(&runqueues, *cpu); \ + if (!rq) { \ + err += 1; \ + continue; \ + } \ + nr_running += rq->nr_running; \ + nr_cpus += 1; \ + \ + groupc = (struct psi_group_cpu *)bpf_per_cpu_ptr(&system_group_pcpu, *cpu); \ + if (!groupc) { \ + err += 1; \ + continue; \Quoted 8 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c line 44
Quoted 7 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +7
-0+ } \ + psi_nr_running += groupc->tasks[NR_RUNNING ]; \ + } \ +BPF_SEQ_PRINTF (meta->seq,"nr_running %u nr_cpus %u psi_running %u\n" , \ + nr_running, nr_cpus, psi_nr_running); \ +} +Does this have to be a gigantic macro? Why can't it be just a function?
It seems that the verifier can't identify a function call between
bpf_rcu_read_lock()andbpf_rcu_read_unlock().
That said, if there's a function call between them, the verifier will fail. Below is the full verifier log if I define it as :
static inline voidread_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)VERIFIER LOG:
=============
0: R1=ctx() R10=fp0
; intBPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
0: (b4) w7 = 0 ; R7_w=0
; intBPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
1: (79) r2 = *(u64 *)(r1 +8) ; R1=ctx()
R2_w=trusted_ptr_or_null_cgroup(id=1)
; if (!cgrp)
2: (15) if r2 ==0x0goto pc+16 ; R2_w=trusted_ptr_cgroup()
; intBPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
3: (79) r6 = *(u64 *)(r1 +0)
func 'bpf_iter_cgroup' arg0 hasbtf_id10966 type STRUCT 'bpf_iter_meta' 4: R1=ctx() R6_w=trusted_ptr_bpf_iter_meta()
;bpf_rcu_read_lock();
4: (85) call bpf_rcu_read_lock#84184 ;
; p =bpf_task_from_pid(pid);
5: (18) r1 =0xffffbc1ad3f72004;
R1_w=map_value(map=cpumask_.bss,ks=4,vs=8,off=4)
7: (61) r1 = *(u32 *)(r1 +0) ;
R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0;0xffffffff)) ; p =bpf_task_from_pid(pid);
8: (85) call bpf_task_from_pid#84204 ;
R0=ptr_or_null_task_struct(id=3,ref_obj_id=3) refs=3
9: (bf) r8 = r0 ;
R0=ptr_or_null_task_struct(id=3,ref_obj_id=3) R8_w=ptr_or_null_task_struct(id=3,ref_obj_id=3) refs=3
10: (b4) w7 = 1 ; R7_w=1 refs=3
; if (!p) {
11: (15) if r8 ==0x0goto pc+6 ;
R8_w=ptr_task_struct(ref_obj_id=3) refs=3
;read_percpu_data(meta, cgrp, p->cpus_ptr);
12: (79) r2 = *(u64 *)(r8 +984) ; R2_w=rcu_ptr_cpumask()
R8_w=ptr_task_struct(ref_obj_id=3) refs=3
;read_percpu_data(meta, cgrp, p->cpus_ptr);
13: (bf) r1 = r6 ;
R1_w=trusted_ptr_bpf_iter_meta() R6=trusted_ptr_bpf_iter_meta() refs=3 14: (85) call pc+6
caller:
R6=trusted_ptr_bpf_iter_meta() R7_w=1
R8_w=ptr_task_struct(ref_obj_id=3) R10=fp0 refs=3
callee:
frame1: R1_w=trusted_ptr_bpf_iter_meta() R2_w=rcu_ptr_cpumask() R10=fp0 refs=3
21: frame1: R1_w=trusted_ptr_bpf_iter_meta() R2_w=rcu_ptr_cpumask() R10=fp0 refs=3
; static inline voidread_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)
21: (bf) r8 = r1 ; frame1:
R1_w=trusted_ptr_bpf_iter_meta() R8_w=trusted_ptr_bpf_iter_meta() refs=3
22: (bf) r7 = r10 ; frame1: R7_w=fp0 R10=fp0 refs=3
;
23: (07) r7 += -24 ; frame1: R7_w=fp-24 refs=3
;bpf_for_each(cpumask, cpu, mask){
24: (bf) r1 = r7 ; frame1: R1_w=fp-24 R7_w=fp-24 refs=3
25: (85) call bpf_iter_cpumask_new#77163 ; frame1: R0=scalar()
fp-24=iter_cpumask(ref_id=4,state=active,depth=0) refs=3,4
;bpf_for_each(cpumask, cpu, mask){
26: (bf) r1 = r7 ; frame1: R1=fp-24 R7=fp-24 refs=3,4
27: (85) call bpf_iter_cpumask_next#77165 ; frame1: R0_w=0
fp-24=iter_cpumask(ref_id=4,state=drained,depth=0) refs=3,428: (bf) r7 = r0 ; frame1: R0_w=0 R7_w=0 refs=3,4 29: (b4) w1 = 0 ; frame1: R1_w=0 refs=3,4 30: (63) *(u32 *)(r10 -40) = r1 ; frame1: R1_w=0 R10=fp0fp-40=????0 refs=3,4
31: (b4) w1 = 0 ; frame1: R1_w=0 refs=3,4
32: (7b) *(u64 *)(r10 -32) = r1 ; frame1: R1_w=0 R10=fp0
fp-32_w=0 refs=3,4
33: (b4) w9 = 0 ; frame1: R9_w=0 refs=3,4
;bpf_for_each(cpumask, cpu, mask){
34: (15) if r7 ==0x0goto pc+57 ; frame1: R7_w=0 refs=3,4
;bpf_for_each(cpumask, cpu, mask){
92: (bf) r1 = r10 ; frame1: R1_w=fp0 R10=fp0 refs=3,4
;bpf_for_each(cpumask, cpu, mask){
93: (07) r1 += -24 ; frame1: R1_w=fp-24 refs=3,4
94: (85) call bpf_iter_cpumask_destroy#77161 ; frame1: refs=3
; BPF_SEQ_PRINTF(meta->seq, "nr_running%unr_cpus%upsi_running%u\n",
95: (61) r1 = *(u32 *)(r10 -40) ; frame1: R1_w=0 R10=fp0
fp-40=????0 refs=396: (bc) w1 = w1 ; frame1: R1_w=0 refs=3 97: (7b) *(u64 *)(r10 -8) = r1 ; frame1: R1_w=0 R10=fp0 fp-8_w=0 refs=3 98: (79) r1 = *(u64 *)(r10 -32) ; frame1: R1_w=0 R10=fp0 fp-32=0 refs=3 99: (7b) *(u64 *)(r10 -16) = r1 ; frame1: R1_w=0 R10=fp0 fp-16_w=0 refs=3 100: (7b) *(u64 *)(r10 -24) = r9 ; frame1: R9=0 R10=fp0 fp-24_w=0 refs=3 101: (79) r1 = *(u64 *)(r8 +0) ; frame1:R1_w=trusted_ptr_seq_file() R8=trusted_ptr_bpf_iter_meta() refs=3
102: (bf) r4 = r10 ; frame1: R4_w=fp0 R10=fp0 refs=3
;bpf_for_each(cpumask, cpu, mask){
103: (07) r4 += -24 ; frame1: R4_w=fp-24 refs=3
; BPF_SEQ_PRINTF(meta->seq, "nr_running%unr_cpus%upsi_running%u\n",
104: (18) r2 =0xffff9bce47e0e210; frame1:
R2_w=map_value(map=cpumask_.rodata,ks=4,vs=41) refs=3106: (b4) w3 = 41 ; frame1: R3_w=41 refs=3 107: (b4) w5 = 24 ; frame1: R5_w=24 refs=3 108: (85) call bpf_seq_printf#126 ; frame1: R0=scalar() refs=3Quoted 6 lines
; }
109: (95) exitbpf_rcu_read_unlockis missing
processed 45 insns (limit 1000000)max_states_per_insn0total_states5peak_states5mark_read3
=============The error is due to the following in verifier:
} else if (opcode == BPF_EXIT) { ... if (env->cur_state->active_rcu_lock && !in_rbtree_lock_required_cb(env)) { verbose(env, "bpf_rcu_read_unlock is missing\n"); return -EINVAL; }
Quoted 2 lines
I guess, we could relax the condition not to return
-EINVALif it is a static function.
Thanks for your suggestion.
It seems Kumar has already submitted a fix for it:
https://lore.kernel.org/bpf/[email protected]/
Quoted 38 lines
Quoted 6 lines
Another workaround is using the
__always_inline:
static__always_inlinevoidread_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)
__always_inlineis also work. But let us improve verifier so we can avoid such workarounds in the future. Note that Kumar just submitted a patch set to relaxspin_lockfor static functions:https://lore.kernel.org/bpf/[email protected]/Quoted 26 lines
Quoted 24 lines
Quoted 23 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +22
-0+SEC("iter.s/cgroup") +int BPF_PROG (test_cpumask_iter_sleepable ,struct bpf_iter_meta *meta,struct cgroup *cgrp) +{ +struct task_struct *p; + +/* epilogue */ +if (!cgrp) +return 0 ; + +bpf_rcu_read_lock (); + p =bpf_task_from_pid (pid); +if (!p) { +bpf_rcu_read_unlock (); +return 1 ; + } + +READ_PERCPU_DATA (meta, cgrp, p->cpus_ptr); +bpf_task_release (p); +bpf_rcu_read_unlock (); +return 0 ; +} +[...]
Signature · 2 lines
Regards Yafang
Re: [PATCH v5 bpf-next 4/4] selftests/bpf: Add selftests for cpumask iter
In reply to Yonghong SongQuoted 252 lines · Yonghong Song
Quoted 226 lines · Yafang Shao
Quoted 9 lines · Andrii Nakryiko
Quoted 8 lines · Yafang Shao
Add selftests for the newly added cpumask iter.
-cpumask_iter_success
- The number of CPUs should be expected when iterating over the cpumask
- percpu data extracted from the percpu struct should be expected
- It can work in both non-sleepable and sleepable prog
- RCU lock is only required bybpf_iter_cpumask_new()
- It is fine without callingbpf_iter_cpumask_next()Quoted 8 lines
Quoted 7 lines
-
cpumask_iter_failure
- RCU lock is required in sleepable prog
- The cpumask to be iterated over can't be NULL
-bpf_iter_cpumask_destroy()is required after callingbpf_iter_cpumask_new()
-bpf_iter_cpumask_destroy()can only destroy an initilialized iter
-bpf_iter_cpumask_next()must use an initilialized itertypos: initialized
will fix it.
Quoted 28 lines
Quoted 23 lines
The result as follows,
#64/37 cpumask/test_cpumask_iter:OK #64/38 cpumask/test_cpumask_iter_sleepable:OK #64/39 cpumask/test_cpumask_iter_sleepable:OK #64/40 cpumask/test_cpumask_iter_next_no_rcu:OK #64/41 cpumask/test_cpumask_iter_no_next:OK #64/42 cpumask/test_cpumask_iter:OK #64/43 cpumask/test_cpumask_iter_no_rcu:OK #64/44 cpumask/test_cpumask_iter_no_destroy:OK #64/45 cpumask/test_cpumask_iter_null_pointer:OK #64/46 cpumask/test_cpumask_iter_next_uninit:OK #64/47 cpumask/test_cpumask_iter_destroy_uninit:OK #64 cpumask:OKSigned-off-by Yafang Shao <[email protected]>--- tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cpumask.c | 152 ++++++++++++++++++ .../selftests/bpf/progs/cpumask_common.h | 3 + .../bpf/progs/cpumask_iter_failure.c | 99 ++++++++++++ .../bpf/progs/cpumask_iter_success.c | 126 +++++++++++++++ 5 files changed, 381 insertions(+)Quoted 3 lines
create mode 100644
tools/testing/selftests/bpf/progs/cpumask_iter_failure.ccreate mode 100644tools/testing/selftests/bpf/progs/cpumask_iter_success.cLGTM overall, except for seemingly unnecessary use of a big macro
Quoted 15 lines
tools/testing/selftests/bpf/progs/cpumask_common.h void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym · line 60
Quoted 11 lines
tools/testing/selftests/bpf/progs/cpumask_common.h +3
-0@@ -55,6 +55,9 @@ void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym u32 bpf_cpumask_any_distribute(const struct cpumask *src) __ksym; u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1, const struct cpumask *src2) __ksym; u32 bpf_cpumask_weight(const struct cpumask *cpumask) __ksym; +int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) __ksym; +int *bpf_iter_cpumask_next (struct bpf_iter_cpumask *it) __ksym; +void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) __ksym;let's mark them
__weakso they don't conflict with definitions that will eventually come fromvmlinux.h(that applies to all the kfunc definitions we currently have and we'll need to clean all that up, but let's not add non-weak kfuncs going forward)will change it.
Quoted 47 lines
Quoted 2 lines
tools/testing/selftests/bpf/progs/cpumask_common.h +0
-0void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym;[...]
Quoted 43 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +37
-0@@ -0,0 +1,126 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2024 Yafang Shao <[email protected]> */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +#include "task_kfunc_common.h" +#include "cpumask_common.h" + +char _license[] SEC("license") = "GPL"; + +extern const struct psi_group_cpu system_group_pcpu __ksym __weak; +extern const struct rq runqueues __ksym __weak; + +int pid; + +#define READ_PERCPU_DATA (meta, cgrp, mask) \ +{ \ + u32 nr_running = 0, psi_nr_running = 0, nr_cpus = 0; \ + struct psi_group_cpu *groupc; \ + struct rq *rq; \ + int *cpu; \ + \ + bpf_for_each(cpumask, cpu, mask) { \ + rq = (struct rq *)bpf_per_cpu_ptr(&runqueues, *cpu); \ + if (!rq) { \ + err += 1; \ + continue; \ + } \ + nr_running += rq->nr_running; \ + nr_cpus += 1; \ + \ + groupc = (struct psi_group_cpu *)bpf_per_cpu_ptr(&system_group_pcpu, *cpu); \ + if (!groupc) { \ + err += 1; \ + continue; \Quoted 8 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c line 44
Quoted 7 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +7
-0+ } \ + psi_nr_running += groupc->tasks[NR_RUNNING ]; \ + } \ +BPF_SEQ_PRINTF (meta->seq,"nr_running %u nr_cpus %u psi_running %u\n" , \ + nr_running, nr_cpus, psi_nr_running); \ +} +Does this have to be a gigantic macro? Why can't it be just a function?
It seems that the verifier can't identify a function call between
bpf_rcu_read_lock()andbpf_rcu_read_unlock().
That said, if there's a function call between them, the verifier will fail. Below is the full verifier log if I define it as :
static inline voidread_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)VERIFIER LOG:
=============
0: R1=ctx() R10=fp0
; intBPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
0: (b4) w7 = 0 ; R7_w=0
; intBPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
1: (79) r2 = *(u64 *)(r1 +8) ; R1=ctx()
R2_w=trusted_ptr_or_null_cgroup(id=1)
; if (!cgrp)
2: (15) if r2 ==0x0goto pc+16 ; R2_w=trusted_ptr_cgroup()
; intBPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
3: (79) r6 = *(u64 *)(r1 +0)
func 'bpf_iter_cgroup' arg0 hasbtf_id10966 type STRUCT 'bpf_iter_meta' 4: R1=ctx() R6_w=trusted_ptr_bpf_iter_meta()
;bpf_rcu_read_lock();
4: (85) call bpf_rcu_read_lock#84184 ;
; p =bpf_task_from_pid(pid);
5: (18) r1 =0xffffbc1ad3f72004;
R1_w=map_value(map=cpumask_.bss,ks=4,vs=8,off=4)
7: (61) r1 = *(u32 *)(r1 +0) ;
R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0;0xffffffff)) ; p =bpf_task_from_pid(pid);
8: (85) call bpf_task_from_pid#84204 ;
R0=ptr_or_null_task_struct(id=3,ref_obj_id=3) refs=3
9: (bf) r8 = r0 ;
R0=ptr_or_null_task_struct(id=3,ref_obj_id=3) R8_w=ptr_or_null_task_struct(id=3,ref_obj_id=3) refs=3
10: (b4) w7 = 1 ; R7_w=1 refs=3
; if (!p) {
11: (15) if r8 ==0x0goto pc+6 ;
R8_w=ptr_task_struct(ref_obj_id=3) refs=3
;read_percpu_data(meta, cgrp, p->cpus_ptr);
12: (79) r2 = *(u64 *)(r8 +984) ; R2_w=rcu_ptr_cpumask()
R8_w=ptr_task_struct(ref_obj_id=3) refs=3
;read_percpu_data(meta, cgrp, p->cpus_ptr);
13: (bf) r1 = r6 ;
R1_w=trusted_ptr_bpf_iter_meta() R6=trusted_ptr_bpf_iter_meta() refs=3 14: (85) call pc+6
caller:
R6=trusted_ptr_bpf_iter_meta() R7_w=1
R8_w=ptr_task_struct(ref_obj_id=3) R10=fp0 refs=3
callee:
frame1: R1_w=trusted_ptr_bpf_iter_meta() R2_w=rcu_ptr_cpumask() R10=fp0 refs=3
21: frame1: R1_w=trusted_ptr_bpf_iter_meta() R2_w=rcu_ptr_cpumask() R10=fp0 refs=3
; static inline voidread_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)
21: (bf) r8 = r1 ; frame1:
R1_w=trusted_ptr_bpf_iter_meta() R8_w=trusted_ptr_bpf_iter_meta() refs=3
22: (bf) r7 = r10 ; frame1: R7_w=fp0 R10=fp0 refs=3
;
23: (07) r7 += -24 ; frame1: R7_w=fp-24 refs=3
;bpf_for_each(cpumask, cpu, mask){
24: (bf) r1 = r7 ; frame1: R1_w=fp-24 R7_w=fp-24 refs=3
25: (85) call bpf_iter_cpumask_new#77163 ; frame1: R0=scalar()
fp-24=iter_cpumask(ref_id=4,state=active,depth=0) refs=3,4
;bpf_for_each(cpumask, cpu, mask){
26: (bf) r1 = r7 ; frame1: R1=fp-24 R7=fp-24 refs=3,4
27: (85) call bpf_iter_cpumask_next#77165 ; frame1: R0_w=0
fp-24=iter_cpumask(ref_id=4,state=drained,depth=0) refs=3,428: (bf) r7 = r0 ; frame1: R0_w=0 R7_w=0 refs=3,4 29: (b4) w1 = 0 ; frame1: R1_w=0 refs=3,4 30: (63) *(u32 *)(r10 -40) = r1 ; frame1: R1_w=0 R10=fp0fp-40=????0 refs=3,4
31: (b4) w1 = 0 ; frame1: R1_w=0 refs=3,4
32: (7b) *(u64 *)(r10 -32) = r1 ; frame1: R1_w=0 R10=fp0
fp-32_w=0 refs=3,4
33: (b4) w9 = 0 ; frame1: R9_w=0 refs=3,4
;bpf_for_each(cpumask, cpu, mask){
34: (15) if r7 ==0x0goto pc+57 ; frame1: R7_w=0 refs=3,4
;bpf_for_each(cpumask, cpu, mask){
92: (bf) r1 = r10 ; frame1: R1_w=fp0 R10=fp0 refs=3,4
;bpf_for_each(cpumask, cpu, mask){
93: (07) r1 += -24 ; frame1: R1_w=fp-24 refs=3,4
94: (85) call bpf_iter_cpumask_destroy#77161 ; frame1: refs=3
; BPF_SEQ_PRINTF(meta->seq, "nr_running%unr_cpus%upsi_running%u\n",
95: (61) r1 = *(u32 *)(r10 -40) ; frame1: R1_w=0 R10=fp0
fp-40=????0 refs=396: (bc) w1 = w1 ; frame1: R1_w=0 refs=3 97: (7b) *(u64 *)(r10 -8) = r1 ; frame1: R1_w=0 R10=fp0 fp-8_w=0 refs=3 98: (79) r1 = *(u64 *)(r10 -32) ; frame1: R1_w=0 R10=fp0 fp-32=0 refs=3 99: (7b) *(u64 *)(r10 -16) = r1 ; frame1: R1_w=0 R10=fp0 fp-16_w=0 refs=3 100: (7b) *(u64 *)(r10 -24) = r9 ; frame1: R9=0 R10=fp0 fp-24_w=0 refs=3 101: (79) r1 = *(u64 *)(r8 +0) ; frame1:R1_w=trusted_ptr_seq_file() R8=trusted_ptr_bpf_iter_meta() refs=3
102: (bf) r4 = r10 ; frame1: R4_w=fp0 R10=fp0 refs=3
;bpf_for_each(cpumask, cpu, mask){
103: (07) r4 += -24 ; frame1: R4_w=fp-24 refs=3
; BPF_SEQ_PRINTF(meta->seq, "nr_running%unr_cpus%upsi_running%u\n",
104: (18) r2 =0xffff9bce47e0e210; frame1:
R2_w=map_value(map=cpumask_.rodata,ks=4,vs=41) refs=3106: (b4) w3 = 41 ; frame1: R3_w=41 refs=3 107: (b4) w5 = 24 ; frame1: R5_w=24 refs=3 108: (85) call bpf_seq_printf#126 ; frame1: R0=scalar() refs=3Quoted 6 lines
; }
109: (95) exitbpf_rcu_read_unlockis missing
processed 45 insns (limit 1000000)max_states_per_insn0total_states5peak_states5mark_read3
=============The error is due to the following in verifier:
} else if (opcode == BPF_EXIT) { ... if (env->cur_state->active_rcu_lock && !in_rbtree_lock_required_cb(env)) { verbose(env, "bpf_rcu_read_unlock is missing\n"); return -EINVAL; }I guess, we could relax the condition not to return
-EINVALif it is a static function.Quoted 2 lines
Quoted 8 lines
Quoted 3 lines
Another workaround is using the
__always_inline:
static__always_inlinevoidread_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)
__always_inlineis also work. But let us improve verifier so we can avoid such workarounds in the future. Note that Kumar just submitted a patch set to relaxspin_lockfor static functions:https://lore.kernel.org/bpf/[email protected]/
Agreed, let's improve verifier, but I wouldn't block on it for this patch set and just use __always_inline for now.
I think we should also work on extending this RCU support to global functions. We can add per-function annotation (similar to __arg_xxx, but which will be applied to a function itself), just __rcu or something like __func_assume_rcu or whatnot, and then there should be no difference between static and global functions in this regard.
In general, global functions are basically mandatory nowadays for big production BPF programs (to reduce verification complexity), so we should strive to keep global functions/subprogs on par with static subprogs as much as we can.
Quoted 27 lines
Quoted 27 lines
Quoted 25 lines
Quoted 24 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +22
-0+SEC("iter.s/cgroup") +int BPF_PROG (test_cpumask_iter_sleepable ,struct bpf_iter_meta *meta,struct cgroup *cgrp) +{ +struct task_struct *p; + +/* epilogue */ +if (!cgrp) +return 0 ; + +bpf_rcu_read_lock (); + p =bpf_task_from_pid (pid); +if (!p) { +bpf_rcu_read_unlock (); +return 1 ; + } + +READ_PERCPU_DATA (meta, cgrp, p->cpus_ptr); +bpf_task_release (p); +bpf_rcu_read_unlock (); +return 0 ; +} +[...]
Re: [PATCH v5 bpf-next 4/4] selftests/bpf: Add selftests for cpumask iter
Quoted 258 lines · Andrii Nakryiko
Quoted 256 lines · Yonghong Song
Quoted 226 lines · Yafang Shao
Quoted 9 lines · Andrii Nakryiko
Quoted 8 lines · Yafang Shao
Add selftests for the newly added cpumask iter.
-cpumask_iter_success
- The number of CPUs should be expected when iterating over the cpumask
- percpu data extracted from the percpu struct should be expected
- It can work in both non-sleepable and sleepable prog
- RCU lock is only required bybpf_iter_cpumask_new()
- It is fine without callingbpf_iter_cpumask_next()Quoted 8 lines
Quoted 7 lines
-
cpumask_iter_failure
- RCU lock is required in sleepable prog
- The cpumask to be iterated over can't be NULL
-bpf_iter_cpumask_destroy()is required after callingbpf_iter_cpumask_new()
-bpf_iter_cpumask_destroy()can only destroy an initilialized iter
-bpf_iter_cpumask_next()must use an initilialized itertypos: initialized
will fix it.
Quoted 28 lines
Quoted 23 lines
The result as follows,
#64/37 cpumask/test_cpumask_iter:OK #64/38 cpumask/test_cpumask_iter_sleepable:OK #64/39 cpumask/test_cpumask_iter_sleepable:OK #64/40 cpumask/test_cpumask_iter_next_no_rcu:OK #64/41 cpumask/test_cpumask_iter_no_next:OK #64/42 cpumask/test_cpumask_iter:OK #64/43 cpumask/test_cpumask_iter_no_rcu:OK #64/44 cpumask/test_cpumask_iter_no_destroy:OK #64/45 cpumask/test_cpumask_iter_null_pointer:OK #64/46 cpumask/test_cpumask_iter_next_uninit:OK #64/47 cpumask/test_cpumask_iter_destroy_uninit:OK #64 cpumask:OKSigned-off-by Yafang Shao <[email protected]>--- tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cpumask.c | 152 ++++++++++++++++++ .../selftests/bpf/progs/cpumask_common.h | 3 + .../bpf/progs/cpumask_iter_failure.c | 99 ++++++++++++ .../bpf/progs/cpumask_iter_success.c | 126 +++++++++++++++ 5 files changed, 381 insertions(+)Quoted 3 lines
create mode 100644
tools/testing/selftests/bpf/progs/cpumask_iter_failure.ccreate mode 100644tools/testing/selftests/bpf/progs/cpumask_iter_success.cLGTM overall, except for seemingly unnecessary use of a big macro
Quoted 15 lines
tools/testing/selftests/bpf/progs/cpumask_common.h void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym · line 60
Quoted 11 lines
tools/testing/selftests/bpf/progs/cpumask_common.h +3
-0@@ -55,6 +55,9 @@ void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym u32 bpf_cpumask_any_distribute(const struct cpumask *src) __ksym; u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1, const struct cpumask *src2) __ksym; u32 bpf_cpumask_weight(const struct cpumask *cpumask) __ksym; +int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) __ksym; +int *bpf_iter_cpumask_next (struct bpf_iter_cpumask *it) __ksym; +void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) __ksym;let's mark them
__weakso they don't conflict with definitions that will eventually come fromvmlinux.h(that applies to all the kfunc definitions we currently have and we'll need to clean all that up, but let's not add non-weak kfuncs going forward)will change it.
Quoted 47 lines
Quoted 2 lines
tools/testing/selftests/bpf/progs/cpumask_common.h +0
-0void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym;[...]
Quoted 43 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +37
-0@@ -0,0 +1,126 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2024 Yafang Shao <[email protected]> */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +#include "task_kfunc_common.h" +#include "cpumask_common.h" + +char _license[] SEC("license") = "GPL"; + +extern const struct psi_group_cpu system_group_pcpu __ksym __weak; +extern const struct rq runqueues __ksym __weak; + +int pid; + +#define READ_PERCPU_DATA (meta, cgrp, mask) \ +{ \ + u32 nr_running = 0, psi_nr_running = 0, nr_cpus = 0; \ + struct psi_group_cpu *groupc; \ + struct rq *rq; \ + int *cpu; \ + \ + bpf_for_each(cpumask, cpu, mask) { \ + rq = (struct rq *)bpf_per_cpu_ptr(&runqueues, *cpu); \ + if (!rq) { \ + err += 1; \ + continue; \ + } \ + nr_running += rq->nr_running; \ + nr_cpus += 1; \ + \ + groupc = (struct psi_group_cpu *)bpf_per_cpu_ptr(&system_group_pcpu, *cpu); \ + if (!groupc) { \ + err += 1; \ + continue; \Quoted 8 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c line 44
Quoted 7 lines
tools/testing/selftests/bpf/progs/cpumask_iter_success.c new file +7
-0+ } \ + psi_nr_running += groupc->tasks[NR_RUNNING ]; \ + } \ +BPF_SEQ_PRINTF (meta->seq,"nr_running %u nr_cpus %u psi_running %u\n" , \ + nr_running, nr_cpus, psi_nr_running); \ +} +Does this have to be a gigantic macro? Why can't it be just a function?
It seems that the verifier can't identify a function call between
bpf_rcu_read_lock()andbpf_rcu_read_unlock().
That said, if there's a function call between them, the verifier will fail. Below is the full verifier log if I define it as :
static inline voidread_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)VERIFIER LOG:
=============
0: R1=ctx() R10=fp0
; intBPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
0: (b4) w7 = 0 ; R7_w=0
; intBPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
1: (79) r2 = *(u64 *)(r1 +8) ; R1=ctx()
R2_w=trusted_ptr_or_null_cgroup(id=1)
; if (!cgrp)
2: (15) if r2 ==0x0goto pc+16 ; R2_w=trusted_ptr_cgroup()
; intBPF_PROG(test_cpumask_iter_sleepable, struct bpf_iter_meta *meta, struct cgroup *cgrp)
3: (79) r6 = *(u64 *)(r1 +0)
func 'bpf_iter_cgroup' arg0 hasbtf_id10966 type STRUCT 'bpf_iter_meta' 4: R1=ctx() R6_w=trusted_ptr_bpf_iter_meta()
;bpf_rcu_read_lock();
4: (85) call bpf_rcu_read_lock#84184 ;
; p =bpf_task_from_pid(pid);
5: (18) r1 =0xffffbc1ad3f72004;
R1_w=map_value(map=cpumask_.bss,ks=4,vs=8,off=4)
7: (61) r1 = *(u32 *)(r1 +0) ;
R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0;0xffffffff)) ; p =bpf_task_from_pid(pid);
8: (85) call bpf_task_from_pid#84204 ;
R0=ptr_or_null_task_struct(id=3,ref_obj_id=3) refs=3
9: (bf) r8 = r0 ;
R0=ptr_or_null_task_struct(id=3,ref_obj_id=3) R8_w=ptr_or_null_task_struct(id=3,ref_obj_id=3) refs=3
10: (b4) w7 = 1 ; R7_w=1 refs=3
; if (!p) {
11: (15) if r8 ==0x0goto pc+6 ;
R8_w=ptr_task_struct(ref_obj_id=3) refs=3
;read_percpu_data(meta, cgrp, p->cpus_ptr);
12: (79) r2 = *(u64 *)(r8 +984) ; R2_w=rcu_ptr_cpumask()
R8_w=ptr_task_struct(ref_obj_id=3) refs=3
;read_percpu_data(meta, cgrp, p->cpus_ptr);
13: (bf) r1 = r6 ;
R1_w=trusted_ptr_bpf_iter_meta() R6=trusted_ptr_bpf_iter_meta() refs=3 14: (85) call pc+6
caller:
R6=trusted_ptr_bpf_iter_meta() R7_w=1
R8_w=ptr_task_struct(ref_obj_id=3) R10=fp0 refs=3
callee:
frame1: R1_w=trusted_ptr_bpf_iter_meta() R2_w=rcu_ptr_cpumask() R10=fp0 refs=3
21: frame1: R1_w=trusted_ptr_bpf_iter_meta() R2_w=rcu_ptr_cpumask() R10=fp0 refs=3
; static inline voidread_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)
21: (bf) r8 = r1 ; frame1:
R1_w=trusted_ptr_bpf_iter_meta() R8_w=trusted_ptr_bpf_iter_meta() refs=3
22: (bf) r7 = r10 ; frame1: R7_w=fp0 R10=fp0 refs=3
;
23: (07) r7 += -24 ; frame1: R7_w=fp-24 refs=3
;bpf_for_each(cpumask, cpu, mask){
24: (bf) r1 = r7 ; frame1: R1_w=fp-24 R7_w=fp-24 refs=3
25: (85) call bpf_iter_cpumask_new#77163 ; frame1: R0=scalar()
fp-24=iter_cpumask(ref_id=4,state=active,depth=0) refs=3,4
;bpf_for_each(cpumask, cpu, mask){
26: (bf) r1 = r7 ; frame1: R1=fp-24 R7=fp-24 refs=3,4
27: (85) call bpf_iter_cpumask_next#77165 ; frame1: R0_w=0
fp-24=iter_cpumask(ref_id=4,state=drained,depth=0) refs=3,428: (bf) r7 = r0 ; frame1: R0_w=0 R7_w=0 refs=3,4 29: (b4) w1 = 0 ; frame1: R1_w=0 refs=3,4 30: (63) *(u32 *)(r10 -40) = r1 ; frame1: R1_w=0 R10=fp0fp-40=????0 refs=3,4
31: (b4) w1 = 0 ; frame1: R1_w=0 refs=3,4
32: (7b) *(u64 *)(r10 -32) = r1 ; frame1: R1_w=0 R10=fp0
fp-32_w=0 refs=3,4
33: (b4) w9 = 0 ; frame1: R9_w=0 refs=3,4
;bpf_for_each(cpumask, cpu, mask){
34: (15) if r7 ==0x0goto pc+57 ; frame1: R7_w=0 refs=3,4
;bpf_for_each(cpumask, cpu, mask){
92: (bf) r1 = r10 ; frame1: R1_w=fp0 R10=fp0 refs=3,4
;bpf_for_each(cpumask, cpu, mask){
93: (07) r1 += -24 ; frame1: R1_w=fp-24 refs=3,4
94: (85) call bpf_iter_cpumask_destroy#77161 ; frame1: refs=3
; BPF_SEQ_PRINTF(meta->seq, "nr_running%unr_cpus%upsi_running%u\n",
95: (61) r1 = *(u32 *)(r10 -40) ; frame1: R1_w=0 R10=fp0
fp-40=????0 refs=396: (bc) w1 = w1 ; frame1: R1_w=0 refs=3 97: (7b) *(u64 *)(r10 -8) = r1 ; frame1: R1_w=0 R10=fp0 fp-8_w=0 refs=3 98: (79) r1 = *(u64 *)(r10 -32) ; frame1: R1_w=0 R10=fp0 fp-32=0 refs=3 99: (7b) *(u64 *)(r10 -16) = r1 ; frame1: R1_w=0 R10=fp0 fp-16_w=0 refs=3 100: (7b) *(u64 *)(r10 -24) = r9 ; frame1: R9=0 R10=fp0 fp-24_w=0 refs=3 101: (79) r1 = *(u64 *)(r8 +0) ; frame1:R1_w=trusted_ptr_seq_file() R8=trusted_ptr_bpf_iter_meta() refs=3
102: (bf) r4 = r10 ; frame1: R4_w=fp0 R10=fp0 refs=3
;bpf_for_each(cpumask, cpu, mask){
103: (07) r4 += -24 ; frame1: R4_w=fp-24 refs=3
; BPF_SEQ_PRINTF(meta->seq, "nr_running%unr_cpus%upsi_running%u\n",
104: (18) r2 =0xffff9bce47e0e210; frame1:
R2_w=map_value(map=cpumask_.rodata,ks=4,vs=41) refs=3106: (b4) w3 = 41 ; frame1: R3_w=41 refs=3 107: (b4) w5 = 24 ; frame1: R5_w=24 refs=3 108: (85) call bpf_seq_printf#126 ; frame1: R0=scalar() refs=3Quoted 6 lines
; }
109: (95) exitbpf_rcu_read_unlockis missing
processed 45 insns (limit 1000000)max_states_per_insn0total_states5peak_states5mark_read3
=============The error is due to the following in verifier:
} else if (opcode == BPF_EXIT) { ... if (env->cur_state->active_rcu_lock && !in_rbtree_lock_required_cb(env)) { verbose(env, "bpf_rcu_read_unlock is missing\n"); return -EINVAL; }I guess, we could relax the condition not to return
-EINVALif it is a static function.Quoted 5 lines
Another workaround is using the
__always_inline:
static__always_inlinevoidread_percpu_data(struct bpf_iter_meta *meta, struct cgroup *cgrp, const cpumask_t *mask)
Quoted 8 lines
Quoted 4 lines
__always_inlineis also work. But let us improve verifier so we can avoid such workarounds in the future. Note that Kumar just submitted a patch set to relaxspin_lockfor static functions:https://lore.kernel.org/bpf/[email protected]/Agreed, let's improve verifier, but I wouldn't block on it for this patch set and just use
__always_inlinefor now.
OK. will do it.
Quoted 11 lines
I think we should also work on extending this RCU support to global functions. We can add per-function annotation (similar to
__arg_xxx, but which will be applied to a function itself), just__rcuor something like__func_assume_rcuor whatnot, and then there should be no difference between static and global functions in this regard.In general, global functions are basically mandatory nowadays for big production BPF programs (to reduce verification complexity), so we should strive to keep global functions/subprogs on par with static subprogs as much as we can.
Understood. Thanks for your explanation.
Signature · 2 lines
Regards Yafang
kernel test robot
build botRe: [PATCH v5 bpf-next 4/4] selftests/bpf: Add selftests for cpumask iter
In reply to Yafang Shaokernel test robot noticed the following build errors:
Hi Yafang,
kernel test robot noticed the following build errors:
[auto build test ERROR on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Yafang-Shao/bpf-Add-bpf_iter_cpumask-kfuncs/20240131-232406 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20240131145454.86990-5-laoar.shao%40gmail.com patch subject: [PATCH v5 bpf-next 4/4] selftests/bpf: Add selftests for cpumask iter compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240205/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> progs/cpumask_iter_success.c:61:2: error: incomplete definition of type 'struct psi_group_cpu'
61 | READ_PERCPU_DATA(meta, cgrp, p->cpus_ptr);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
progs/cpumask_iter_success.c:39:27: note: expanded from macro 'READ_PERCPU_DATA'
39 | psi_nr_running += groupc->tasks[NR_RUNNING]; \
| ~~~~~~^
progs/cpumask_iter_success.c:13:21: note: forward declaration of 'struct psi_group_cpu'
13 | extern const struct psi_group_cpu system_group_pcpu __ksym __weak;
| ^
>> progs/cpumask_iter_success.c:61:2: error: use of undeclared identifier 'NR_RUNNING'; did you mean 'T_RUNNING'?
61 | READ_PERCPU_DATA(meta, cgrp, p->cpus_ptr);
| ^
progs/cpumask_iter_success.c:39:35: note: expanded from macro 'READ_PERCPU_DATA'
39 | psi_nr_running += groupc->tasks[NR_RUNNING]; \
| ^
/tools/include/vmlinux.h:28263:3: note: 'T_RUNNING' declared here
28263 | T_RUNNING = 0,
| ^
progs/cpumask_iter_success.c:80:2: error: incomplete definition of type 'struct psi_group_cpu'
80 | READ_PERCPU_DATA(meta, cgrp, p->cpus_ptr);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
progs/cpumask_iter_success.c:39:27: note: expanded from macro 'READ_PERCPU_DATA'
39 | psi_nr_running += groupc->tasks[NR_RUNNING]; \
| ~~~~~~^
progs/cpumask_iter_success.c:13:21: note: forward declaration of 'struct psi_group_cpu'
13 | extern const struct psi_group_cpu system_group_pcpu __ksym __weak;
| ^
progs/cpumask_iter_success.c:80:2: error: use of undeclared identifier 'NR_RUNNING'; did you mean 'T_RUNNING'?
80 | READ_PERCPU_DATA(meta, cgrp, p->cpus_ptr);
| ^
progs/cpumask_iter_success.c:39:35: note: expanded from macro 'READ_PERCPU_DATA'
39 | psi_nr_running += groupc->tasks[NR_RUNNING]; \
| ^
/tools/include/vmlinux.h:28263:3: note: 'T_RUNNING' declared here
28263 | T_RUNNING = 0,
| ^4 errors generated.
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v5 bpf-next 4/4] selftests/bpf: Add selftests for cpumask iter
Quoted 14 lines · kernel test robot
Hi Yafang,
kernel test robot noticed the following build errors:
[auto build test ERROR on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Yafang-Shao/bpf-Add-bpf_iter_cpumask-kfuncs/20240131-232406 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20240131145454.86990-5-laoar.shao%40gmail.com patch subject: [PATCH v5 bpf-next 4/4] selftests/bpf: Add selftests for cpumask iter compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240205/[email protected]/reproduce)
Quoted 4 lines
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
Thanks for your report.
It seems that the issue is caused by missing CONFIG_PSI=y. The kernel config should align with tools/testing/selftests/bpf/config.
Quoted 44 lines
All errors (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
progs/cpumask_iter_success.c:61:2: error: incomplete definition of type 'structpsi_group_cpu'61 | READ_PERCPU_DATA(meta, cgrp, p->cpus_ptr); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ progs/cpumask_iter_success.c:39:27: note: expanded from macro 'READ_PERCPU_DATA' 39 | psi_nr_running += groupc->tasks[NR_RUNNING]; \ | ~~~~~~^ progs/cpumask_iter_success.c:13:21: note: forward declaration of 'struct psi_group_cpu' 13 | extern const struct psi_group_cpu system_group_pcpu __ksym __weak; | ^Quoted 1 line
Quoted 1 line
progs/cpumask_iter_success.c:61:2: error: use of undeclared identifier 'NR_RUNNING'; did you mean 'T_RUNNING'?61 | READ_PERCPU_DATA(meta, cgrp, p->cpus_ptr); | ^ progs/cpumask_iter_success.c:39:35: note: expanded from macro 'READ_PERCPU_DATA' 39 | psi_nr_running += groupc->tasks[NR_RUNNING]; \ | ^ /tools/include/vmlinux.h:28263:3: note: 'T_RUNNING' declared here 28263 | T_RUNNING = 0, | ^ progs/cpumask_iter_success.c:80:2: error: incomplete definition of type 'struct psi_group_cpu' 80 | READ_PERCPU_DATA(meta, cgrp, p->cpus_ptr); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ progs/cpumask_iter_success.c:39:27: note: expanded from macro 'READ_PERCPU_DATA' 39 | psi_nr_running += groupc->tasks[NR_RUNNING]; \ | ~~~~~~^ progs/cpumask_iter_success.c:13:21: note: forward declaration of 'struct psi_group_cpu' 13 | extern const struct psi_group_cpu system_group_pcpu __ksym __weak; | ^ progs/cpumask_iter_success.c:80:2: error: use of undeclared identifier 'NR_RUNNING'; did you mean 'T_RUNNING'? 80 | READ_PERCPU_DATA(meta, cgrp, p->cpus_ptr); | ^ progs/cpumask_iter_success.c:39:35: note: expanded from macro 'READ_PERCPU_DATA' 39 | psi_nr_running += groupc->tasks[NR_RUNNING]; \ | ^ /tools/include/vmlinux.h:28263:3: note: 'T_RUNNING' declared here 28263 | T_RUNNING = 0, | ^4 errors generated.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
Signature · 2 lines
Regards Yafang