bpf: Add bpf_iter_cpumask
Subsystems (4)
v3 cover bpf: Add bpf_iter_cpumask
Three new kfuncs, namely bpf_iter_cpumask_{new,next,destroy}, have been added for the new bpf_iter_cpumask functionality. These kfuncs enable the iteration of percpu data, such as runqueues, system_group_pcpu, and more.
In our specific use case, we leverage the cgroup iterator to traverse percpu data, subsequently exposing it to userspace through a seq file. Refer to the test cases in patch #2 for further context and examples.
Changes:
- v2 -> v3:
- Define KF_RCU_PROTECTED for bpf_iter_cpumask_new (Alexei)
- Code improvement in selftests
- Fix build error in selftest due to CONFIG_PSI=n reported by kernel test robot <[email protected]>
- v1 -> v2:
- Avoid changing cgroup subsystem (Tejun)
- Remove bpf_cpumask_set_from_pid(), and use bpf_cpumask_copy() instead (Tejun)
- Use int cpu; field in bpf_iter_cpumask_kern (Andrii)
- bpf: Add new bpf helper bpf_for_each_cpu
https://lwn.net/ml/bpf/[email protected]/
Yafang Shao (3):
bpf: Add bpf_iter_cpumask kfuncs
bpf, doc: Add document for cpumask iter
selftests/bpf: Add selftests for cpumask iter
Documentation/bpf/cpumasks.rst | 17 +++ kernel/bpf/cpumask.c | 69 +++++++++ tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cpumask_iter.c | 134 ++++++++++++++++++ .../selftests/bpf/progs/cpumask_common.h | 3 + .../selftests/bpf/progs/test_cpumask_iter.c | 56 ++++++++ 6 files changed, 280 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/cpumask_iter.c create mode 100644 tools/testing/selftests/bpf/progs/test_cpumask_iter.c
Signature · 1 line
2.39.1
PATCH v3 1/3 bpf: Add bpf_iter_cpumask kfuncs
Add three new kfuncs for bpf_iter_cpumask.
- bpf_iter_cpumask_new
It is defined with KF_RCU_PROTECTED and KF_RCU. KF_RCU_PROTECTED is defined because we must use it under the protection of RCU.
KF_RCU is defined because the cpumask must be a RCU trusted pointer such as task->cpus_ptr.
- bpf_iter_cpumask_next
- bpf_iter_cpumask_destroy
These new kfuncs facilitate the iteration of percpu data, such as runqueues, psi_cgroup_cpu, and more.
--- kernel/bpf/cpumask.c | 69 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+)
kernel/bpf/cpumask.c +69 -0
@@ -422,6 +422,72 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)return cpumask_weight (cpumask); } +struct bpf_iter_cpumask { +__u64 __opaque[2 ]; +}__aligned(8) ; + +struct bpf_iter_cpumask_kern { +const struct cpumask *mask; +int cpu; +}__aligned(8) ; + +/** +* bpf_iter_cpumask_new() - Create a new bpf_iter_cpumask for a specified cpumask +* @it: The new bpf_iter_cpumask to be created. +* @mask: The cpumask to be iterated over. +* +* This function initializes a new bpf_iter_cpumask structure for iterating over +* the specified CPU mask. It assigns the provided cpumask to the newly created +* bpf_iter_cpumask @it for subsequent iteration operations. +* +* On success, 0 is returen. On failure, ERR is returned. +*/ +__bpf_kfunc int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; + +BUILD_BUG_ON (sizeof (struct bpf_iter_cpumask_kern ) >sizeof (struct bpf_iter_cpumask )); +BUILD_BUG_ON (__alignof__(struct bpf_iter_cpumask_kern ) != + __alignof__(struct bpf_iter_cpumask )); + + kit->mask = mask; + kit->cpu =-1 ; +return 0 ; +} + +/** +* bpf_iter_cpumask_next() - Get the next CPU in a bpf_iter_cpumask +* @it: The bpf_iter_cpumask +* +* This function retrieves a pointer to the number of the next CPU within the +* specified bpf_iter_cpumask. It allows sequential access to CPUs within the +* cpumask. If there are no further CPUs available, it returns NULL. +* +* Returns a pointer to the number of the next CPU in the cpumask or NULL if no +* further CPUs. +*/ +__bpf_kfunc int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; +const struct cpumask *mask = kit->mask; +int cpu; + + cpu =cpumask_next (kit->cpu, mask); +if (cpu >= nr_cpu_ids) +return NULL ; + + kit->cpu = cpu; +return &kit->cpu; +} + +/** +* bpf_iter_cpumask_destroy() - Destroy a bpf_iter_cpumask +* @it: The bpf_iter_cpumask to be destroyed. +*/ +__bpf_kfunc void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) +{ +} +__bpf_kfunc_end_defs ();BTF_SET8_START (cpumask_kfunc_btf_ids)
@@ -450,6 +516,9 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_new, KF_ITER_NEW | KF_RCU_PROTECTED | KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_cpumask_destroy, KF_ITER_DESTROY)BTF_SET8_END (cpumask_kfunc_btf_ids) static const struct btf_kfunc_id_set cpumask_kfunc_set = {
Signature · 1 line
2.39.1
Yonghong Song
reviewerRe: [PATCH v3 bpf-next 1/3] bpf: Add bpf_iter_cpumask kfuncs
Quoted 7 lines · Yafang Shao
Add three new kfuncs for
bpf_iter_cpumask.
-bpf_iter_cpumask_new
It is defined withKF_RCU_PROTECTEDandKF_RCU.KF_RCU_PROTECTEDis defined because we must use it under the protection of RCU.
KF_RCUis defined because the cpumask must be a RCU trusted pointer such astask->cpus_ptr.
I am not sure whether we need both or not.
KF_RCU_PROTECTED means the function call needs within the rcu cs. KF_RCU means the argument usage needs within the rcu cs. We only need one of them (preferrably KF_RCU).
Quoted 41 lines
-
bpf_iter_cpumask_next
-bpf_iter_cpumask_destroyThese new kfuncs facilitate the iteration of percpu data, such as runqueues,
psi_cgroup_cpu, and more.Signed-off-by Yafang Shao <[email protected]>--- kernel/bpf/cpumask.c | 69 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+)kernel/bpf/cpumask.c +22
-0@@ -422,6 +422,72 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)return cpumask_weight (cpumask); } +struct bpf_iter_cpumask { +__u64 __opaque[2 ]; +}__aligned(8) ; + +struct bpf_iter_cpumask_kern { +const struct cpumask *mask; +int cpu; +}__aligned(8) ; + +/** +* bpf_iter_cpumask_new() - Create a new bpf_iter_cpumask for a specified cpumask +* @it: The new bpf_iter_cpumask to be created. +* @mask: The cpumask to be iterated over. +* +* This function initializes a new bpf_iter_cpumask structure for iterating over +* the specified CPU mask. It assigns the provided cpumask to the newly created +* bpf_iter_cpumask @it for subsequent iteration operations. +* +* On success, 0 is returen. On failure, ERR is returned. +*/ +__bpf_kfunc int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) +{
kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 456
Quoted 10 lines
kernel/bpf/cpumask.c +10
-0+struct bpf_iter_cpumask_kern *kit = (void *)it; + +BUILD_BUG_ON (sizeof (struct bpf_iter_cpumask_kern ) >sizeof (struct bpf_iter_cpumask )); +BUILD_BUG_ON (__alignof__(struct bpf_iter_cpumask_kern ) != + __alignof__(struct bpf_iter_cpumask )); + + kit->mask = mask; + kit->cpu =-1 ; +return 0 ; +}
We have problem here. Let us say bpf_iter_cpumask_new() is called inside rcu cs. Once the control goes out of rcu cs, 'mask' could be freed, right? Or you require bpf_iter_cpumask_next() needs to be in the same rcu cs as bpf_iter_cpumask_new(). But such a requirement seems odd.
I think we can do things similar to bpf_iter_task_vma. You can allocate memory with bpf_mem_alloc() in bpf_iter_cpumask_new() to keep a copy of mask. This way, you do not need to worry about potential use-after-free issue. The memory can be freed with bpf_iter_cpumask_destroy().
Quoted 47 lines
kernel/bpf/cpumask.c +37
-0+ +/** +* bpf_iter_cpumask_next() - Get the next CPU in a bpf_iter_cpumask +* @it: The bpf_iter_cpumask +* +* This function retrieves a pointer to the number of the next CPU within the +* specified bpf_iter_cpumask. It allows sequential access to CPUs within the +* cpumask. If there are no further CPUs available, it returns NULL. +* +* Returns a pointer to the number of the next CPU in the cpumask or NULL if no +* further CPUs. +*/ +__bpf_kfunc int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; +const struct cpumask *mask = kit->mask; +int cpu; + + cpu =cpumask_next (kit->cpu, mask); +if (cpu >= nr_cpu_ids) +return NULL ; + + kit->cpu = cpu; +return &kit->cpu; +} + +/** +* bpf_iter_cpumask_destroy() - Destroy a bpf_iter_cpumask +* @it: The bpf_iter_cpumask to be destroyed. +*/ +__bpf_kfunc void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) +{ +} +__bpf_kfunc_end_defs ();BTF_SET8_START (cpumask_kfunc_btf_ids)@@ -450,6 +516,9 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_new, KF_ITER_NEW | KF_RCU_PROTECTED | KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_cpumask_destroy, KF_ITER_DESTROY)BTF_SET8_END (cpumask_kfunc_btf_ids) static const struct btf_kfunc_id_set cpumask_kfunc_set = {
Re: [PATCH v3 bpf-next 1/3] bpf: Add bpf_iter_cpumask kfuncs
Hi,
Quoted 70 lines · Yonghong Song
Quoted 7 lines · Yafang Shao
Add three new kfuncs for
bpf_iter_cpumask.
-bpf_iter_cpumask_new
It is defined withKF_RCU_PROTECTEDandKF_RCU.KF_RCU_PROTECTEDis defined because we must use it under the protection of RCU.
KF_RCUis defined because the cpumask must be a RCU trusted pointer such astask->cpus_ptr.I am not sure whether we need both or not.
KF_RCU_PROTECTEDmeans the function call needs within the rcu cs.KF_RCUmeans the argument usage needs within the rcu cs. We only need one of them (preferrablyKF_RCU).Quoted 54 lines
-
bpf_iter_cpumask_next
-bpf_iter_cpumask_destroyThese new kfuncs facilitate the iteration of percpu data, such as runqueues,
psi_cgroup_cpu, and more.Signed-off-by Yafang Shao <[email protected]>---
kernel/bpf/cpumask.c| 69 ++++++++++++++++++++++++++++++++++++++++++++1 file changed, 69 insertions(+)kernel/bpf/cpumask.c +10
-0@@ -422,6 +422,72 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) return cpumask_weight(cpumask); } +struct bpf_iter_cpumask { + __u64 __opaque[2]; +}__aligned(8) ; + +struct bpf_iter_cpumask_kern { + const struct cpumask *mask; + int cpu; +}__aligned(8) ; + +/** +* bpf_iter_cpumask_new() - Create a new bpf_iter_cpumask for a specified cpumask
+ * @it: The newbpf_iter_cpumaskto be created.
+ * @mask: The cpumask to be iterated over.
+ *
+ * This function initializes a newbpf_iter_cpumaskstructure for iterating over
+ * the specified CPU mask. It assigns the provided cpumask to the newly created
+ *bpf_iter_cpumask@it for subsequent iteration operations.
+ *
+ * On success, 0 is returen. On failure, ERR is returned.
+ */
+__bpf_kfunc intbpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask)kernel/bpf/cpumask.c +4
-0+{ + struct bpf_iter_cpumask_kern *kit = (void *)it; + + BUILD_BUG_ON(sizeof(struct bpf_iter_cpumask_kern) >
sizeof(struct bpf_iter_cpumask));
+ BUILD_BUG_ON(__alignof__(structbpf_iter_cpumask_kern) != +__alignof__(struct bpf_iter_cpumask)); +
+kit->mask= mask;
+kit->cpu= -1;
Quoted 8 lines
Quoted 2 lines
+ return 0;
+}We have problem here. Let us say
bpf_iter_cpumask_new()is called inside rcu cs.
Once the control goes out of rcu cs, 'mask' could be freed, right? Or you requirebpf_iter_cpumask_next()needs to be in the same rcu cs asbpf_iter_cpumask_new(). But such a requirement seems odd.
So the case is possible when using bpf_iter_cpumask_new() and bpf_iter_cpumask_next() in sleepable program and these two kfuncs are used in two different rcu_read_lock/rcu_read_unlock code blocks, right ?
Quoted 59 lines
I think we can do things similar to
bpf_iter_task_vma. You can allocate memory
withbpf_mem_alloc()inbpf_iter_cpumask_new()to keep a copy of mask. This
way, you do not need to worry about potential use-after-free issue. The memory can be freed withbpf_iter_cpumask_destroy().Quoted 40 lines
kernel/bpf/cpumask.c +6
-0+ +/** +* bpf_iter_cpumask_next() - Get the next CPU in a bpf_iter_cpumask +* @it: The bpf_iter_cpumask +* +* This function retrieves a pointer to the number of the next CPU within the
+ * specifiedbpf_iter_cpumask. It allows sequential access to CPUs within the
+ * cpumask. If there are no further CPUs available, it returns NULL.
+ *
+ * Returns a pointer to the number of the next CPU in the cpumask or NULL if no
+ * further CPUs.
+ */kernel/bpf/cpumask.c +22
-0+__bpf_kfunc int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it) +{ + struct bpf_iter_cpumask_kern *kit = (void *)it; + const struct cpumask *mask = kit->mask; + int cpu; + + cpu = cpumask_next(kit->cpu, mask); + if (cpu >= nr_cpu_ids) + return NULL; + + kit->cpu = cpu; + return &kit->cpu; +} + +/** +* bpf_iter_cpumask_destroy() - Destroy a bpf_iter_cpumask +* @it: The bpf_iter_cpumask to be destroyed. +*/ +__bpf_kfunc void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) +{ +} +
__bpf_kfunc_end_defs();
BTF_SET8_START(cpumask_kfunc_btf_ids)kernel/bpf/cpumask.c BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) · line 524
Quoted 9 lines
kernel/bpf/cpumask.c +3
-0@@ -450,6 +516,9 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_new, KF_ITER_NEW | KF_RCU_PROTECTED | KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_cpumask_destroy, KF_ITER_DESTROY) BTF_SET8_END(cpumask_kfunc_btf_ids) static const struct btf_kfunc_id_set cpumask_kfunc_set = {.
Yonghong Song
reviewerRe: [PATCH v3 bpf-next 1/3] bpf: Add bpf_iter_cpumask kfuncs
Quoted 78 lines · Hou Tao
Hi,
Quoted 67 lines · Yonghong Song
Quoted 7 lines · Yafang Shao
Add three new kfuncs for
bpf_iter_cpumask.
-bpf_iter_cpumask_new
It is defined withKF_RCU_PROTECTEDandKF_RCU.KF_RCU_PROTECTEDis defined because we must use it under the protection of RCU.
KF_RCUis defined because the cpumask must be a RCU trusted pointer such astask->cpus_ptr.I am not sure whether we need both or not.
KF_RCU_PROTECTEDmeans the function call needs within the rcu cs.KF_RCUmeans the argument usage needs within the rcu cs. We only need one of them (preferrablyKF_RCU).Quoted 53 lines
-
bpf_iter_cpumask_next
-bpf_iter_cpumask_destroyThese new kfuncs facilitate the iteration of percpu data, such as runqueues,
psi_cgroup_cpu, and more.Signed-off-by Yafang Shao <[email protected]>--- kernel/bpf/cpumask.c | 69 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+)kernel/bpf/cpumask.c +10
-0@@ -422,6 +422,72 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) return cpumask_weight(cpumask); } +struct bpf_iter_cpumask { + __u64 __opaque[2]; +}__aligned(8) ; + +struct bpf_iter_cpumask_kern { + const struct cpumask *mask; + int cpu; +}__aligned(8) ; + +/** +* bpf_iter_cpumask_new() - Create a new bpf_iter_cpumask for a specified cpumask
+ * @it: The newbpf_iter_cpumaskto be created.
+ * @mask: The cpumask to be iterated over.
+ *
+ * This function initializes a newbpf_iter_cpumaskstructure for iterating over
+ * the specified CPU mask. It assigns the provided cpumask to the newly created
+ *bpf_iter_cpumask@it for subsequent iteration operations.
+ *
+ * On success, 0 is returen. On failure, ERR is returned.
+ */
+__bpf_kfunc intbpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask)kernel/bpf/cpumask.c +4
-0+{ + struct bpf_iter_cpumask_kern *kit = (void *)it; + + BUILD_BUG_ON(sizeof(struct bpf_iter_cpumask_kern) >
sizeof(struct bpf_iter_cpumask));
+ BUILD_BUG_ON(__alignof__(structbpf_iter_cpumask_kern) != +__alignof__(struct bpf_iter_cpumask)); +
+kit->mask= mask;Quoted 8 lines
Quoted 3 lines
+
kit->cpu= -1;
+ return 0;
+}We have problem here. Let us say
bpf_iter_cpumask_new()is called inside rcu cs.
Once the control goes out of rcu cs, 'mask' could be freed, right? Or you requirebpf_iter_cpumask_next()needs to be in the same rcu cs asbpf_iter_cpumask_new(). But such a requirement seems odd.
Quoted 3 lines
So the case is possible when using
bpf_iter_cpumask_new()andbpf_iter_cpumask_next()in sleepable program and these two kfuncs are used in two different rcu_read_lock/rcu_read_unlock code blocks, right ?
Right, or bpf_iter_cpumask_new() inside rcu cs and bpf_iter_cpumask_next() not.
Quoted 57 lines
Quoted 57 lines
I think we can do things similar to
bpf_iter_task_vma. You can allocate memory
withbpf_mem_alloc()inbpf_iter_cpumask_new()to keep a copy of mask. This
way, you do not need to worry about potential use-after-free issue. The memory can be freed withbpf_iter_cpumask_destroy().Quoted 40 lines
kernel/bpf/cpumask.c +6
-0+ +/** +* bpf_iter_cpumask_next() - Get the next CPU in a bpf_iter_cpumask +* @it: The bpf_iter_cpumask +* +* This function retrieves a pointer to the number of the next CPU within the
+ * specifiedbpf_iter_cpumask. It allows sequential access to CPUs within the
+ * cpumask. If there are no further CPUs available, it returns NULL.
+ *
+ * Returns a pointer to the number of the next CPU in the cpumask or NULL if no
+ * further CPUs.
+ */kernel/bpf/cpumask.c +22
-0+__bpf_kfunc int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it) +{ + struct bpf_iter_cpumask_kern *kit = (void *)it; + const struct cpumask *mask = kit->mask; + int cpu; + + cpu = cpumask_next(kit->cpu, mask); + if (cpu >= nr_cpu_ids) + return NULL; + + kit->cpu = cpu; + return &kit->cpu; +} + +/** +* bpf_iter_cpumask_destroy() - Destroy a bpf_iter_cpumask +* @it: The bpf_iter_cpumask to be destroyed. +*/ +__bpf_kfunc void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) +{ +} + __bpf_kfunc_end_defs(); BTF_SET8_START(cpumask_kfunc_btf_ids)kernel/bpf/cpumask.c BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) · line 524
Quoted 9 lines
kernel/bpf/cpumask.c +3
-0@@ -450,6 +516,9 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_new, KF_ITER_NEW | KF_RCU_PROTECTED | KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_cpumask_destroy, KF_ITER_DESTROY) BTF_SET8_END(cpumask_kfunc_btf_ids) static const struct btf_kfunc_id_set cpumask_kfunc_set = {.
Re: [PATCH v3 bpf-next 1/3] bpf: Add bpf_iter_cpumask kfuncs
In reply to Yonghong SongQuoted 11 lines · Yonghong Song
Quoted 7 lines · Yafang Shao
Add three new kfuncs for
bpf_iter_cpumask.
-bpf_iter_cpumask_new
It is defined withKF_RCU_PROTECTEDandKF_RCU.KF_RCU_PROTECTEDis defined because we must use it under the protection of RCU.
KF_RCUis defined because the cpumask must be a RCU trusted pointer such astask->cpus_ptr.
Quoted 5 lines
I am not sure whether we need both or not.
KF_RCU_PROTECTEDmeans the function call needs within the rcu cs.KF_RCUmeans the argument usage needs within the rcu cs. We only need one of them (preferrablyKF_RCU).
As you explained below, KF_RCU_PROTECTED is actually for bpf_iter_cpumask_next().
Quoted 58 lines
Quoted 42 lines
-
bpf_iter_cpumask_next
-bpf_iter_cpumask_destroyThese new kfuncs facilitate the iteration of percpu data, such as runqueues,
psi_cgroup_cpu, and more.Signed-off-by Yafang Shao <[email protected]>--- kernel/bpf/cpumask.c | 69 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+)kernel/bpf/cpumask.c +22
-0@@ -422,6 +422,72 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)return cpumask_weight (cpumask); } +struct bpf_iter_cpumask { +__u64 __opaque[2 ]; +}__aligned(8) ; + +struct bpf_iter_cpumask_kern { +const struct cpumask *mask; +int cpu; +}__aligned(8) ; + +/** +* bpf_iter_cpumask_new() - Create a new bpf_iter_cpumask for a specified cpumask +* @it: The new bpf_iter_cpumask to be created. +* @mask: The cpumask to be iterated over. +* +* This function initializes a new bpf_iter_cpumask structure for iterating over +* the specified CPU mask. It assigns the provided cpumask to the newly created +* bpf_iter_cpumask @it for subsequent iteration operations. +* +* On success, 0 is returen. On failure, ERR is returned. +*/ +__bpf_kfunc int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) +{kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 456
Quoted 10 lines
kernel/bpf/cpumask.c +10
-0+struct bpf_iter_cpumask_kern *kit = (void *)it; + +BUILD_BUG_ON (sizeof (struct bpf_iter_cpumask_kern ) >sizeof (struct bpf_iter_cpumask )); +BUILD_BUG_ON (__alignof__(struct bpf_iter_cpumask_kern ) != + __alignof__(struct bpf_iter_cpumask )); + + kit->mask = mask; + kit->cpu =-1 ; +return 0 ; +}We have problem here. Let us say
bpf_iter_cpumask_new()is called inside rcu cs. Once the control goes out of rcu cs, 'mask' could be freed, right? Or you requirebpf_iter_cpumask_next()needs to be in the same rcu cs asbpf_iter_cpumask_new(). But such a requirement seems odd.
Quoted 4 lines
I think we can do things similar to
bpf_iter_task_vma. You can allocate memory withbpf_mem_alloc()inbpf_iter_cpumask_new()to keep a copy of mask. This way, you do not need to worry about potential use-after-free issue. The memory can be freed withbpf_iter_cpumask_destroy().
Good suggestion. That seems better.
Quoted 47 lines
Quoted 47 lines
kernel/bpf/cpumask.c +37
-0+ +/** +* bpf_iter_cpumask_next() - Get the next CPU in a bpf_iter_cpumask +* @it: The bpf_iter_cpumask +* +* This function retrieves a pointer to the number of the next CPU within the +* specified bpf_iter_cpumask. It allows sequential access to CPUs within the +* cpumask. If there are no further CPUs available, it returns NULL. +* +* Returns a pointer to the number of the next CPU in the cpumask or NULL if no +* further CPUs. +*/ +__bpf_kfunc int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it) +{ +struct bpf_iter_cpumask_kern *kit = (void *)it; +const struct cpumask *mask = kit->mask; +int cpu; + + cpu =cpumask_next (kit->cpu, mask); +if (cpu >= nr_cpu_ids) +return NULL ; + + kit->cpu = cpu; +return &kit->cpu; +} + +/** +* bpf_iter_cpumask_destroy() - Destroy a bpf_iter_cpumask +* @it: The bpf_iter_cpumask to be destroyed. +*/ +__bpf_kfunc void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) +{ +} +__bpf_kfunc_end_defs ();BTF_SET8_START (cpumask_kfunc_btf_ids)@@ -450,6 +516,9 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU) BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_new, KF_ITER_NEW | KF_RCU_PROTECTED | KF_RCU) +BTF_ID_FLAGS(func, bpf_iter_cpumask_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_cpumask_destroy, KF_ITER_DESTROY)BTF_SET8_END (cpumask_kfunc_btf_ids)
Quoted 1 line
kernel/bpf/cpumask.c BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU) · line 524
Quoted 1 line
kernel/bpf/cpumask.c +0
-0static const struct btf_kfunc_id_set cpumask_kfunc_set = {
--
Regards
Yafang
PATCH v3 2/3 bpf, doc: Add document for cpumask iter
In reply to Yafang ShaoThis patch adds the document for the newly added cpumask iterator kfuncs.
--- Documentation/bpf/cpumasks.rst | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
Documentation/bpf/cpumasks.rst +17 -0
@@ -372,6 +372,23 @@ used. .. _tools/testing/selftests/bpf/progs/cpumask_success.c: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/progs/cpumask_success.c +3.3 cpumask iterator +-------------------- + +The cpumask iterator enables the iteration of percpu data, such as runqueues, +system_group_pcpu, and more. + +.. kernel-doc:: kernel/bpf/cpumask.c + :identifiers: bpf_iter_cpumask_new bpf_iter_cpumask_next + bpf_iter_cpumask_destroy + +---- + +Some example usages of the cpumask iterator can be found in +`tools/testing/selftests/bpf/progs/test_cpumask_iter.c`_. + +.. _tools/testing/selftests/bpf/progs/test_cpumask_iter.c: + https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/progs/test_cpumask_iter.c 4. Adding BPF cpumask kfuncs ============================
Signature · 1 line
2.39.1
PATCH v3 3/3 selftests/bpf: Add selftests for cpumask iter
In reply to Yafang ShaoWithin the BPF program, we leverage the cgroup iterator to iterate through percpu runqueue data, specifically the 'nr_running' metric. Subsequently
we expose this data to userspace by means of a sequence file.
The CPU affinity for the cpumask is determined by the PID of a task:
- PID of the init task (PID 1)
We typically don't set CPU affinity for init task and thus we can iterate across all possible CPUs. However, in scenarios where you've set CPU affinity for the init task, you should set the cpumask of your current task to full-F. Then proceed to iterate through all possible CPUs using the current task.
- PID of a task with defined CPU affinity
The aim here is to iterate through a specific cpumask. This scenario aligns with tasks residing within a cpuset cgroup.
- Invalid PID (e.g., PID -1)
No cpumask is available in this case.
The result as follows,
#65/1 cpumask_iter/init_pid:OK #65/2 cpumask_iter/invalid_pid:OK #65/3 cpumask_iter/self_pid_one_cpu:OK #65/4 cpumask_iter/self_pid_multi_cpus:OK #65 cpumask_iter:OK
Summary: 1/4 PASSED, 0 SKIPPED, 0 FAILED
CONFIG_PSI=y is required for this testcase.
--- tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cpumask_iter.c | 134 ++++++++++++++++++ .../selftests/bpf/progs/cpumask_common.h | 3 + .../selftests/bpf/progs/test_cpumask_iter.c | 56 ++++++++ 4 files changed, 194 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/cpumask_iter.c create mode 100644 tools/testing/selftests/bpf/progs/test_cpumask_iter.c
tools/testing/selftests/bpf/config +1 -0
@@ -78,6 +78,7 @@ CONFIG_NF_CONNTRACK_MARK=y CONFIG_NF_DEFRAG_IPV4=y CONFIG_NF_DEFRAG_IPV6=y CONFIG_NF_NAT=y +CONFIG_PSI=y CONFIG_RC_CORE=y CONFIG_SECURITY=y CONFIG_SECURITYFS=y
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +134 -0
134 lines not on this page · raw message
tools/testing/selftests/bpf/progs/cpumask_common.h +3 -0
@@ -55,6 +55,9 @@ void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym u32 bpf_cpumask_any_distribute(const struct cpumask *src) __ksym; u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1, const struct cpumask *src2) __ksym; u32 bpf_cpumask_weight(const struct cpumask *cpumask) __ksym; +int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) __ksym; +int *bpf_iter_cpumask_next (struct bpf_iter_cpumask *it) __ksym; +void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) __ksym; void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym;
tools/testing/selftests/bpf/progs/test_cpumask_iter.c new file +56 -0
56 lines not on this page · raw message
Signature · 1 line
2.39.1
Yonghong Song
reviewerRe: [PATCH v3 bpf-next 3/3] selftests/bpf: Add selftests for cpumask iter
Quoted 11 lines · Yafang Shao
Within the BPF program, we leverage the cgroup iterator to iterate through percpu runqueue data, specifically the '
nr_running' metric. Subsequently
we expose this data to userspace by means of a sequence file.The CPU affinity for the cpumask is determined by the PID of a task:
- PID of the init task (PID 1)
We typically don't set CPU affinity for init task and thus we can iterate across all possible CPUs. However, in scenarios where you've set CPU affinity for the init task, you should set the cpumask of your current task to full-F. Then proceed to iterate through all possible CPUs using
Wat is full-F? It would be good if you can clarify in the commit message.
Quoted 51 lines
the current task.
- PID of a task with defined CPU affinity
The aim here is to iterate through a specific cpumask. This scenario aligns with tasks residing within a cpuset cgroup.
- Invalid PID (e.g., PID -1)
No cpumask is available in this case.The result as follows,
#65/1 cpumask_iter/init_pid:OK #65/2 cpumask_iter/invalid_pid:OK #65/3 cpumask_iter/self_pid_one_cpu:OK #65/4 cpumask_iter/self_pid_multi_cpus:OK #65 cpumask_iter:OKSummary: 1/4 PASSED, 0 SKIPPED, 0 FAILED
CONFIG_PSI=yis required for this testcase.Signed-off-by Yafang Shao <[email protected]>--- tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cpumask_iter.c | 134 ++++++++++++++++++ .../selftests/bpf/progs/cpumask_common.h | 3 + .../selftests/bpf/progs/test_cpumask_iter.c | 56 ++++++++ 4 files changed, 194 insertions(+)create mode 100644
tools/testing/selftests/bpf/prog_tests/cpumask_iter.ccreate mode 100644tools/testing/selftests/bpf/progs/test_cpumask_iter.ctools/testing/selftests/bpf/config +1
-0@@ -78,6 +78,7 @@ CONFIG_NF_CONNTRACK_MARK=y CONFIG_NF_DEFRAG_IPV4=y CONFIG_NF_DEFRAG_IPV6=y CONFIG_NF_NAT=y +CONFIG_PSI=y CONFIG_RC_CORE=y CONFIG_SECURITY=y CONFIG_SECURITYFS=ytools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +6
-0@@ -0,0 +1,134 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Yafang Shao <[email protected]> */ + +#define _GNU_SOURCE +#include <sched.h> +#include <stdio.h>
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 16
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10
-0+#include <unistd.h> + +#include <test_progs.h> +#include "cgroup_helpers.h" +#include "test_cpumask_iter.skel.h" + +static void verify_percpu_data (struct bpf_link *link,int nr_cpu_exp,int nr_running_exp) +{ +int iter_fd, len, item, nr_running, psi_running, nr_cpus; +static char buf[128 ];
why static?
Quoted 20 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +20
-0+size_t left; +char *p; + + iter_fd =bpf_iter_create (bpf_link__fd (link)); +if (!ASSERT_GE (iter_fd,0 ,"iter_fd" )) +return ; + +memset (buf,0 ,sizeof (buf)); + left =ARRAY_SIZE (buf); + p = buf; +while ((len =read (iter_fd, p, left)) >0 ) { + p += len; + left -= len; + } + + item =sscanf (buf,"nr_running %u nr_cpus %u psi_running %u\n" , + &nr_running, &nr_cpus, &psi_running); +if (nr_cpu_exp ==-1 ) { +ASSERT_EQ (item,-1 ,"seq_format" ); + goto out;
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 46
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10
-0+ } + +ASSERT_EQ (item,3 ,"seq_format" ); +ASSERT_GE (nr_running, nr_running_exp,"nr_running" ); +ASSERT_GE (psi_running, nr_running_exp,"psi_running" ); +ASSERT_EQ (nr_cpus, nr_cpu_exp,"nr_cpus" ); + +/* read() after iter finishes should be ok. */ +if (len ==0 ) +ASSERT_OK (read (iter_fd, buf,sizeof (buf)),"second_read" );
The above 'if' statement is irrelevant to the main purpose of this test and can be removed.
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 56
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10
-0+ +out: +close (iter_fd); +} + +void test_cpumask_iter (void ) +{ +DECLARE_LIBBPF_OPTS (bpf_iter_attach_opts, opts); +int nr_possible, cgrp_fd, pid, err, cnt, i; +struct test_cpumask_iter *skel =NULL ;
= NULL is not needed.
Quoted 22 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +22
-0+union bpf_iter_link_info linfo; +int cpu_ids[] = {1 ,3 ,4 ,5 }; +struct bpf_link *link; +cpu_set_t set; + + skel =test_cpumask_iter__open_and_load (); +if (!ASSERT_OK_PTR (skel,"test_for_each_cpu__open_and_load" )) +return ; + +if (setup_cgroup_environment ()) + goto destroy; + +/* Utilize the cgroup iter */ + cgrp_fd =get_root_cgroup (); +if (!ASSERT_GE (cgrp_fd,0 ,"create cgrp" )) + goto cleanup; + +memset (&linfo,0 ,sizeof (linfo)); + linfo.cgroup.cgroup_fd = cgrp_fd; + linfo.cgroup.order =BPF_CGROUP_ITER_SELF_ONLY ; + opts.link_info = &linfo; + opts.link_info_len =sizeof (linfo);
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 88
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10
-0+ + link =bpf_program__attach_iter (skel->progs.cpu_cgroup, &opts); +if (!ASSERT_OK_PTR (link,"attach_iter" )) + goto close_fd; + + skel->bss->target_pid =1 ; +/* In case init task is set CPU affinity */ + err =sched_getaffinity (1 ,sizeof (set), &set); +if (!ASSERT_OK (err,"setaffinity" )) + goto close_fd;
goto free_link.
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +6
-0+ + cnt =CPU_COUNT (&set); + nr_possible =bpf_num_possible_cpus (); +if (test__start_subtest ("init_pid" )) +/* curent task is running. */ +verify_percpu_data (link, cnt, cnt == nr_possible ?1 :0 );
[...]
Re: [PATCH v3 bpf-next 3/3] selftests/bpf: Add selftests for cpumask iter
Quoted 9 lines · Yonghong Song
Quoted 6 lines · Yafang Shao
Within the BPF program, we leverage the cgroup iterator to iterate through percpu runqueue data, specifically the '
nr_running' metric. Subsequently
we expose this data to userspace by means of a sequence file.The CPU affinity for the cpumask is determined by the PID of a task:
Quoted 7 lines
Quoted 5 lines
- PID of the init task (PID 1)
We typically don't set CPU affinity for init task and thus we can iterate across all possible CPUs. However, in scenarios where you've set CPU affinity for the init task, you should set the cpumask of your current task to full-F. Then proceed to iterate through all possible CPUs usingWat is full-F? It would be good if you can clarify in the commit message.
I mean set all available CPUs for the task.
Will clarify it in the next version.
Quoted 56 lines
Quoted 56 lines
the current task.
- PID of a task with defined CPU affinity
The aim here is to iterate through a specific cpumask. This scenario aligns with tasks residing within a cpuset cgroup.
- Invalid PID (e.g., PID -1)
No cpumask is available in this case.The result as follows,
#65/1 cpumask_iter/init_pid:OK #65/2 cpumask_iter/invalid_pid:OK #65/3 cpumask_iter/self_pid_one_cpu:OK #65/4 cpumask_iter/self_pid_multi_cpus:OK #65 cpumask_iter:OKSummary: 1/4 PASSED, 0 SKIPPED, 0 FAILED
CONFIG_PSI=yis required for this testcase.Signed-off-by Yafang Shao <[email protected]>--- tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cpumask_iter.c | 134 ++++++++++++++++++ .../selftests/bpf/progs/cpumask_common.h | 3 + .../selftests/bpf/progs/test_cpumask_iter.c | 56 ++++++++ 4 files changed, 194 insertions(+)create mode 100644
tools/testing/selftests/bpf/prog_tests/cpumask_iter.ccreate mode 100644tools/testing/selftests/bpf/progs/test_cpumask_iter.ctools/testing/selftests/bpf/config +1
-0@@ -78,6 +78,7 @@ CONFIG_NF_CONNTRACK_MARK=y CONFIG_NF_DEFRAG_IPV4=y CONFIG_NF_DEFRAG_IPV6=y CONFIG_NF_NAT=y +CONFIG_PSI=y CONFIG_RC_CORE=y CONFIG_SECURITY=y CONFIG_SECURITYFS=ytools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10
-0@@ -0,0 +1,134 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Yafang Shao <[email protected]> */ + +#define _GNU_SOURCE +#include <sched.h> +#include <stdio.h> +#include <unistd.h> + +#include <test_progs.h> +#include "cgroup_helpers.h"
Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 16
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +6
-0+#include "test_cpumask_iter.skel.h" + +static void verify_percpu_data (struct bpf_link *link,int nr_cpu_exp,int nr_running_exp) +{ +int iter_fd, len, item, nr_running, psi_running, nr_cpus; +static char buf[128 ];why static?
Will remove it.
Quoted 26 lines
Quoted 26 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +25
-0+size_t left; +char *p; + + iter_fd =bpf_iter_create (bpf_link__fd (link)); +if (!ASSERT_GE (iter_fd,0 ,"iter_fd" )) +return ; + +memset (buf,0 ,sizeof (buf)); + left =ARRAY_SIZE (buf); + p = buf; +while ((len =read (iter_fd, p, left)) >0 ) { + p += len; + left -= len; + } + + item =sscanf (buf,"nr_running %u nr_cpus %u psi_running %u\n" , + &nr_running, &nr_cpus, &psi_running); +if (nr_cpu_exp ==-1 ) { +ASSERT_EQ (item,-1 ,"seq_format" ); + goto out; + } + +ASSERT_EQ (item,3 ,"seq_format" ); +ASSERT_GE (nr_running, nr_running_exp,"nr_running" ); +ASSERT_GE (psi_running, nr_running_exp,"psi_running" );
Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 46
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +5
-0+ASSERT_EQ (nr_cpus, nr_cpu_exp,"nr_cpus" ); + +/* read() after iter finishes should be ok. */ +if (len ==0 ) +ASSERT_OK (read (iter_fd, buf,sizeof (buf)),"second_read" );The above 'if' statement is irrelevant to the main purpose of this test and can be removed.
Will remove it.
Quoted 13 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 56
Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10
-0+ +out: +close (iter_fd); +} + +void test_cpumask_iter (void ) +{ +DECLARE_LIBBPF_OPTS (bpf_iter_attach_opts, opts); +int nr_possible, cgrp_fd, pid, err, cnt, i; +struct test_cpumask_iter *skel =NULL ;= NULL is not needed.
Will change it.
Quoted 27 lines
Quoted 27 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +26
-0+union bpf_iter_link_info linfo; +int cpu_ids[] = {1 ,3 ,4 ,5 }; +struct bpf_link *link; +cpu_set_t set; + + skel =test_cpumask_iter__open_and_load (); +if (!ASSERT_OK_PTR (skel,"test_for_each_cpu__open_and_load" )) +return ; + +if (setup_cgroup_environment ()) + goto destroy; + +/* Utilize the cgroup iter */ + cgrp_fd =get_root_cgroup (); +if (!ASSERT_GE (cgrp_fd,0 ,"create cgrp" )) + goto cleanup; + +memset (&linfo,0 ,sizeof (linfo)); + linfo.cgroup.cgroup_fd = cgrp_fd; + linfo.cgroup.order =BPF_CGROUP_ITER_SELF_ONLY ; + opts.link_info = &linfo; + opts.link_info_len =sizeof (linfo); + + link =bpf_program__attach_iter (skel->progs.cpu_cgroup, &opts); +if (!ASSERT_OK_PTR (link,"attach_iter" )) + goto close_fd;
Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 88
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +6
-0+ + skel->bss->target_pid =1 ; +/* In case init task is set CPU affinity */ + err =sched_getaffinity (1 ,sizeof (set), &set); +if (!ASSERT_OK (err,"setaffinity" )) + goto close_fd;goto
free_link.
Nice catch. will change it.
Quoted 8 lines
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +6
-0+ + cnt =CPU_COUNT (&set); + nr_possible =bpf_num_possible_cpus (); +if (test__start_subtest ("init_pid" )) +/* curent task is running. */ +verify_percpu_data (link, cnt, cnt == nr_possible ?1 :0 );[...]
Signature · 2 lines
Regards Yafang