Skip to contentSkip to conversation

bpf: Add bpf_iter_cpumask

PATCH Superseded Yafang Shao 3 participants 12 messages Jan 23, 2024 – Jan 25, 2024

Subsystems (4)

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v4 cover bpf: Add bpf_iter_cpumask

Three new kfuncs, namely bpf_iter_cpumask_{new,next,destroy}, have been added for the new bpf_iter_cpumask functionality. These kfuncs enable the iteration of percpu data, such as runqueues, system_group_pcpu, and more.

In our specific use case, we leverage the cgroup iterator to traverse percpu data, subsequently exposing it to userspace through a seq file. Refer to the test cases in patch #2 for further context and examples.

Changes:
- v3 -> v4:
- Use a copy of cpumask to avoid potential use-after-free (Yonghong)
- Various code improvement in selftests (Yonghong)
- v2 -> v3:
- Define KF_RCU_PROTECTED for bpf_iter_cpumask_new (Alexei)
- Code improvement in selftests
- Fix build error in selftest due to CONFIG_PSI=n reported by kernel test robot <[email protected]>
- v1 -> v2:
- Avoid changing cgroup subsystem (Tejun)
- Remove bpf_cpumask_set_from_pid(), and use bpf_cpumask_copy() instead (Tejun)
- Use int cpu; field in bpf_iter_cpumask_kern (Andrii)
- bpf: Add new bpf helper bpf_for_each_cpu
https://lwn.net/ml/bpf/[email protected]/

Yafang Shao (3):
bpf: Add bpf_iter_cpumask kfuncs
bpf, doc: Add document for cpumask iter
selftests/bpf: Add selftests for cpumask iter

 Documentation/bpf/cpumasks.rst                |  17 +++
 kernel/bpf/cpumask.c                          |  82 +++++++++++
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/cpumask_iter.c   | 130 ++++++++++++++++++
 .../selftests/bpf/progs/cpumask_common.h      |   3 +
 .../selftests/bpf/progs/test_cpumask_iter.c   |  56 ++++++++
 6 files changed, 289 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cpumask_iter.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_cpumask_iter.c
Signature · 1 line
2.39.1
#1 lore raw

PATCH v4 1/3 bpf: Add bpf_iter_cpumask kfuncs

Add three new kfuncs for bpf_iter_cpumask.
- bpf_iter_cpumask_new
KF_RCU is defined because the cpumask must be a RCU trusted pointer such as task->cpus_ptr.
- bpf_iter_cpumask_next
- bpf_iter_cpumask_destroy

These new kfuncs facilitate the iteration of percpu data, such as runqueues, psi_cgroup_cpu, and more.

---
 kernel/bpf/cpumask.c | 82 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)
kernel/bpf/cpumask.c +82 -0
index 2e73533a3811..474072a235d6 100644
@@ -422,6 +422,85 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)
 	return cpumask_weight(cpumask);
 }
 
+struct bpf_iter_cpumask {
+	__u64 __opaque[2];
+} __aligned(8);
+
+struct bpf_iter_cpumask_kern {
+	struct cpumask *mask;
+	int cpu;
+} __aligned(8);
+
+/**
+ * bpf_iter_cpumask_new() - Create a new bpf_iter_cpumask for a specified cpumask
+ * @it: The new bpf_iter_cpumask to be created.
+ * @mask: The cpumask to be iterated over.
+ *
+ * This function initializes a new bpf_iter_cpumask structure for iterating over
+ * the specified CPU mask. It assigns the provided cpumask to the newly created
+ * bpf_iter_cpumask @it for subsequent iteration operations.
+ *
+ * On success, 0 is returen. On failure, ERR is returned.
+ */
+__bpf_kfunc int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask)
+{
+	struct bpf_iter_cpumask_kern *kit = (void *)it;
+
+	BUILD_BUG_ON(sizeof(struct bpf_iter_cpumask_kern) > sizeof(struct bpf_iter_cpumask));
+	BUILD_BUG_ON(__alignof__(struct bpf_iter_cpumask_kern) !=
+		     __alignof__(struct bpf_iter_cpumask));
+
+	kit->mask = bpf_mem_alloc(&bpf_global_ma, sizeof(struct cpumask));
+	if (!kit->mask)
+		return -ENOMEM;
+
+	cpumask_copy(kit->mask, mask);
+	kit->cpu = -1;
+	return 0;
+}
+
+/**
+ * bpf_iter_cpumask_next() - Get the next CPU in a bpf_iter_cpumask
+ * @it: The bpf_iter_cpumask
+ *
+ * This function retrieves a pointer to the number of the next CPU within the
+ * specified bpf_iter_cpumask. It allows sequential access to CPUs within the
+ * cpumask. If there are no further CPUs available, it returns NULL.
+ *
+ * Returns a pointer to the number of the next CPU in the cpumask or NULL if no
+ * further CPUs.
+ */
+__bpf_kfunc int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it)
+{
+	struct bpf_iter_cpumask_kern *kit = (void *)it;
+	const struct cpumask *mask = kit->mask;
+	int cpu;
+
+	if (!mask)
+		return NULL;
+	cpu = cpumask_next(kit->cpu, mask);
+	if (cpu >= nr_cpu_ids)
+		return NULL;
+
+	kit->cpu = cpu;
+	return &kit->cpu;
+}
+
+/**
+ * bpf_iter_cpumask_destroy() - Destroy a bpf_iter_cpumask
+ * @it: The bpf_iter_cpumask to be destroyed.
+ *
+ * Destroy the resource assiciated with the bpf_iter_cpumask.
+ */
+__bpf_kfunc void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it)
+{
+	struct bpf_iter_cpumask_kern *kit = (void *)it;
+
+	if (!kit->mask)
+		return;
+	bpf_mem_free(&bpf_global_ma, kit->mask);
+}
+
 __bpf_kfunc_end_defs();
 
 BTF_SET8_START(cpumask_kfunc_btf_ids)
@@ -450,6 +529,9 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU)
 BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU)
 BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU)
 BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU)
+BTF_ID_FLAGS(func, bpf_iter_cpumask_new, KF_ITER_NEW | KF_RCU)
+BTF_ID_FLAGS(func, bpf_iter_cpumask_next, KF_ITER_NEXT | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_iter_cpumask_destroy, KF_ITER_DESTROY)
 BTF_SET8_END(cpumask_kfunc_btf_ids)
 
 static const struct btf_kfunc_id_set cpumask_kfunc_set = {
Signature · 1 line
2.39.1
#2 lore raw

Re: [PATCH v4 bpf-next 1/3] bpf: Add bpf_iter_cpumask kfuncs

Quoted 11 lines · Yafang Shao

Add three new kfuncs for bpf_iter_cpumask.
- bpf_iter_cpumask_new
KF_RCU is defined because the cpumask must be a RCU trusted pointer such as task->cpus_ptr.
- bpf_iter_cpumask_next
- bpf_iter_cpumask_destroy

These new kfuncs facilitate the iteration of percpu data, such as runqueues, psi_cgroup_cpu, and more.

Thanks for working on this, this will be nice to have!

Quoted 10 lines
---
 kernel/bpf/cpumask.c | 82 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)
kernel/bpf/cpumask.c +0 -0
index 2e73533a3811..474072a235d6 100644
@@ -422,6 +422,85 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)
 	return cpumask_weight(cpumask);

kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 432

Quoted 10 lines
kernel/bpf/cpumask.c +8 -0
 }
 
+struct bpf_iter_cpumask {
+	__u64 __opaque[2];
+} __aligned(8);
+
+struct bpf_iter_cpumask_kern {
+	struct cpumask *mask;
+	int cpu;
+} __aligned(8);

Why do we need both of these if we're not going to put the opaque iterator in UAPI?

Quoted 9 lines
kernel/bpf/cpumask.c +9 -0
+
+/**
+ * bpf_iter_cpumask_new() - Create a new bpf_iter_cpumask for a specified cpumask
+ * @it: The new bpf_iter_cpumask to be created.
+ * @mask: The cpumask to be iterated over.
+ *
+ * This function initializes a new bpf_iter_cpumask structure for iterating over
+ * the specified CPU mask. It assigns the provided cpumask to the newly created
+ * bpf_iter_cpumask @it for subsequent iteration operations.

kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 451

Quoted 10 lines
kernel/bpf/cpumask.c +10 -0
+ *
+ * On success, 0 is returen. On failure, ERR is returned.
+ */
+__bpf_kfunc int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask)
+{
+	struct bpf_iter_cpumask_kern *kit = (void *)it;
+
+	BUILD_BUG_ON(sizeof(struct bpf_iter_cpumask_kern) > sizeof(struct bpf_iter_cpumask));
+	BUILD_BUG_ON(__alignof__(struct bpf_iter_cpumask_kern) !=
+		     __alignof__(struct bpf_iter_cpumask));

Why are we checking > in the first expression instead of just plain equality?

kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 453

Quoted 2 lines
kernel/bpf/cpumask.c +2 -0
+
+	kit->mask = bpf_mem_alloc(&bpf_global_ma, sizeof(struct cpumask));

Probably better to use cpumask_size() here.

Quoted 67 lines
kernel/bpf/cpumask.c +53 -0
+	if (!kit->mask)
+		return -ENOMEM;
+
+	cpumask_copy(kit->mask, mask);
+	kit->cpu = -1;
+	return 0;
+}
+
+/**
+ * bpf_iter_cpumask_next() - Get the next CPU in a bpf_iter_cpumask
+ * @it: The bpf_iter_cpumask
+ *
+ * This function retrieves a pointer to the number of the next CPU within the
+ * specified bpf_iter_cpumask. It allows sequential access to CPUs within the
+ * cpumask. If there are no further CPUs available, it returns NULL.
+ *
+ * Returns a pointer to the number of the next CPU in the cpumask or NULL if no
+ * further CPUs.
+ */
+__bpf_kfunc int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it)
+{
+	struct bpf_iter_cpumask_kern *kit = (void *)it;
+	const struct cpumask *mask = kit->mask;
+	int cpu;
+
+	if (!mask)
+		return NULL;
+	cpu = cpumask_next(kit->cpu, mask);
+	if (cpu >= nr_cpu_ids)
+		return NULL;
+
+	kit->cpu = cpu;
+	return &kit->cpu;
+}
+
+/**
+ * bpf_iter_cpumask_destroy() - Destroy a bpf_iter_cpumask
+ * @it: The bpf_iter_cpumask to be destroyed.
+ *
+ * Destroy the resource assiciated with the bpf_iter_cpumask.
+ */
+__bpf_kfunc void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it)
+{
+	struct bpf_iter_cpumask_kern *kit = (void *)it;
+
+	if (!kit->mask)
+		return;
+	bpf_mem_free(&bpf_global_ma, kit->mask);
+}
+
 __bpf_kfunc_end_defs();
 
 BTF_SET8_START(cpumask_kfunc_btf_ids)
@@ -450,6 +529,9 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU)
 BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU)
 BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU)
 BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU)
+BTF_ID_FLAGS(func, bpf_iter_cpumask_new, KF_ITER_NEW | KF_RCU)
+BTF_ID_FLAGS(func, bpf_iter_cpumask_next, KF_ITER_NEXT | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_iter_cpumask_destroy, KF_ITER_DESTROY)
 BTF_SET8_END(cpumask_kfunc_btf_ids)
 
 static const struct btf_kfunc_id_set cpumask_kfunc_set = {
Signature · 1 line
2.39.1
#3 lore raw

Re: [PATCH v4 bpf-next 1/3] bpf: Add bpf_iter_cpumask kfuncs

Quoted 28 lines · David Vernet
Quoted 11 lines · Yafang Shao

Add three new kfuncs for bpf_iter_cpumask.
- bpf_iter_cpumask_new
KF_RCU is defined because the cpumask must be a RCU trusted pointer such as task->cpus_ptr.
- bpf_iter_cpumask_next
- bpf_iter_cpumask_destroy

These new kfuncs facilitate the iteration of percpu data, such as runqueues, psi_cgroup_cpu, and more.

Thanks for working on this, this will be nice to have!

Quoted 12 lines
---
 kernel/bpf/cpumask.c | 82 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)
kernel/bpf/cpumask.c +0 -0
index 2e73533a3811..474072a235d6 100644
@@ -422,6 +422,85 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)
      return cpumask_weight(cpumask);
 }
 
Quoted 11 lines

kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 432

Quoted 8 lines
kernel/bpf/cpumask.c +8 -0
+struct bpf_iter_cpumask {
+     __u64 __opaque[2];
+} __aligned(8);
+
+struct bpf_iter_cpumask_kern {
+     struct cpumask *mask;
+     int cpu;
+} __aligned(8);

Why do we need both of these if we're not going to put the opaque iterator in UAPI?

Good point! Will remove it.
It aligns with the pattern seen in bpf_iter_{css,task,task_vma,task_css}_kern, suggesting that we should indeed eliminate them.

Quoted 15 lines
Quoted 15 lines
kernel/bpf/cpumask.c +14 -0
+
+/**
+ * bpf_iter_cpumask_new() - Create a new bpf_iter_cpumask for a specified cpumask
+ * @it: The new bpf_iter_cpumask to be created.
+ * @mask: The cpumask to be iterated over.
+ *
+ * This function initializes a new bpf_iter_cpumask structure for iterating over
+ * the specified CPU mask. It assigns the provided cpumask to the newly created
+ * bpf_iter_cpumask @it for subsequent iteration operations.
+ *
+ * On success, 0 is returen. On failure, ERR is returned.
+ */
+__bpf_kfunc int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask)
+{
Quoted 8 lines

kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 451

Quoted 5 lines
kernel/bpf/cpumask.c +5 -0
+     struct bpf_iter_cpumask_kern *kit = (void *)it;
+
+     BUILD_BUG_ON(sizeof(struct bpf_iter_cpumask_kern) > sizeof(struct bpf_iter_cpumask));
+     BUILD_BUG_ON(__alignof__(struct bpf_iter_cpumask_kern) !=
+                  __alignof__(struct bpf_iter_cpumask));

Why are we checking > in the first expression instead of just plain equality?

Similar to the previous case, it aligns with others. Once we eliminate the struct bpf_iter_cpumask_kern, we can safely discard these BUILD_BUG_ON() statements as well.

Quoted 5 lines

kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 453

Quoted 3 lines
kernel/bpf/cpumask.c +2 -0
+
+     kit->mask = bpf_mem_alloc(&bpf_global_ma, sizeof(struct cpumask));

Probably better to use cpumask_size() here.

will use it.

Quoted 68 lines
Quoted 68 lines
kernel/bpf/cpumask.c +53 -1
+     if (!kit->mask)
+             return -ENOMEM;
+
+     cpumask_copy(kit->mask, mask);
+     kit->cpu = -1;
+     return 0;
+}
+
+/**
+ * bpf_iter_cpumask_next() - Get the next CPU in a bpf_iter_cpumask
+ * @it: The bpf_iter_cpumask
+ *
+ * This function retrieves a pointer to the number of the next CPU within the
+ * specified bpf_iter_cpumask. It allows sequential access to CPUs within the
+ * cpumask. If there are no further CPUs available, it returns NULL.
+ *
+ * Returns a pointer to the number of the next CPU in the cpumask or NULL if no
+ * further CPUs.
+ */
+__bpf_kfunc int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it)
+{
+     struct bpf_iter_cpumask_kern *kit = (void *)it;
+     const struct cpumask *mask = kit->mask;
+     int cpu;
+
+     if (!mask)
+             return NULL;
+     cpu = cpumask_next(kit->cpu, mask);
+     if (cpu >= nr_cpu_ids)
+             return NULL;
+
+     kit->cpu = cpu;
+     return &kit->cpu;
+}
+
+/**
+ * bpf_iter_cpumask_destroy() - Destroy a bpf_iter_cpumask
+ * @it: The bpf_iter_cpumask to be destroyed.
+ *
+ * Destroy the resource assiciated with the bpf_iter_cpumask.
+ */
+__bpf_kfunc void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it)
+{
+     struct bpf_iter_cpumask_kern *kit = (void *)it;
+
+     if (!kit->mask)
+             return;
+     bpf_mem_free(&bpf_global_ma, kit->mask);
+}
+
 __bpf_kfunc_end_defs();
 
 BTF_SET8_START(cpumask_kfunc_btf_ids)
@@ -450,6 +529,9 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU)
 BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU)
 BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU)
 BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU)
+BTF_ID_FLAGS(func, bpf_iter_cpumask_new, KF_ITER_NEW | KF_RCU)
+BTF_ID_FLAGS(func, bpf_iter_cpumask_next, KF_ITER_NEXT | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_iter_cpumask_destroy, KF_ITER_DESTROY)
 BTF_SET8_END(cpumask_kfunc_btf_ids)
 
 static const struct btf_kfunc_id_set cpumask_kfunc_set = {
--

2.39.1

Signature · 2 lines
Regards
Yafang
#4 lore raw

Re: [PATCH v4 bpf-next 1/3] bpf: Add bpf_iter_cpumask kfuncs

Quoted 42 lines · Yafang Shao
Quoted 28 lines · David Vernet
Quoted 11 lines · Yafang Shao

Add three new kfuncs for bpf_iter_cpumask.
- bpf_iter_cpumask_new
KF_RCU is defined because the cpumask must be a RCU trusted pointer such as task->cpus_ptr.
- bpf_iter_cpumask_next
- bpf_iter_cpumask_destroy

These new kfuncs facilitate the iteration of percpu data, such as runqueues, psi_cgroup_cpu, and more.

Thanks for working on this, this will be nice to have!

Quoted 12 lines
---
 kernel/bpf/cpumask.c | 82 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)
kernel/bpf/cpumask.c +0 -0
index 2e73533a3811..474072a235d6 100644
@@ -422,6 +422,85 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)
      return cpumask_weight(cpumask);
 }
 
Quoted 11 lines

kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 432

Quoted 8 lines
kernel/bpf/cpumask.c +8 -0
+struct bpf_iter_cpumask {
+     __u64 __opaque[2];
+} __aligned(8);
+
+struct bpf_iter_cpumask_kern {
+     struct cpumask *mask;
+     int cpu;
+} __aligned(8);

Why do we need both of these if we're not going to put the opaque iterator in UAPI?

Quoted 5 lines

Good point! Will remove it.
It aligns with the pattern seen in bpf_iter_{css,task,task_vma,task_css}_kern, suggesting that we should indeed eliminate them.

It feels a bit cleaner to have API-oriented (despite being unstable and coming from vmlinux.h) iter struct like bpf_iter_cpumask with just "__opaque" field. And then having _kern variant with actual memory layout. Technically _kern struct could grow smaller.

I certainly wanted this split for bpf_iter_num as that one is more of a general purpose and stable struct. It's less relevant for more unstable iters here.

Quoted 108 lines
Quoted 15 lines
Quoted 15 lines
kernel/bpf/cpumask.c +14 -0
+
+/**
+ * bpf_iter_cpumask_new() - Create a new bpf_iter_cpumask for a specified cpumask
+ * @it: The new bpf_iter_cpumask to be created.
+ * @mask: The cpumask to be iterated over.
+ *
+ * This function initializes a new bpf_iter_cpumask structure for iterating over
+ * the specified CPU mask. It assigns the provided cpumask to the newly created
+ * bpf_iter_cpumask @it for subsequent iteration operations.
+ *
+ * On success, 0 is returen. On failure, ERR is returned.
+ */
+__bpf_kfunc int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask)
+{
Quoted 8 lines

kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 451

Quoted 5 lines
kernel/bpf/cpumask.c +5 -0
+     struct bpf_iter_cpumask_kern *kit = (void *)it;
+
+     BUILD_BUG_ON(sizeof(struct bpf_iter_cpumask_kern) > sizeof(struct bpf_iter_cpumask));
+     BUILD_BUG_ON(__alignof__(struct bpf_iter_cpumask_kern) !=
+                  __alignof__(struct bpf_iter_cpumask));

Why are we checking > in the first expression instead of just plain equality?

Similar to the previous case, it aligns with others. Once we eliminate the struct bpf_iter_cpumask_kern, we can safely discard these BUILD_BUG_ON() statements as well.

Quoted 5 lines

kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 453

Quoted 3 lines
kernel/bpf/cpumask.c +2 -0
+
+     kit->mask = bpf_mem_alloc(&bpf_global_ma, sizeof(struct cpumask));

Probably better to use cpumask_size() here.

will use it.

Quoted 63 lines
Quoted 63 lines
kernel/bpf/cpumask.c +53 -0
+     if (!kit->mask)
+             return -ENOMEM;
+
+     cpumask_copy(kit->mask, mask);
+     kit->cpu = -1;
+     return 0;
+}
+
+/**
+ * bpf_iter_cpumask_next() - Get the next CPU in a bpf_iter_cpumask
+ * @it: The bpf_iter_cpumask
+ *
+ * This function retrieves a pointer to the number of the next CPU within the
+ * specified bpf_iter_cpumask. It allows sequential access to CPUs within the
+ * cpumask. If there are no further CPUs available, it returns NULL.
+ *
+ * Returns a pointer to the number of the next CPU in the cpumask or NULL if no
+ * further CPUs.
+ */
+__bpf_kfunc int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it)
+{
+     struct bpf_iter_cpumask_kern *kit = (void *)it;
+     const struct cpumask *mask = kit->mask;
+     int cpu;
+
+     if (!mask)
+             return NULL;
+     cpu = cpumask_next(kit->cpu, mask);
+     if (cpu >= nr_cpu_ids)
+             return NULL;
+
+     kit->cpu = cpu;
+     return &kit->cpu;
+}
+
+/**
+ * bpf_iter_cpumask_destroy() - Destroy a bpf_iter_cpumask
+ * @it: The bpf_iter_cpumask to be destroyed.
+ *
+ * Destroy the resource assiciated with the bpf_iter_cpumask.
+ */
+__bpf_kfunc void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it)
+{
+     struct bpf_iter_cpumask_kern *kit = (void *)it;
+
+     if (!kit->mask)
+             return;
+     bpf_mem_free(&bpf_global_ma, kit->mask);
+}
+
 __bpf_kfunc_end_defs();
 
 BTF_SET8_START(cpumask_kfunc_btf_ids)
@@ -450,6 +529,9 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU)
 BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU)
 BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU)
 BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU)
+BTF_ID_FLAGS(func, bpf_iter_cpumask_new, KF_ITER_NEW | KF_RCU)
+BTF_ID_FLAGS(func, bpf_iter_cpumask_next, KF_ITER_NEXT | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_iter_cpumask_destroy, KF_ITER_DESTROY)
 BTF_SET8_END(cpumask_kfunc_btf_ids)
 
Quoted 5 lines
Quoted 5 lines
kernel/bpf/cpumask.c +0 -1
 static const struct btf_kfunc_id_set cpumask_kfunc_set = {
--

2.39.1

--
Regards
Yafang

#5 lore raw

Re: [PATCH v4 bpf-next 1/3] bpf: Add bpf_iter_cpumask kfuncs

Quoted 50 lines · Andrii Nakryiko
Quoted 42 lines · Yafang Shao
Quoted 28 lines · David Vernet
Quoted 11 lines · Yafang Shao

Add three new kfuncs for bpf_iter_cpumask.
- bpf_iter_cpumask_new
KF_RCU is defined because the cpumask must be a RCU trusted pointer such as task->cpus_ptr.
- bpf_iter_cpumask_next
- bpf_iter_cpumask_destroy

These new kfuncs facilitate the iteration of percpu data, such as runqueues, psi_cgroup_cpu, and more.

Thanks for working on this, this will be nice to have!

Quoted 12 lines
---
 kernel/bpf/cpumask.c | 82 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)
kernel/bpf/cpumask.c +0 -0
index 2e73533a3811..474072a235d6 100644
@@ -422,6 +422,85 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)
      return cpumask_weight(cpumask);
 }
 
Quoted 11 lines

kernel/bpf/cpumask.c __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) · line 432

Quoted 8 lines
kernel/bpf/cpumask.c +8 -0
+struct bpf_iter_cpumask {
+     __u64 __opaque[2];
+} __aligned(8);
+
+struct bpf_iter_cpumask_kern {
+     struct cpumask *mask;
+     int cpu;
+} __aligned(8);

Why do we need both of these if we're not going to put the opaque iterator in UAPI?

Quoted 5 lines

Good point! Will remove it.
It aligns with the pattern seen in bpf_iter_{css,task,task_vma,task_css}_kern, suggesting that we should indeed eliminate them.

Quoted 8 lines

It feels a bit cleaner to have API-oriented (despite being unstable and coming from vmlinux.h) iter struct like bpf_iter_cpumask with just "__opaque" field. And then having _kern variant with actual memory layout. Technically _kern struct could grow smaller.

I certainly wanted this split for bpf_iter_num as that one is more of a general purpose and stable struct. It's less relevant for more unstable iters here.

Understood. Thanks for your explanation.

Signature · 2 lines
Regards
Yafang
#6 lore raw

PATCH v4 2/3 bpf, doc: Add document for cpumask iter

In reply to Yafang Shao

This patch adds the document for the newly added cpumask iterator kfuncs.

---
 Documentation/bpf/cpumasks.rst | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
Documentation/bpf/cpumasks.rst +17 -0
index b5d47a04da5d..523f377afc6e 100644
@@ -372,6 +372,23 @@ used.
 .. _tools/testing/selftests/bpf/progs/cpumask_success.c:
    https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/progs/cpumask_success.c
 
+3.3 cpumask iterator
+--------------------
+
+The cpumask iterator enables the iteration of percpu data, such as runqueues,
+system_group_pcpu, and more.
+
+.. kernel-doc:: kernel/bpf/cpumask.c
+   :identifiers: bpf_iter_cpumask_new bpf_iter_cpumask_next
+                 bpf_iter_cpumask_destroy
+
+----
+
+Some example usages of the cpumask iterator can be found in
+`tools/testing/selftests/bpf/progs/test_cpumask_iter.c`_.
+
+.. _tools/testing/selftests/bpf/progs/test_cpumask_iter.c:
+   https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/progs/test_cpumask_iter.c
 
 4. Adding BPF cpumask kfuncs
 ============================
Signature · 1 line
2.39.1
#7 lore raw

Re: [PATCH v4 bpf-next 2/3] bpf, doc: Add document for cpumask iter

Quoted 15 lines · Yafang Shao

This patch adds the document for the newly added cpumask iterator kfuncs.

---
 Documentation/bpf/cpumasks.rst | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
Documentation/bpf/cpumasks.rst +0 -0
index b5d47a04da5d..523f377afc6e 100644
@@ -372,6 +372,23 @@ used.
 .. _tools/testing/selftests/bpf/progs/cpumask_success.c:
    https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/progs/cpumask_success.c

Documentation/bpf/cpumasks.rst used. · line 383

Quoted 10 lines
Documentation/bpf/cpumasks.rst +9 -0
 
+3.3 cpumask iterator
+--------------------
+
+The cpumask iterator enables the iteration of percpu data, such as runqueues,
+system_group_pcpu, and more.
+
+.. kernel-doc:: kernel/bpf/cpumask.c
+   :identifiers: bpf_iter_cpumask_new bpf_iter_cpumask_next
+                 bpf_iter_cpumask_destroy

Practically speaking I don't think documenting these kfuncs is going to be super useful to most users. I expect we'd wrap this in a macro, just like we do for bpf_for(), and I think it would be much more useful to a reader to show how they can use such a macro with a full, self-contained example rather than just embedding the doxygen comment here.

Documentation/bpf/cpumasks.rst used. · line 391

Quoted 8 lines
Documentation/bpf/cpumasks.rst +8 -0
+
+----
+
+Some example usages of the cpumask iterator can be found in
+`tools/testing/selftests/bpf/progs/test_cpumask_iter.c`_.
+
+.. _tools/testing/selftests/bpf/progs/test_cpumask_iter.c:
+   https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/progs/test_cpumask_iter.c

I know it's typical for BPF to link to selftests like this, but I personally strongly prefer actual examples in the documentation. We have examples elsewhere in this file, so can we please do the same here?

Quoted 7 lines
Documentation/bpf/cpumasks.rst +0 -0
 4. Adding BPF cpumask kfuncs
 ============================
Signature · 1 line
2.39.1
#8 lore raw

Re: [PATCH v4 bpf-next 2/3] bpf, doc: Add document for cpumask iter

Quoted 18 lines · David Vernet
Quoted 16 lines · Yafang Shao

This patch adds the document for the newly added cpumask iterator kfuncs.

---
 Documentation/bpf/cpumasks.rst | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
Documentation/bpf/cpumasks.rst +0 -0
index b5d47a04da5d..523f377afc6e 100644
@@ -372,6 +372,23 @@ used.
 .. _tools/testing/selftests/bpf/progs/cpumask_success.c:
    https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/progs/cpumask_success.c
 
Quoted 15 lines

Documentation/bpf/cpumasks.rst used. · line 383

Quoted 9 lines
Documentation/bpf/cpumasks.rst +9 -0
+3.3 cpumask iterator
+--------------------
+
+The cpumask iterator enables the iteration of percpu data, such as runqueues,
+system_group_pcpu, and more.
+
+.. kernel-doc:: kernel/bpf/cpumask.c
+   :identifiers: bpf_iter_cpumask_new bpf_iter_cpumask_next
+                 bpf_iter_cpumask_destroy

Practically speaking I don't think documenting these kfuncs is going to be super useful to most users. I expect we'd wrap this in a macro, just like we do for bpf_for(), and I think it would be much more useful to a reader to show how they can use such a macro with a full, self-contained example rather than just embedding the doxygen comment here.

Agree.

Quoted 13 lines

Documentation/bpf/cpumasks.rst used. · line 391

Quoted 9 lines
Documentation/bpf/cpumasks.rst +8 -0
+
+----
+
+Some example usages of the cpumask iterator can be found in
+`tools/testing/selftests/bpf/progs/test_cpumask_iter.c`_.
+
+.. _tools/testing/selftests/bpf/progs/test_cpumask_iter.c:
+   https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/progs/test_cpumask_iter.c

I know it's typical for BPF to link to selftests like this, but I personally strongly prefer actual examples in the documentation. We have examples elsewhere in this file, so can we please do the same here?

will do it.

Signature · 2 lines
Regards
Yafang
#9 lore raw

PATCH v4 3/3 selftests/bpf: Add selftests for cpumask iter

In reply to Yafang Shao

Within the BPF program, we leverage the cgroup iterator to iterate through percpu runqueue data, specifically the 'nr_running' metric. Subsequently
we expose this data to userspace by means of a sequence file.

The CPU affinity for the cpumask is determined by the PID of a task:

- PID of the init task (PID 1)
We typically don't set CPU affinity for init task and thus we can iterate across all possible CPUs using the init task. However, in scenarios where you've set CPU affinity for the init task, you should set your current-task's cpu affinity to all possible CPUs and then proceed to iterate through all possible CPUs using the current task.
- PID of a task with defined CPU affinity
The aim here is to iterate through a specific cpumask. This scenario aligns with tasks residing within a cpuset cgroup.
- Invalid PID (e.g., PID -1)
No cpumask is available in this case.

The result as follows,

  #65/1    cpumask_iter/init_pid:OK
  #65/2    cpumask_iter/invalid_pid:OK
  #65/3    cpumask_iter/self_pid_one_cpu:OK
  #65/4    cpumask_iter/self_pid_multi_cpus:OK
  #65      cpumask_iter:OK

Summary: 1/4 PASSED, 0 SKIPPED, 0 FAILED

CONFIG_PSI=y is required for this testcase.

---
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/cpumask_iter.c   | 130 ++++++++++++++++++
 .../selftests/bpf/progs/cpumask_common.h      |   3 +
 .../selftests/bpf/progs/test_cpumask_iter.c   |  56 ++++++++
 4 files changed, 190 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cpumask_iter.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_cpumask_iter.c
tools/testing/selftests/bpf/config +1 -0
index c125c441abc7..9c42568ed376 100644
@@ -78,6 +78,7 @@ CONFIG_NF_CONNTRACK_MARK=y
 CONFIG_NF_DEFRAG_IPV4=y
 CONFIG_NF_DEFRAG_IPV6=y
 CONFIG_NF_NAT=y
+CONFIG_PSI=y
 CONFIG_RC_CORE=y
 CONFIG_SECURITY=y
 CONFIG_SECURITYFS=y
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +130 -0

130 lines not on this page · raw message

tools/testing/selftests/bpf/progs/cpumask_common.h +3 -0
index 0cd4aebb97cf..cdb9dc95e9d9 100644
@@ -55,6 +55,9 @@ void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym
 u32 bpf_cpumask_any_distribute(const struct cpumask *src) __ksym;
 u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1, const struct cpumask *src2) __ksym;
 u32 bpf_cpumask_weight(const struct cpumask *cpumask) __ksym;
+int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) __ksym;
+int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it) __ksym;
+void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) __ksym;
 
 void bpf_rcu_read_lock(void) __ksym;
 void bpf_rcu_read_unlock(void) __ksym;
tools/testing/selftests/bpf/progs/test_cpumask_iter.c new file +56 -0

56 lines not on this page · raw message

Signature · 1 line
2.39.1
#10 lore raw

Re: [PATCH v4 bpf-next 3/3] selftests/bpf: Add selftests for cpumask iter

Quoted 59 lines · Yafang Shao

Within the BPF program, we leverage the cgroup iterator to iterate through percpu runqueue data, specifically the 'nr_running' metric. Subsequently
we expose this data to userspace by means of a sequence file.

The CPU affinity for the cpumask is determined by the PID of a task:

- PID of the init task (PID 1)
We typically don't set CPU affinity for init task and thus we can iterate across all possible CPUs using the init task. However, in scenarios where you've set CPU affinity for the init task, you should set your current-task's cpu affinity to all possible CPUs and then proceed to iterate through all possible CPUs using the current task.
- PID of a task with defined CPU affinity
The aim here is to iterate through a specific cpumask. This scenario aligns with tasks residing within a cpuset cgroup.
- Invalid PID (e.g., PID -1)
No cpumask is available in this case.

The result as follows,

  #65/1    cpumask_iter/init_pid:OK
  #65/2    cpumask_iter/invalid_pid:OK
  #65/3    cpumask_iter/self_pid_one_cpu:OK
  #65/4    cpumask_iter/self_pid_multi_cpus:OK
  #65      cpumask_iter:OK

Summary: 1/4 PASSED, 0 SKIPPED, 0 FAILED

CONFIG_PSI=y is required for this testcase.

---
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/cpumask_iter.c   | 130 ++++++++++++++++++
 .../selftests/bpf/progs/cpumask_common.h      |   3 +
 .../selftests/bpf/progs/test_cpumask_iter.c   |  56 ++++++++
 4 files changed, 190 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cpumask_iter.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_cpumask_iter.c
tools/testing/selftests/bpf/config +1 -0
index c125c441abc7..9c42568ed376 100644
@@ -78,6 +78,7 @@ CONFIG_NF_CONNTRACK_MARK=y
 CONFIG_NF_DEFRAG_IPV4=y
 CONFIG_NF_DEFRAG_IPV6=y
 CONFIG_NF_NAT=y
+CONFIG_PSI=y
 CONFIG_RC_CORE=y
 CONFIG_SECURITY=y
 CONFIG_SECURITYFS=y
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +3 -0
new file mode 100644
index 000000000000..1db4efc57c5f
@@ -0,0 +1,130 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Yafang Shao <[email protected]> */
+

tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 13

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10 -0
+#define _GNU_SOURCE
+#include <sched.h>
+#include <stdio.h>
+#include <unistd.h>
+
+#include <test_progs.h>
+#include "cgroup_helpers.h"
+#include "test_cpumask_iter.skel.h"
+
+static void verify_percpu_data(struct bpf_link *link, int nr_cpu_exp, int nr_running_exp)

In general it seems wrong for this to be void. If we fail to verify something, why would we want to keep chugging along in the caller? That seems like it would just add a bunch of test failure noise for all the stuff that fails after.

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10 -0
+{
+	int iter_fd, len, item, nr_running, psi_running, nr_cpus;
+	char buf[128];
+	size_t left;
+	char *p;
+
+	iter_fd = bpf_iter_create(bpf_link__fd(link));
+	if (!ASSERT_GE(iter_fd, 0, "iter_fd"))
+		return;
+

tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 33

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10 -0
+	memset(buf, 0, sizeof(buf));
+	left = ARRAY_SIZE(buf);
+	p = buf;
+	while ((len = read(iter_fd, p, left)) > 0) {
+		p += len;
+		left -= len;
+	}
+
+	item = sscanf(buf, "nr_running %u nr_cpus %u psi_running %u\n",
+		      &nr_running, &nr_cpus, &psi_running);

I don't think there's any reason we should need to do string parsing like this. We can set all of these variables from BPF and then check them in the skeleton from user space.

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +5 -0
+	if (nr_cpu_exp == -1) {
+		ASSERT_EQ(item, -1, "seq_format");
+		goto out;
+	}
+

tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 48

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10 -0
+	ASSERT_EQ(item, 3, "seq_format");
+	ASSERT_GE(nr_running, nr_running_exp, "nr_running");
+	ASSERT_GE(psi_running, nr_running_exp, "psi_running");
+	ASSERT_EQ(nr_cpus, nr_cpu_exp, "nr_cpus");
+
+out:
+	close(iter_fd);
+}
+
+void test_cpumask_iter(void)

We should also have negative testcases that verify that we fail to load with bogus / unsafe inputs. See e.g. [0].

[0]: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/tree/tools/testing/selftests/bpf/progs/cpumask_failure.c

Quoted 30 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +30 -0
+{
+	DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	int nr_possible, cgrp_fd, pid, err, cnt, i;
+	struct test_cpumask_iter *skel;
+	union bpf_iter_link_info linfo;
+	int cpu_ids[] = {1, 3, 4, 5};
+	struct bpf_link *link;
+	cpu_set_t set;
+
+	skel = test_cpumask_iter__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "test_for_each_cpu__open_and_load"))
+		return;
+
+	if (setup_cgroup_environment())
+		goto destroy;
+
+	/* Utilize the cgroup iter */
+	cgrp_fd = get_root_cgroup();
+	if (!ASSERT_GE(cgrp_fd, 0, "create cgrp"))
+		goto cleanup;
+
+	memset(&linfo, 0, sizeof(linfo));
+	linfo.cgroup.cgroup_fd = cgrp_fd;
+	linfo.cgroup.order = BPF_CGROUP_ITER_SELF_ONLY;
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+
+	link = bpf_program__attach_iter(skel->progs.cpu_cgroup, &opts);
+	if (!ASSERT_OK_PTR(link, "attach_iter"))
+		goto close_fd;

tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 88

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10 -0
+
+	skel->bss->target_pid = 1;
+	/* In case init task is set CPU affinity */
+	err = sched_getaffinity(1, sizeof(set), &set);
+	if (!ASSERT_OK(err, "setaffinity"))
+		goto free_link;
+
+	cnt = CPU_COUNT(&set);
+	nr_possible = bpf_num_possible_cpus();
+	if (test__start_subtest("init_pid"))

Can you instead please do what we do in [1] and have each testcase be self contained and separately invoked? Doing things the way you have it in this patch has a few drawbacks:

1. The testcases are now all interdependent. If one fails we continue onto others even though we may or may not have any reason to believe that the system state is sane or what we expect. If the later testcases fail, what does it even mean? Or if they pass, is that expected? We should be setting up and tearing down whatever state we need between testcases to have confidence that the signal from each testcase is independent of whatever signal was in the previous one.

2. This is also confusing because we're doing a bunch of validation and testing even if test__start_subtest() returns false. It's unclear what the expecations are in such a case, and I think it's a lot more logical if you just don't do anything and skip the testcase.

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/tree/tools/testing/selftests/bpf/prog_tests/cpumask.c#n8

Quoted 81 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +42 -0
+		/* current task is running. */
+		verify_percpu_data(link, cnt, cnt == nr_possible ? 1 : 0);
+
+	skel->bss->target_pid = -1;
+	if (test__start_subtest("invalid_pid"))
+		verify_percpu_data(link, -1, -1);
+
+	pid = getpid();
+	skel->bss->target_pid = pid;
+	CPU_ZERO(&set);
+	CPU_SET(0, &set);
+	err = sched_setaffinity(pid, sizeof(set), &set);
+	if (!ASSERT_OK(err, "setaffinity"))
+		goto free_link;
+
+	if (test__start_subtest("self_pid_one_cpu"))
+		verify_percpu_data(link, 1, 1);
+
+	/* Assume there are at least 8 CPUs on the testbed */
+	if (nr_possible < 8)
+		goto free_link;
+
+	CPU_ZERO(&set);
+	/* Set the CPU affinitiy: 1,3-5 */
+	for (i = 0; i < ARRAY_SIZE(cpu_ids); i++)
+		CPU_SET(cpu_ids[i], &set);
+	err = sched_setaffinity(pid, sizeof(set), &set);
+	if (!ASSERT_OK(err, "setaffinity"))
+		goto free_link;
+
+	if (test__start_subtest("self_pid_multi_cpus"))
+		verify_percpu_data(link, ARRAY_SIZE(cpu_ids), 1);
+
+free_link:
+	bpf_link__destroy(link);
+close_fd:
+	close(cgrp_fd);
+cleanup:
+	cleanup_cgroup_environment();
+destroy:
+	test_cpumask_iter__destroy(skel);
+}
tools/testing/selftests/bpf/progs/cpumask_common.h +3 -0
index 0cd4aebb97cf..cdb9dc95e9d9 100644
@@ -55,6 +55,9 @@ void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym
 u32 bpf_cpumask_any_distribute(const struct cpumask *src) __ksym;
 u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1, const struct cpumask *src2) __ksym;
 u32 bpf_cpumask_weight(const struct cpumask *cpumask) __ksym;
+int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) __ksym;
+int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it) __ksym;
+void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) __ksym;
 
 void bpf_rcu_read_lock(void) __ksym;
 void bpf_rcu_read_unlock(void) __ksym;
tools/testing/selftests/bpf/progs/test_cpumask_iter.c new file +19 -0
new file mode 100644
index 000000000000..cb8b8359516b
@@ -0,0 +1,56 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2024 Yafang Shao <[email protected]> */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+#include "task_kfunc_common.h"
+#include "cpumask_common.h"
+
+extern const struct psi_group_cpu system_group_pcpu __ksym __weak;
+extern const struct rq runqueues __ksym __weak;
+
+int target_pid;
+
+SEC("iter.s/cgroup")
+int BPF_PROG(cpu_cgroup, struct bpf_iter_meta *meta, struct cgroup *cgrp)
+{
+	u32 nr_running = 0, psi_nr_running = 0, nr_cpus = 0;

tools/testing/selftests/bpf/progs/test_cpumask_iter.c line 29

Quoted 10 lines
tools/testing/selftests/bpf/progs/test_cpumask_iter.c new file +10 -0
+	struct psi_group_cpu *groupc;
+	struct task_struct *p;
+	struct rq *rq;
+	int *cpu;
+
+	/* epilogue */
+	if (cgrp == NULL)
+		return 0;
+
+	bpf_rcu_read_lock();

Why is this required? p->cpus_ptr should be trusted given that @p is trusted, no? Does this fail to load if you don't have all of this in an RCU read region?

tools/testing/selftests/bpf/progs/test_cpumask_iter.c line 39

Quoted 10 lines
tools/testing/selftests/bpf/progs/test_cpumask_iter.c new file +10 -0
+	p = bpf_task_from_pid(target_pid);
+	if (!p) {
+		bpf_rcu_read_unlock();
+		return 1;
+	}
+
+	bpf_for_each(cpumask, cpu, p->cpus_ptr) {
+		rq = (struct rq *)bpf_per_cpu_ptr(&runqueues, *cpu);
+		if (!rq)
+			continue;

If this happens we should set an error code and check it in user space.

tools/testing/selftests/bpf/progs/test_cpumask_iter.c line 45

Quoted 6 lines
tools/testing/selftests/bpf/progs/test_cpumask_iter.c new file +6 -0
+		nr_running += rq->nr_running;
+		nr_cpus += 1;
+
+		groupc = (struct psi_group_cpu *)bpf_per_cpu_ptr(&system_group_pcpu, *cpu);
+		if (!groupc)
+			continue;

Same here.

tools/testing/selftests/bpf/progs/test_cpumask_iter.c line 56

Quoted 11 lines
tools/testing/selftests/bpf/progs/test_cpumask_iter.c new file +11 -0
+		psi_nr_running += groupc->tasks[NR_RUNNING];
+	}
+	BPF_SEQ_PRINTF(meta->seq, "nr_running %u nr_cpus %u psi_running %u\n",
+		       nr_running, nr_cpus, psi_nr_running);
+
+	bpf_task_release(p);
+	bpf_rcu_read_unlock();
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";

Can you please move this to the top of the file?

Quoted 4 lines
Signature · 1 line
2.39.1
#11 lore raw

Re: [PATCH v4 bpf-next 3/3] selftests/bpf: Add selftests for cpumask iter

Quoted 68 lines · David Vernet
Quoted 66 lines · Yafang Shao

Within the BPF program, we leverage the cgroup iterator to iterate through percpu runqueue data, specifically the 'nr_running' metric. Subsequently
we expose this data to userspace by means of a sequence file.

The CPU affinity for the cpumask is determined by the PID of a task:

- PID of the init task (PID 1)
We typically don't set CPU affinity for init task and thus we can iterate across all possible CPUs using the init task. However, in scenarios where you've set CPU affinity for the init task, you should set your current-task's cpu affinity to all possible CPUs and then proceed to iterate through all possible CPUs using the current task.
- PID of a task with defined CPU affinity
The aim here is to iterate through a specific cpumask. This scenario aligns with tasks residing within a cpuset cgroup.
- Invalid PID (e.g., PID -1)
No cpumask is available in this case.

The result as follows,

  #65/1    cpumask_iter/init_pid:OK
  #65/2    cpumask_iter/invalid_pid:OK
  #65/3    cpumask_iter/self_pid_one_cpu:OK
  #65/4    cpumask_iter/self_pid_multi_cpus:OK
  #65      cpumask_iter:OK

Summary: 1/4 PASSED, 0 SKIPPED, 0 FAILED

CONFIG_PSI=y is required for this testcase.

---
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/cpumask_iter.c   | 130 ++++++++++++++++++
 .../selftests/bpf/progs/cpumask_common.h      |   3 +
 .../selftests/bpf/progs/test_cpumask_iter.c   |  56 ++++++++
 4 files changed, 190 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cpumask_iter.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_cpumask_iter.c
tools/testing/selftests/bpf/config +1 -0
index c125c441abc7..9c42568ed376 100644
@@ -78,6 +78,7 @@ CONFIG_NF_CONNTRACK_MARK=y
 CONFIG_NF_DEFRAG_IPV4=y
 CONFIG_NF_DEFRAG_IPV6=y
 CONFIG_NF_NAT=y
+CONFIG_PSI=y
 CONFIG_RC_CORE=y
 CONFIG_SECURITY=y
 CONFIG_SECURITYFS=y
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10 -0
new file mode 100644
index 000000000000..1db4efc57c5f
@@ -0,0 +1,130 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Yafang Shao <[email protected]> */
+
+#define _GNU_SOURCE
+#include <sched.h>
+#include <stdio.h>
+#include <unistd.h>
+
+#include <test_progs.h>
+#include "cgroup_helpers.h"
Quoted 8 lines

tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 13

Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +3 -0
+#include "test_cpumask_iter.skel.h"
+
+static void verify_percpu_data(struct bpf_link *link, int nr_cpu_exp, int nr_running_exp)

In general it seems wrong for this to be void. If we fail to verify something, why would we want to keep chugging along in the caller? That seems like it would just add a bunch of test failure noise for all the stuff that fails after.

makes sense.

Quoted 17 lines
Quoted 17 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +16 -0
+{
+     int iter_fd, len, item, nr_running, psi_running, nr_cpus;
+     char buf[128];
+     size_t left;
+     char *p;
+
+     iter_fd = bpf_iter_create(bpf_link__fd(link));
+     if (!ASSERT_GE(iter_fd, 0, "iter_fd"))
+             return;
+
+     memset(buf, 0, sizeof(buf));
+     left = ARRAY_SIZE(buf);
+     p = buf;
+     while ((len = read(iter_fd, p, left)) > 0) {
+             p += len;
+             left -= len;
Quoted 8 lines

tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 33

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +4 -0
+     }
+
+     item = sscanf(buf, "nr_running %u nr_cpus %u psi_running %u\n",
+                   &nr_running, &nr_cpus, &psi_running);

I don't think there's any reason we should need to do string parsing like this. We can set all of these variables from BPF and then check them in the skeleton from user space.

I agree with your point about the simplification achieved through skeleton variables. However, it's worth noting that a seq file in this context serves to demonstrate how we expose the data through a seq file.

Quoted 11 lines
Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10 -0
+     if (nr_cpu_exp == -1) {
+             ASSERT_EQ(item, -1, "seq_format");
+             goto out;
+     }
+
+     ASSERT_EQ(item, 3, "seq_format");
+     ASSERT_GE(nr_running, nr_running_exp, "nr_running");
+     ASSERT_GE(psi_running, nr_running_exp, "psi_running");
+     ASSERT_EQ(nr_cpus, nr_cpu_exp, "nr_cpus");
+
Quoted 8 lines

tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 48

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +5 -0
+out:
+     close(iter_fd);
+}
+
+void test_cpumask_iter(void)

We should also have negative testcases that verify that we fail to load with bogus / unsafe inputs. See e.g. [0].

will add it.

Quoted 56 lines

[0]: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/tree/tools/testing/selftests/bpf/progs/cpumask_failure.c

Quoted 30 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +30 -0
+{
+     DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+     int nr_possible, cgrp_fd, pid, err, cnt, i;
+     struct test_cpumask_iter *skel;
+     union bpf_iter_link_info linfo;
+     int cpu_ids[] = {1, 3, 4, 5};
+     struct bpf_link *link;
+     cpu_set_t set;
+
+     skel = test_cpumask_iter__open_and_load();
+     if (!ASSERT_OK_PTR(skel, "test_for_each_cpu__open_and_load"))
+             return;
+
+     if (setup_cgroup_environment())
+             goto destroy;
+
+     /* Utilize the cgroup iter */
+     cgrp_fd = get_root_cgroup();
+     if (!ASSERT_GE(cgrp_fd, 0, "create cgrp"))
+             goto cleanup;
+
+     memset(&linfo, 0, sizeof(linfo));
+     linfo.cgroup.cgroup_fd = cgrp_fd;
+     linfo.cgroup.order = BPF_CGROUP_ITER_SELF_ONLY;
+     opts.link_info = &linfo;
+     opts.link_info_len = sizeof(linfo);
+
+     link = bpf_program__attach_iter(skel->progs.cpu_cgroup, &opts);
+     if (!ASSERT_OK_PTR(link, "attach_iter"))
+             goto close_fd;

tools/testing/selftests/bpf/prog_tests/cpumask_iter.c line 88

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +10 -0
+
+     skel->bss->target_pid = 1;
+     /* In case init task is set CPU affinity */
+     err = sched_getaffinity(1, sizeof(set), &set);
+     if (!ASSERT_OK(err, "setaffinity"))
+             goto free_link;
+
+     cnt = CPU_COUNT(&set);
+     nr_possible = bpf_num_possible_cpus();
+     if (test__start_subtest("init_pid"))

Can you instead please do what we do in [1] and have each testcase be self contained and separately invoked? Doing things the way you have it in this patch has a few drawbacks:

1. The testcases are now all interdependent. If one fails we continue onto others even though we may or may not have any reason to believe that the system state is sane or what we expect. If the later testcases fail, what does it even mean? Or if they pass, is that expected? We should be setting up and tearing down whatever state we need between testcases to have confidence that the signal from each testcase is independent of whatever signal was in the previous one.

Quoted 6 lines

2. This is also confusing because we're doing a bunch of validation and testing even if test__start_subtest() returns false. It's unclear what the expecations are in such a case, and I think it's a lot more logical if you just don't do anything and skip the testcase.

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/tree/tools/testing/selftests/bpf/prog_tests/cpumask.c#n8

Will check it. Appreciate your insights. Thank you.

Quoted 88 lines
Quoted 88 lines
tools/testing/selftests/bpf/prog_tests/cpumask_iter.c new file +42 -0
+             /* current task is running. */
+             verify_percpu_data(link, cnt, cnt == nr_possible ? 1 : 0);
+
+     skel->bss->target_pid = -1;
+     if (test__start_subtest("invalid_pid"))
+             verify_percpu_data(link, -1, -1);
+
+     pid = getpid();
+     skel->bss->target_pid = pid;
+     CPU_ZERO(&set);
+     CPU_SET(0, &set);
+     err = sched_setaffinity(pid, sizeof(set), &set);
+     if (!ASSERT_OK(err, "setaffinity"))
+             goto free_link;
+
+     if (test__start_subtest("self_pid_one_cpu"))
+             verify_percpu_data(link, 1, 1);
+
+     /* Assume there are at least 8 CPUs on the testbed */
+     if (nr_possible < 8)
+             goto free_link;
+
+     CPU_ZERO(&set);
+     /* Set the CPU affinitiy: 1,3-5 */
+     for (i = 0; i < ARRAY_SIZE(cpu_ids); i++)
+             CPU_SET(cpu_ids[i], &set);
+     err = sched_setaffinity(pid, sizeof(set), &set);
+     if (!ASSERT_OK(err, "setaffinity"))
+             goto free_link;
+
+     if (test__start_subtest("self_pid_multi_cpus"))
+             verify_percpu_data(link, ARRAY_SIZE(cpu_ids), 1);
+
+free_link:
+     bpf_link__destroy(link);
+close_fd:
+     close(cgrp_fd);
+cleanup:
+     cleanup_cgroup_environment();
+destroy:
+     test_cpumask_iter__destroy(skel);
+}
tools/testing/selftests/bpf/progs/cpumask_common.h +3 -0
index 0cd4aebb97cf..cdb9dc95e9d9 100644
@@ -55,6 +55,9 @@ void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym
 u32 bpf_cpumask_any_distribute(const struct cpumask *src) __ksym;
 u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1, const struct cpumask *src2) __ksym;
 u32 bpf_cpumask_weight(const struct cpumask *cpumask) __ksym;
+int bpf_iter_cpumask_new(struct bpf_iter_cpumask *it, const struct cpumask *mask) __ksym;
+int *bpf_iter_cpumask_next(struct bpf_iter_cpumask *it) __ksym;
+void bpf_iter_cpumask_destroy(struct bpf_iter_cpumask *it) __ksym;
 
 void bpf_rcu_read_lock(void) __ksym;
 void bpf_rcu_read_unlock(void) __ksym;
tools/testing/selftests/bpf/progs/test_cpumask_iter.c new file +25 -0
new file mode 100644
index 000000000000..cb8b8359516b
@@ -0,0 +1,56 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2024 Yafang Shao <[email protected]> */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+#include "task_kfunc_common.h"
+#include "cpumask_common.h"
+
+extern const struct psi_group_cpu system_group_pcpu __ksym __weak;
+extern const struct rq runqueues __ksym __weak;
+
+int target_pid;
+
+SEC("iter.s/cgroup")
+int BPF_PROG(cpu_cgroup, struct bpf_iter_meta *meta, struct cgroup *cgrp)
+{
+     u32 nr_running = 0, psi_nr_running = 0, nr_cpus = 0;
+     struct psi_group_cpu *groupc;
+     struct task_struct *p;
+     struct rq *rq;
+     int *cpu;
+
+     /* epilogue */
Quoted 8 lines

tools/testing/selftests/bpf/progs/test_cpumask_iter.c line 29

Quoted 4 lines
tools/testing/selftests/bpf/progs/test_cpumask_iter.c new file +4 -0
+     if (cgrp == NULL)
+             return 0;
+
+     bpf_rcu_read_lock();

Why is this required? p->cpus_ptr should be trusted given that @p is trusted, no? Does this fail to load if you don't have all of this in an RCU read region?

It is a sleepable prog iter.s, so we must explicitly add a rcu lock, otherwise the load will fail.

Quoted 12 lines

tools/testing/selftests/bpf/progs/test_cpumask_iter.c line 39

Quoted 11 lines
tools/testing/selftests/bpf/progs/test_cpumask_iter.c new file +10 -0
+     p = bpf_task_from_pid(target_pid);
+     if (!p) {
+             bpf_rcu_read_unlock();
+             return 1;
+     }
+
+     bpf_for_each(cpumask, cpu, p->cpus_ptr) {
+             rq = (struct rq *)bpf_per_cpu_ptr(&runqueues, *cpu);
+             if (!rq)
+                     continue;
Quoted 1 line

If this happens we should set an error code and check it in user space.

sure, will do it.

Quoted 9 lines

tools/testing/selftests/bpf/progs/test_cpumask_iter.c line 45

Quoted 7 lines
tools/testing/selftests/bpf/progs/test_cpumask_iter.c new file +6 -0
+             nr_running += rq->nr_running;
+             nr_cpus += 1;
+
+             groupc = (struct psi_group_cpu *)bpf_per_cpu_ptr(&system_group_pcpu, *cpu);
+             if (!groupc)
+                     continue;

Same here.

will do it.

Quoted 14 lines

tools/testing/selftests/bpf/progs/test_cpumask_iter.c line 56

Quoted 12 lines
tools/testing/selftests/bpf/progs/test_cpumask_iter.c new file +11 -0
+             psi_nr_running += groupc->tasks[NR_RUNNING];
+     }
+     BPF_SEQ_PRINTF(meta->seq, "nr_running %u nr_cpus %u psi_running %u\n",
+                    nr_running, nr_cpus, psi_nr_running);
+
+     bpf_task_release(p);
+     bpf_rcu_read_unlock();
+     return 0;
+}
+
+char _license[] SEC("license") = "GPL";

Can you please move this to the top of the file?

will do it.

Signature · 2 lines
Regards
Yafang