Skip to contentSkip to conversation

Introduce BPF_F_CPU flag for percpu maps

PATCH Superseded Leon Hwang 2 participants 17 messages Aug 21, 2025 – Aug 26, 2025

Subsystems (5)
Kselftest BPF BPF selftests BPF core +1 more subsystems

v3 of 13 Superseded Latest is v13 →

lore
Share & tools
lore

Leon Hwang

author
#0 lore raw

v3 cover Introduce BPF_F_CPU flag for percpu maps

This patch set introduces the BPF_F_CPU and BPF_F_ALL_CPUS flags for percpu maps, as the requirement of BPF_F_ALL_CPUS flag for percpu_array maps was discussed in the thread of
"[PATCH bpf-next v3 0/4] bpf: Introduce global percpu data"[1].

The goal of BPF_ALL_CPUS flag is to reduce data caching overhead in light skeletons by allowing a single value to be reused across all CPUs. This avoids the M:N problem where M cached values are used to update a map on N CPUs kernel.

The BPF_F_CPU flag is accompanied by *flags*-embedded cpu info, which specifies the target CPUs for the operation:

* For lookup operations: the flag field alongside cpu info enable querying a value on the specified CPU.
* For update operations: the flag field alongside cpu info enable updating value for specified CPU.

Links:
[1] https://lore.kernel.org/bpf/[email protected]/

Changes:
v2 -> v3:
* Address comments from Alexei:
* Use BPF_F_ALL_CPUS instead of BPF_ALL_CPUS magic.
* Introduce these two cpu flags for all percpu maps.
* Address comments from Jiri:
* Reduce some unnecessary u32 cast.
* Refactor more generic map flags check function.
* A code style issue.
v2 link: https://lore.kernel.org/bpf/[email protected]/

v1 -> v2:
* Address comments from Andrii:
* Embed cpu info as high 32 bits of *flags* totally.
* Use ERANGE instead of E2BIG.
* Few format issues.

RFC v2 -> v1:
* Address comments from Andrii:
* Use '&=' and '|='.
* Replace 'reuse_value' with simpler and less duplication code.
* Replace 'ASSERT_FALSE' with two 'ASSERT_OK_PTR's in self test.

RFC v1 -> RFC v2:
* Address comments from Andrii:
* Embed cpu to flags on kernel side.
* Change BPF_ALL_CPU macro to BPF_ALL_CPUS enum.
* Copy/update element within RCU protection.
* Update bpf_map_value_size() including BPF_F_CPU case.
* Use zero as default value to get cpu option.
* Update documents of APIs to be generic.
* Add size_t:0 to opts definitions.
* Update validate_map_op() including BPF_F_CPU case.
* Use LIBBPF_OPTS instead of DECLARE_LIBBPF_OPTS.

Leon Hwang (6):
bpf: Introduce internal check_map_flags helper function
bpf: Introduce BPF_F_CPU flag for percpu_array maps
bpf: Introduce BPF_F_CPU flag for percpu_hash and lru_percpu_hash maps
bpf: Introduce BPF_F_CPU flag for percpu_cgroup_storage maps
libbpf: Support BPF_F_CPU for percpu maps
selftests/bpf: Add cases to test BPF_F_CPU flag

 include/linux/bpf-cgroup.h                    |   5 +-
 include/linux/bpf.h                           |  58 ++++-
 include/uapi/linux/bpf.h                      |   2 +
 kernel/bpf/arraymap.c                         |  51 ++--
 kernel/bpf/hashtab.c                          | 111 ++++++---
 kernel/bpf/local_storage.c                    |  47 +++-
 kernel/bpf/syscall.c                          |  60 ++---
 tools/include/uapi/linux/bpf.h                |   2 +
 tools/lib/bpf/bpf.h                           |   8 +
 tools/lib/bpf/libbpf.c                        |  25 +-
 tools/lib/bpf/libbpf.h                        |  21 +-
 .../selftests/bpf/prog_tests/percpu_alloc.c   | 224 ++++++++++++++++++
 .../selftests/bpf/progs/percpu_alloc_array.c  |  32 +++
 13 files changed, 527 insertions(+), 119 deletions(-)

--
2.50.1

author
#1 lore raw

PATCH v3 1/6 bpf: Introduce internal check_map_flags helper function

It is to unify map flags checking for lookup, update, lookup_batch and update_batch.

Therefore, it will be convenient to check BPF_F_CPU flag in this helper function for them in next patch.

---
 kernel/bpf/syscall.c | 45 ++++++++++++++++++++++----------------------
 1 file changed, 22 insertions(+), 23 deletions(-)
kernel/bpf/syscall.c +22 -23
index 0fbfa8532c392..19f7f5de5e7dc 100644
@@ -1654,6 +1654,17 @@ static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size)
 	return NULL;
 }
 
+static int check_map_flags(struct bpf_map *map, u64 flags, bool check_flag)
+{
+	if (check_flag && (flags & ~BPF_F_LOCK))
+		return -EINVAL;
+
+	if ((flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))
+		return -EINVAL;
+
+	return 0;
+}
+
 /* last field in 'union bpf_attr' used by this command */
 #define BPF_MAP_LOOKUP_ELEM_LAST_FIELD flags
 
@@ -1669,9 +1680,6 @@ static int map_lookup_elem(union bpf_attr *attr)
 	if (CHECK_ATTR(BPF_MAP_LOOKUP_ELEM))
 		return -EINVAL;
 
-	if (attr->flags & ~BPF_F_LOCK)
-		return -EINVAL;
-
 	CLASS(fd, f)(attr->map_fd);
 	map = __bpf_map_get(f);
 	if (IS_ERR(map))
@@ -1679,9 +1687,9 @@ static int map_lookup_elem(union bpf_attr *attr)
 	if (!(map_get_sys_perms(map, f) & FMODE_CAN_READ))
 		return -EPERM;
 
-	if ((attr->flags & BPF_F_LOCK) &&
-	    !btf_record_has_field(map->record, BPF_SPIN_LOCK))
-		return -EINVAL;
+	err = check_map_flags(map, attr->flags, true);
+	if (err)
+		return err;
 
 	key = __bpf_copy_key(ukey, map->key_size);
 	if (IS_ERR(key))
@@ -1744,11 +1752,9 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
 		goto err_put;
 	}
 
-	if ((attr->flags & BPF_F_LOCK) &&
-	    !btf_record_has_field(map->record, BPF_SPIN_LOCK)) {
-		err = -EINVAL;
+	err = check_map_flags(map, attr->flags, false);
+	if (err)
 		goto err_put;
-	}
 
 	key = ___bpf_copy_key(ukey, map->key_size);
 	if (IS_ERR(key)) {
@@ -1952,13 +1958,9 @@ int generic_map_update_batch(struct bpf_map *map, struct file *map_file,
 	void *key, *value;
 	int err = 0;
 
-	if (attr->batch.elem_flags & ~BPF_F_LOCK)
-		return -EINVAL;
-
-	if ((attr->batch.elem_flags & BPF_F_LOCK) &&
-	    !btf_record_has_field(map->record, BPF_SPIN_LOCK)) {
-		return -EINVAL;
-	}
+	err = check_map_flags(map, attr->batch.elem_flags, true);
+	if (err)
+		return err;
 
 	value_size = bpf_map_value_size(map);
 
@@ -2015,12 +2017,9 @@ int generic_map_lookup_batch(struct bpf_map *map,
 	u32 value_size, cp, max_count;
 	int err;
 
-	if (attr->batch.elem_flags & ~BPF_F_LOCK)
-		return -EINVAL;
-
-	if ((attr->batch.elem_flags & BPF_F_LOCK) &&
-	    !btf_record_has_field(map->record, BPF_SPIN_LOCK))
-		return -EINVAL;
+	err = check_map_flags(map, attr->batch.elem_flags, true);
+	if (err)
+		return err;
 
 	value_size = bpf_map_value_size(map);
 
Signature · 1 line
2.50.1
author
#5 lore raw

PATCH v3 2/6 bpf: Introduce BPF_F_CPU flag for percpu_array maps

In reply to Leon Hwang

Introduce support for the BPF_F_ALL_CPUS flag in percpu_array maps to allow updating values for all CPUs with a single value.

Introduce support for the BPF_F_CPU flag in percpu_array maps to allow updating value for specified CPU.

This enhancement enables:

* Efficient update values across all CPUs with a single value when BPF_F_ALL_CPUS is set for update_elem and update_batch APIs.
* Targeted update or lookup for a specified CPU when BPF_F_CPU is set.

The BPF_F_CPU flag is passed via:

* map_flags of lookup_elem and update_elem APIs along with embedded cpu field.
* elem_flags of lookup_batch and update_batch APIs along with embedded cpu field.

---
 include/linux/bpf.h            |  3 +-
 include/uapi/linux/bpf.h       |  2 ++
 kernel/bpf/arraymap.c          | 56 ++++++++++++++++++++++++++--------
 kernel/bpf/syscall.c           | 27 ++++++++++------
 tools/include/uapi/linux/bpf.h |  2 ++
 5 files changed, 67 insertions(+), 23 deletions(-)
include/linux/bpf.h +2 -1
index 8f6e87f0f3a89..b2191b1e455a6 100644
@@ -2697,7 +2697,8 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env,
 				   struct bpf_func_state *callee);
 
 int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value);
-int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value);
+int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value,
+			  u64 flags);
 int bpf_percpu_hash_update(struct bpf_map *map, void *key, void *value,
 			   u64 flags);
 int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
include/uapi/linux/bpf.h +2 -0
index 233de8677382e..be1fdc5042744 100644
@@ -1372,6 +1372,8 @@ enum {
 	BPF_NOEXIST	= 1, /* create new element if it didn't exist */
 	BPF_EXIST	= 2, /* update existing element */
 	BPF_F_LOCK	= 4, /* spin_lock-ed map_lookup/map_update */
+	BPF_F_CPU	= 8, /* cpu flag for percpu maps, upper 32-bit of flags is a cpu number */
+	BPF_F_ALL_CPUS	= 16, /* update value across all CPUs for percpu maps */
 };
 
 /* flags for BPF_MAP_CREATE command */
kernel/bpf/arraymap.c +43 -13

94 lines not on this page · raw message

kernel/bpf/syscall.c +18 -9
index 19f7f5de5e7dc..6251ac9bc7e42 100644
@@ -131,9 +131,11 @@ bool bpf_map_write_active(const struct bpf_map *map)
 	return atomic64_read(&map->writecnt) != 0;
 }
 
-static u32 bpf_map_value_size(const struct bpf_map *map)
+static u32 bpf_map_value_size(const struct bpf_map *map, u64 flags)
 {
-	if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
+	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY && (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)))
+		return round_up(map->value_size, 8);
+	else if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
 	    map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
 	    map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY ||
 	    map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE)
@@ -314,7 +316,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value,
 	    map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) {
 		err = bpf_percpu_hash_copy(map, key, value);
 	} else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
-		err = bpf_percpu_array_copy(map, key, value);
+		err = bpf_percpu_array_copy(map, key, value, flags);
 	} else if (map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) {
 		err = bpf_percpu_cgroup_storage_copy(map, key, value);
 	} else if (map->map_type == BPF_MAP_TYPE_STACK_TRACE) {
@@ -1656,12 +1658,19 @@ static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size)
 
 static int check_map_flags(struct bpf_map *map, u64 flags, bool check_flag)
 {
-	if (check_flag && (flags & ~BPF_F_LOCK))
+	if (check_flag && ((u32)flags & ~(BPF_F_LOCK | BPF_F_CPU | BPF_F_ALL_CPUS)))
 		return -EINVAL;
 
 	if ((flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))
 		return -EINVAL;
 
+	if (!(flags & BPF_F_CPU) && flags >> 32)
+		return -EINVAL;
+
+	if ((flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) &&
+		map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY)
+		return -EINVAL;
+
 	return 0;
 }
 
@@ -1695,7 +1704,7 @@ static int map_lookup_elem(union bpf_attr *attr)
 	if (IS_ERR(key))
 		return PTR_ERR(key);
 
-	value_size = bpf_map_value_size(map);
+	value_size = bpf_map_value_size(map, attr->flags);
 
 	err = -ENOMEM;
 	value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN);
@@ -1762,7 +1771,7 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
 		goto err_put;
 	}
 
-	value_size = bpf_map_value_size(map);
+	value_size = bpf_map_value_size(map, attr->flags);
 	value = kvmemdup_bpfptr(uvalue, value_size);
 	if (IS_ERR(value)) {
 		err = PTR_ERR(value);
@@ -1962,7 +1971,7 @@ int generic_map_update_batch(struct bpf_map *map, struct file *map_file,
 	if (err)
 		return err;
 
-	value_size = bpf_map_value_size(map);
+	value_size = bpf_map_value_size(map, attr->batch.elem_flags);
 
 	max_count = attr->batch.count;
 	if (!max_count)
@@ -2021,7 +2030,7 @@ int generic_map_lookup_batch(struct bpf_map *map,
 	if (err)
 		return err;
 
-	value_size = bpf_map_value_size(map);
+	value_size = bpf_map_value_size(map, attr->batch.elem_flags);
 
 	max_count = attr->batch.count;
 	if (!max_count)
@@ -2143,7 +2152,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr)
 		goto err_put;
 	}
 
-	value_size = bpf_map_value_size(map);
+	value_size = bpf_map_value_size(map, 0);
 
 	err = -ENOMEM;
 	value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN);
tools/include/uapi/linux/bpf.h +2 -0
index 233de8677382e..be1fdc5042744 100644
@@ -1372,6 +1372,8 @@ enum {
 	BPF_NOEXIST	= 1, /* create new element if it didn't exist */
 	BPF_EXIST	= 2, /* update existing element */
 	BPF_F_LOCK	= 4, /* spin_lock-ed map_lookup/map_update */
+	BPF_F_CPU	= 8, /* cpu flag for percpu maps, upper 32-bit of flags is a cpu number */
+	BPF_F_ALL_CPUS	= 16, /* update value across all CPUs for percpu maps */
 };
 
 /* flags for BPF_MAP_CREATE command */
Signature · 1 line
2.50.1
author
#8 lore raw

PATCH v3 3/6 bpf: Introduce BPF_F_CPU flag for percpu_hash and lru_percpu_hash maps

In reply to Leon Hwang

Introduce BPF_F_ALL_CPUS flag support for percpu_hash and lru_percpu_hash maps to allow updating values for all CPUs with a single value.

Introduce BPF_F_CPU flag support for percpu_hash and lru_percpu_hash maps to allow updating value for specified CPU.

This enhancement enables:

* Efficient update values across all CPUs with a single value when BPF_F_ALL_CPUS is set for update_elem and update_batch APIs.
* Targeted update or lookup for a specified CPU when BPF_F_CPU is set.

The BPF_F_CPU flag is passed via:

* map_flags of lookup_elem and update_elem APIs along with embedded cpu field.
* elem_flags of lookup_batch and update_batch APIs along with embedded cpu field.

---
 include/linux/bpf.h   |  54 +++++++++++++++++++-
 kernel/bpf/arraymap.c |  29 ++++-------
 kernel/bpf/hashtab.c  | 111 +++++++++++++++++++++++++++++-------------
 kernel/bpf/syscall.c  |  30 +++---------
 4 files changed, 147 insertions(+), 77 deletions(-)
include/linux/bpf.h +53 -1
index b2191b1e455a6..dc715eef9cbf4 100644
@@ -2696,7 +2696,7 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env,
 				   struct bpf_func_state *caller,
 				   struct bpf_func_state *callee);
 
-int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value);
+int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value, u64 flags);
 int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value,
 			  u64 flags);
 int bpf_percpu_hash_update(struct bpf_map *map, void *key, void *value,
@@ -3710,4 +3710,56 @@ int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char *
 			   const char **linep, int *nump);
 struct bpf_prog *bpf_prog_find_from_stack(void);
 
+static inline int bpf_map_check_cpu_flags(u64 flags, bool check_all_cpus)
+{
+	const u64 cpu_flags = BPF_F_CPU | BPF_F_ALL_CPUS;
+	u32 cpu;
+
+	if (check_all_cpus) {
+		if (unlikely((u32)flags > BPF_F_ALL_CPUS))
+			/* unknown flags */
+			return -EINVAL;
+		if (unlikely((flags & cpu_flags) == cpu_flags))
+			return -EINVAL;
+	} else {
+		if (unlikely((u32)flags & ~BPF_F_CPU))
+			return -EINVAL;
+	}
+
+	cpu = flags >> 32;
+	if (unlikely((flags & BPF_F_CPU) && cpu >= num_possible_cpus()))
+		return -ERANGE;
+
+	return 0;
+}
+
+static inline bool bpf_map_support_cpu_flags(enum bpf_map_type map_type)
+{
+	switch (map_type) {
+	case BPF_MAP_TYPE_PERCPU_ARRAY:
+	case BPF_MAP_TYPE_PERCPU_HASH:
+	case BPF_MAP_TYPE_LRU_PERCPU_HASH:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static inline int bpf_map_check_flags(struct bpf_map *map, u64 flags, bool check_flag)
+{
+	if (check_flag && ((u32)flags & ~(BPF_F_LOCK | BPF_F_CPU | BPF_F_ALL_CPUS)))
+		return -EINVAL;
+
+	if ((flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))
+		return -EINVAL;
+
+	if (!(flags & BPF_F_CPU) && flags >> 32)
+		return -EINVAL;
+
+	if ((flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) && !bpf_map_support_cpu_flags(map->map_type))
+		return -EINVAL;
+
+	return 0;
+}
+
 #endif /* _LINUX_BPF_H */
kernel/bpf/arraymap.c +10 -19
index 1efa730105e24..f7646bcabb3c8 100644
@@ -300,18 +300,15 @@ int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value, u64 flags
 	struct bpf_array *array = container_of(map, struct bpf_array, map);
 	u32 index = *(u32 *)key;
 	void __percpu *pptr;
+	int off = 0, err;
 	u32 size, cpu;
-	int off = 0;
 
 	if (unlikely(index >= array->map.max_entries))
 		return -ENOENT;
 
-	if (unlikely((u32)flags & ~BPF_F_CPU))
-		return -EINVAL;
-
-	cpu = flags >> 32;
-	if (unlikely((flags & BPF_F_CPU) && cpu >= num_possible_cpus()))
-		return -ERANGE;
+	err = bpf_map_check_cpu_flags(flags, false);
+	if (unlikely(err))
+		return err;
 
 	/* per_cpu areas are zero-filled and bpf programs can only
 	 * access 'value_size' of them, so copying rounded areas
@@ -321,6 +318,7 @@ int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value, u64 flags
 	rcu_read_lock();
 	pptr = array->pptrs[index & array->index_mask];
 	if (flags & BPF_F_CPU) {
+		cpu = flags >> 32;
 		copy_map_value_long(map, value, per_cpu_ptr(pptr, cpu));
 		check_and_init_map_value(map, value);
 	} else {
@@ -397,22 +395,14 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
 			    u64 map_flags)
 {
 	struct bpf_array *array = container_of(map, struct bpf_array, map);
-	const u64 cpu_flags = BPF_F_CPU | BPF_F_ALL_CPUS;
 	u32 index = *(u32 *)key;
 	void __percpu *pptr;
+	int off = 0, err;
 	u32 size, cpu;
-	int off = 0;
-
-	if (unlikely((u32)map_flags > BPF_F_ALL_CPUS))
-		/* unknown flags */
-		return -EINVAL;
-	if (unlikely((map_flags & cpu_flags) == cpu_flags))
-		return -EINVAL;
 
-	cpu = map_flags >> 32;
-	if (unlikely((map_flags & BPF_F_CPU) && cpu >= num_possible_cpus()))
-		/* invalid cpu */
-		return -ERANGE;
+	err = bpf_map_check_cpu_flags(map_flags, true);
+	if (unlikely(err))
+		return err;
 
 	if (unlikely(index >= array->map.max_entries))
 		/* all elements were pre-allocated, cannot insert a new one */
@@ -432,6 +422,7 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
 	rcu_read_lock();
 	pptr = array->pptrs[index & array->index_mask];
 	if (map_flags & BPF_F_CPU) {
+		cpu = map_flags >> 32;
 		copy_map_value_long(map, per_cpu_ptr(pptr, cpu), value);
 		bpf_obj_free_fields(array->map.record, per_cpu_ptr(pptr, cpu));
 	} else {
kernel/bpf/hashtab.c +78 -33

215 lines not on this page · raw message

kernel/bpf/syscall.c +6 -24
index 6251ac9bc7e42..430f013f38f06 100644
@@ -133,7 +133,7 @@ bool bpf_map_write_active(const struct bpf_map *map)
 
 static u32 bpf_map_value_size(const struct bpf_map *map, u64 flags)
 {
-	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY && (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)))
+	if (bpf_map_support_cpu_flags(map->map_type) && (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)))
 		return round_up(map->value_size, 8);
 	else if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
 	    map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
@@ -314,7 +314,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value,
 	bpf_disable_instrumentation();
 	if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
 	    map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) {
-		err = bpf_percpu_hash_copy(map, key, value);
+		err = bpf_percpu_hash_copy(map, key, value, flags);
 	} else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
 		err = bpf_percpu_array_copy(map, key, value, flags);
 	} else if (map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) {
@@ -1656,24 +1656,6 @@ static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size)
 	return NULL;
 }
 
-static int check_map_flags(struct bpf_map *map, u64 flags, bool check_flag)
-{
-	if (check_flag && ((u32)flags & ~(BPF_F_LOCK | BPF_F_CPU | BPF_F_ALL_CPUS)))
-		return -EINVAL;
-
-	if ((flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))
-		return -EINVAL;
-
-	if (!(flags & BPF_F_CPU) && flags >> 32)
-		return -EINVAL;
-
-	if ((flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) &&
-		map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY)
-		return -EINVAL;
-
-	return 0;
-}
-
 /* last field in 'union bpf_attr' used by this command */
 #define BPF_MAP_LOOKUP_ELEM_LAST_FIELD flags
 
@@ -1696,7 +1678,7 @@ static int map_lookup_elem(union bpf_attr *attr)
 	if (!(map_get_sys_perms(map, f) & FMODE_CAN_READ))
 		return -EPERM;
 
-	err = check_map_flags(map, attr->flags, true);
+	err = bpf_map_check_flags(map, attr->flags, true);
 	if (err)
 		return err;
 
@@ -1761,7 +1743,7 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
 		goto err_put;
 	}
 
-	err = check_map_flags(map, attr->flags, false);
+	err = bpf_map_check_flags(map, attr->flags, false);
 	if (err)
 		goto err_put;
 
@@ -1967,7 +1949,7 @@ int generic_map_update_batch(struct bpf_map *map, struct file *map_file,
 	void *key, *value;
 	int err = 0;
 
-	err = check_map_flags(map, attr->batch.elem_flags, true);
+	err = bpf_map_check_flags(map, attr->batch.elem_flags, true);
 	if (err)
 		return err;
 
@@ -2026,7 +2008,7 @@ int generic_map_lookup_batch(struct bpf_map *map,
 	u32 value_size, cp, max_count;
 	int err;
 
-	err = check_map_flags(map, attr->batch.elem_flags, true);
+	err = bpf_map_check_flags(map, attr->batch.elem_flags, true);
 	if (err)
 		return err;
 
Signature · 1 line
2.50.1
author
#11 lore raw

PATCH v3 4/6 bpf: Introduce BPF_F_CPU flag for percpu_cgroup_storage maps

In reply to Leon Hwang

Introduce BPF_F_ALL_CPUS flag support for percpu_cgroup_storage maps to allow updating values for all CPUs with a single value.

Introduce BPF_F_CPU flag support for percpu_cgroup_storage maps to allow updating value for specified CPU.

This enhancement enables:

* Efficient update values across all CPUs with a single value when BPF_F_ALL_CPUS is set for update_elem API.
* Targeted update or lookup for a specified CPU when BPF_F_CPU is set.

The BPF_F_CPU flag is passed via map_flags of lookup_elem and update_elem APIs along with embedded cpu field.

---
 include/linux/bpf-cgroup.h |  5 ++--
 include/linux/bpf.h        |  5 ++--
 kernel/bpf/local_storage.c | 47 +++++++++++++++++++++++++++++---------
 kernel/bpf/syscall.c       | 12 ++++------
 4 files changed, 46 insertions(+), 23 deletions(-)
include/linux/bpf-cgroup.h +3 -2
index aedf573bdb426..1cb28660aa866 100644
@@ -172,7 +172,8 @@ void bpf_cgroup_storage_link(struct bpf_cgroup_storage *storage,
 void bpf_cgroup_storage_unlink(struct bpf_cgroup_storage *storage);
 int bpf_cgroup_storage_assign(struct bpf_prog_aux *aux, struct bpf_map *map);
 
-int bpf_percpu_cgroup_storage_copy(struct bpf_map *map, void *key, void *value);
+int bpf_percpu_cgroup_storage_copy(struct bpf_map *map, void *key, void *value,
+				   u64 flags);
 int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,
 				     void *value, u64 flags);
 
@@ -467,7 +468,7 @@ static inline struct bpf_cgroup_storage *bpf_cgroup_storage_alloc(
 static inline void bpf_cgroup_storage_free(
 	struct bpf_cgroup_storage *storage) {}
 static inline int bpf_percpu_cgroup_storage_copy(struct bpf_map *map, void *key,
-						 void *value) {
+						 void *value, u64 flags) {
 	return 0;
 }
 static inline int bpf_percpu_cgroup_storage_update(struct bpf_map *map,
include/linux/bpf.h +3 -2
index dc715eef9cbf4..2684ba32bba0a 100644
@@ -3733,12 +3733,13 @@ static inline int bpf_map_check_cpu_flags(u64 flags, bool check_all_cpus)
 	return 0;
 }
 
-static inline bool bpf_map_support_cpu_flags(enum bpf_map_type map_type)
+static inline bool bpf_map_is_percpu(enum bpf_map_type map_type)
 {
 	switch (map_type) {
 	case BPF_MAP_TYPE_PERCPU_ARRAY:
 	case BPF_MAP_TYPE_PERCPU_HASH:
 	case BPF_MAP_TYPE_LRU_PERCPU_HASH:
+	case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE:
 		return true;
 	default:
 		return false;
@@ -3756,7 +3757,7 @@ static inline int bpf_map_check_flags(struct bpf_map *map, u64 flags, bool check
 	if (!(flags & BPF_F_CPU) && flags >> 32)
 		return -EINVAL;
 
-	if ((flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) && !bpf_map_support_cpu_flags(map->map_type))
+	if ((flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) && !bpf_map_is_percpu(map->map_type))
 		return -EINVAL;
 
 	return 0;
kernel/bpf/local_storage.c +36 -11

97 lines not on this page · raw message

kernel/bpf/syscall.c +4 -8
index 430f013f38f06..3fc52cd0c12de 100644
@@ -133,13 +133,9 @@ bool bpf_map_write_active(const struct bpf_map *map)
 
 static u32 bpf_map_value_size(const struct bpf_map *map, u64 flags)
 {
-	if (bpf_map_support_cpu_flags(map->map_type) && (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)))
-		return round_up(map->value_size, 8);
-	else if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
-	    map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
-	    map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY ||
-	    map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE)
-		return round_up(map->value_size, 8) * num_possible_cpus();
+	if (bpf_map_is_percpu(map->map_type))
+		return flags & (BPF_F_CPU | BPF_F_ALL_CPUS) ? round_up(map->value_size, 8) :
+			round_up(map->value_size, 8) * num_possible_cpus();
 	else if (IS_FD_MAP(map))
 		return sizeof(u32);
 	else
@@ -318,7 +314,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value,
 	} else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
 		err = bpf_percpu_array_copy(map, key, value, flags);
 	} else if (map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) {
-		err = bpf_percpu_cgroup_storage_copy(map, key, value);
+		err = bpf_percpu_cgroup_storage_copy(map, key, value, flags);
 	} else if (map->map_type == BPF_MAP_TYPE_STACK_TRACE) {
 		err = bpf_stackmap_copy(map, key, value);
 	} else if (IS_FD_ARRAY(map) || IS_FD_PROG_ARRAY(map)) {
Signature · 1 line
2.50.1
author
#12 lore raw

PATCH v3 5/6 libbpf: Support BPF_F_CPU for percpu maps

In reply to Leon Hwang

Add libbpf support for the BPF_F_CPU flag for percpu maps by embedding the cpu info into the high 32 bits of:

1. **flags**: bpf_map_lookup_elem_flags(), bpf_map__lookup_elem(), bpf_map_update_elem() and bpf_map__update_elem()
2. **opts->elem_flags**: bpf_map_lookup_batch() and bpf_map_update_batch()

And the flag can be BPF_F_ALL_CPUS, but cannot be
'BPF_F_CPU | BPF_F_ALL_CPUS'.

Behavior:

* If the flag is BPF_F_ALL_CPUS, the update is applied across all CPUs.
* If the flag is BPF_F_CPU, it updates value only to the specified CPU.
* If the flag is BPF_F_CPU, lookup value only from the specified CPU.
* lookup does not support BPF_F_ALL_CPUS.

---
 tools/lib/bpf/bpf.h    |  8 ++++++++
 tools/lib/bpf/libbpf.c | 25 +++++++++++++++++++------
 tools/lib/bpf/libbpf.h | 21 ++++++++-------------
 3 files changed, 35 insertions(+), 19 deletions(-)
tools/lib/bpf/bpf.h +8 -0
index 7252150e7ad35..28acb15e982b3 100644
@@ -286,6 +286,14 @@ LIBBPF_API int bpf_map_lookup_and_delete_batch(int fd, void *in_batch,
  *    Update spin_lock-ed map elements. This must be
  *    specified if the map value contains a spinlock.
  *
+ * **BPF_F_CPU**
+ *    As for percpu maps, update value on the specified CPU. And the cpu
+ *    info is embedded into the high 32 bits of **opts->elem_flags**.
+ *
+ * **BPF_F_ALL_CPUS**
+ *    As for percpu maps, update value across all CPUs. This flag cannot
+ *    be used with BPF_F_CPU at the same time.
+ *
  * @param fd BPF map file descriptor
  * @param keys pointer to an array of *count* keys
  * @param values pointer to an array of *count* values
tools/lib/bpf/libbpf.c +19 -6
index fe4fc5438678c..c949281984880 100644
@@ -10603,7 +10603,7 @@ bpf_object__find_map_fd_by_name(const struct bpf_object *obj, const char *name)
 }
 
 static int validate_map_op(const struct bpf_map *map, size_t key_sz,
-			   size_t value_sz, bool check_value_sz)
+			   size_t value_sz, bool check_value_sz, __u64 flags)
 {
 	if (!map_is_created(map)) /* map is not yet created */
 		return -ENOENT;
@@ -10630,6 +10630,19 @@ static int validate_map_op(const struct bpf_map *map, size_t key_sz,
 		int num_cpu = libbpf_num_possible_cpus();
 		size_t elem_sz = roundup(map->def.value_size, 8);
 
+		if (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) {
+			if ((flags & BPF_F_CPU) && (flags & BPF_F_ALL_CPUS))
+				return -EINVAL;
+			if ((flags >> 32) >= num_cpu)
+				return -ERANGE;
+			if (value_sz != elem_sz) {
+				pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %zu\n",
+					map->name, value_sz, elem_sz);
+				return -EINVAL;
+			}
+			break;
+		}
+
 		if (value_sz != num_cpu * elem_sz) {
 			pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %d * %zu = %zd\n",
 				map->name, value_sz, num_cpu, elem_sz, num_cpu * elem_sz);
@@ -10654,7 +10667,7 @@ int bpf_map__lookup_elem(const struct bpf_map *map,
 {
 	int err;
 
-	err = validate_map_op(map, key_sz, value_sz, true);
+	err = validate_map_op(map, key_sz, value_sz, true, flags);
 	if (err)
 		return libbpf_err(err);
 
@@ -10667,7 +10680,7 @@ int bpf_map__update_elem(const struct bpf_map *map,
 {
 	int err;
 
-	err = validate_map_op(map, key_sz, value_sz, true);
+	err = validate_map_op(map, key_sz, value_sz, true, flags);
 	if (err)
 		return libbpf_err(err);
 
@@ -10679,7 +10692,7 @@ int bpf_map__delete_elem(const struct bpf_map *map,
 {
 	int err;
 
-	err = validate_map_op(map, key_sz, 0, false /* check_value_sz */);
+	err = validate_map_op(map, key_sz, 0, false /* check_value_sz */, 0);
 	if (err)
 		return libbpf_err(err);
 
@@ -10692,7 +10705,7 @@ int bpf_map__lookup_and_delete_elem(const struct bpf_map *map,
 {
 	int err;
 
-	err = validate_map_op(map, key_sz, value_sz, true);
+	err = validate_map_op(map, key_sz, value_sz, true, 0);
 	if (err)
 		return libbpf_err(err);
 
@@ -10704,7 +10717,7 @@ int bpf_map__get_next_key(const struct bpf_map *map,
 {
 	int err;
 
-	err = validate_map_op(map, key_sz, 0, false /* check_value_sz */);
+	err = validate_map_op(map, key_sz, 0, false /* check_value_sz */, 0);
 	if (err)
 		return libbpf_err(err);
 
tools/lib/bpf/libbpf.h +8 -13
index 2e91148d9b44d..6a972a8d060c3 100644
@@ -1196,12 +1196,13 @@ LIBBPF_API struct bpf_map *bpf_map__inner_map(struct bpf_map *map);
  * @param key_sz size in bytes of key data, needs to match BPF map definition's **key_size**
  * @param value pointer to memory in which looked up value will be stored
  * @param value_sz size in byte of value data memory; it has to match BPF map
- * definition's **value_size**. For per-CPU BPF maps value size has to be
- * a product of BPF map value size and number of possible CPUs in the system
- * (could be fetched with **libbpf_num_possible_cpus()**). Note also that for
- * per-CPU values value size has to be aligned up to closest 8 bytes for
- * alignment reasons, so expected size is: `round_up(value_size, 8)
- * * libbpf_num_possible_cpus()`.
+ * definition's **value_size**. For per-CPU BPF maps, value size can be
+ * definition's **value_size** if **BPF_F_CPU** or **BPF_F_ALL_CPUS** is
+ * specified in **flags**, otherwise a product of BPF map value size and number
+ * of possible CPUs in the system (could be fetched with
+ * **libbpf_num_possible_cpus()**). Note else that for per-CPU values value
+ * size has to be aligned up to closest 8 bytes for alignment reasons, so
+ * expected size is: `round_up(value_size, 8) * libbpf_num_possible_cpus()`.
  * @flags extra flags passed to kernel for this operation
  * @return 0, on success; negative error, otherwise
  *
@@ -1219,13 +1220,7 @@ LIBBPF_API int bpf_map__lookup_elem(const struct bpf_map *map,
  * @param key pointer to memory containing bytes of the key
  * @param key_sz size in bytes of key data, needs to match BPF map definition's **key_size**
  * @param value pointer to memory containing bytes of the value
- * @param value_sz size in byte of value data memory; it has to match BPF map
- * definition's **value_size**. For per-CPU BPF maps value size has to be
- * a product of BPF map value size and number of possible CPUs in the system
- * (could be fetched with **libbpf_num_possible_cpus()**). Note also that for
- * per-CPU values value size has to be aligned up to closest 8 bytes for
- * alignment reasons, so expected size is: `round_up(value_size, 8)
- * * libbpf_num_possible_cpus()`.
+ * @param value_sz refer to **bpf_map__lookup_elem**'s description.'
  * @flags extra flags passed to kernel for this operation
  * @return 0, on success; negative error, otherwise
  *
Signature · 1 line
2.50.1
author
#16 lore raw

PATCH v3 6/6 selftests/bpf: Add cases to test BPF_F_CPU flag

In reply to Leon Hwang

Add test coverage for the new BPF_F_CPU flag support in percpu maps. The following APIs are exercised:

* bpf_map_update_batch()
* bpf_map_lookup_batch()
* bpf_map_update_elem()
* bpf_map__update_elem()
* bpf_map_lookup_elem_flags()
* bpf_map__lookup_elem()

cd tools/testing/selftests/bpf/
./test_progs -t percpu_alloc

253/13  percpu_alloc/cpu_flag_percpu_array:OK
253/14  percpu_alloc/cpu_flag_percpu_hash:OK
253/15  percpu_alloc/cpu_flag_lru_percpu_hash:OK
253/16  percpu_alloc/cpu_flag_percpu_cgroup_storage:OK
253     percpu_alloc:OK

Summary: 1/16 PASSED, 0 SKIPPED, 0 FAILED

---
 .../selftests/bpf/prog_tests/percpu_alloc.c   | 224 ++++++++++++++++++
 .../selftests/bpf/progs/percpu_alloc_array.c  |  32 +++
 2 files changed, 256 insertions(+)
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +224 -0
index 343da65864d6d..98b6e8cc7ae60 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 #include <test_progs.h>
+#include "cgroup_helpers.h"
 #include "percpu_alloc_array.skel.h"
 #include "percpu_alloc_cgrp_local_storage.skel.h"
 #include "percpu_alloc_fail.skel.h"
@@ -115,6 +116,221 @@ static void test_failure(void) {
 	RUN_TESTS(percpu_alloc_fail);
 }
 
+static void test_percpu_map_op_cpu_flag(struct bpf_map *map, void *keys, size_t key_sz,
+					u32 max_entries, bool test_batch)
+{
+	int i, j, cpu, map_fd, value_size, nr_cpus, err;
+	u64 *values = NULL, batch = 0, flags;
+	const u64 value = 0xDEADC0DE;
+	size_t value_sz = sizeof(u64);
+	u32 count;
+	LIBBPF_OPTS(bpf_map_batch_opts, batch_opts);
+
+	nr_cpus = libbpf_num_possible_cpus();
+	if (!ASSERT_GT(nr_cpus, 0, "libbpf_num_possible_cpus"))
+		return;
+
+	value_size = value_sz * nr_cpus;
+	values = calloc(max_entries, value_size);
+	if (!ASSERT_OK_PTR(values, "calloc values"))
+		goto out;
+	memset(values, 0, value_size * max_entries);
+
+	map_fd = bpf_map__fd(map);
+	flags = (u64)nr_cpus << 32 | BPF_F_CPU;
+	err = bpf_map_update_elem(map_fd, keys, values, flags);
+	if (!ASSERT_EQ(err, -ERANGE, "bpf_map_update_elem_opts -ERANGE"))
+		goto out;
+
+	err = bpf_map__update_elem(map, keys, key_sz, values, value_sz, flags);
+	if (!ASSERT_EQ(err, -ERANGE, "bpf_map__update_elem_opts -ERANGE"))
+		goto out;
+
+	err = bpf_map_lookup_elem_flags(map_fd, keys, values, flags);
+	if (!ASSERT_EQ(err, -ERANGE, "bpf_map_lookup_elem_opts -ERANGE"))
+		goto out;
+
+	err = bpf_map__lookup_elem(map, keys, key_sz, values, value_sz, flags);
+	if (!ASSERT_EQ(err, -ERANGE, "bpf_map__lookup_elem_opts -ERANGE"))
+		goto out;
+
+	for (cpu = 0; cpu < nr_cpus; cpu++) {
+		/* clear value on all cpus */
+		values[0] = 0;
+		flags = BPF_F_ALL_CPUS;
+		for (i = 0; i < max_entries; i++) {
+			err = bpf_map__update_elem(map, keys + i * key_sz, key_sz, values,
+						   value_sz, flags);
+			if (!ASSERT_OK(err, "bpf_map__update_elem"))
+				goto out;
+		}
+
+		/* update value on specified cpu */
+		for (i = 0; i < max_entries; i++) {
+			values[0] = value;
+			flags = (u64)cpu << 32 | BPF_F_CPU;
+			err = bpf_map__update_elem(map, keys + i * key_sz, key_sz, values,
+						   value_sz, flags);
+			if (!ASSERT_OK(err, "bpf_map__update_elem specified cpu"))
+				goto out;
+
+			/* lookup then check value on CPUs */
+			for (j = 0; j < nr_cpus; j++) {
+				flags = (u64)j << 32 | BPF_F_CPU;
+				err = bpf_map__lookup_elem(map, keys + i * key_sz, key_sz, values,
+							   value_sz, flags);
+				if (!ASSERT_OK(err, "bpf_map__lookup_elem specified cpu"))
+					goto out;
+				if (!ASSERT_EQ(values[0], j != cpu ? 0 : value,
+					       "bpf_map__lookup_elem value on specified cpu"))
+					goto out;
+			}
+		}
+	}
+
+	if (!test_batch)
+		goto out;
+
+	batch_opts.elem_flags = (u64)nr_cpus << 32 | BPF_F_CPU;
+	err = bpf_map_update_batch(map_fd, keys, values, &max_entries, &batch_opts);
+	if (!ASSERT_EQ(err, -ERANGE, "bpf_map_update_batch -ERANGE"))
+		goto out;
+
+	for (cpu = 0; cpu < nr_cpus; cpu++) {
+		memset(values, 0, max_entries * value_size);
+
+		/* clear values across all CPUs */
+		batch_opts.elem_flags = BPF_F_ALL_CPUS;
+		err = bpf_map_update_batch(map_fd, keys, values, &max_entries, &batch_opts);
+		if (!ASSERT_OK(err, "bpf_map_update_batch all cpus"))
+			goto out;
+
+		/* update values on specified CPU */
+		for (i = 0; i < max_entries; i++)
+			values[i] = value;
+
+		batch_opts.elem_flags = (u64)cpu << 32 | BPF_F_CPU;
+		err = bpf_map_update_batch(map_fd, keys, values, &max_entries, &batch_opts);
+		if (!ASSERT_OK(err, "bpf_map_update_batch specified cpu"))
+			goto out;
+
+		/* lookup values on specified CPU */
+		memset(values, 0, max_entries * value_sz);
+		err = bpf_map_lookup_batch(map_fd, NULL, &batch, keys, values, &count, &batch_opts);
+		if (!ASSERT_TRUE(!err || err == -ENOENT, "bpf_map_lookup_batch specified cpu"))
+			goto out;
+
+		for (i = 0; i < max_entries; i++)
+			if (!ASSERT_EQ(values[i], value, "value on specified cpu"))
+				goto out;
+
+		/* lookup values from all CPUs */
+		batch_opts.elem_flags = 0;
+		memset(values, 0, max_entries * value_size);
+		err = bpf_map_lookup_batch(map_fd, NULL, &batch, keys, values, &count, &batch_opts);
+		if (!ASSERT_TRUE(!err || err == -ENOENT, "bpf_map_lookup_batch all cpus"))
+			goto out;
+
+		for (i = 0; i < max_entries; i++) {
+			for (j = 0; j < nr_cpus; j++) {
+				if (!ASSERT_EQ(values[i*nr_cpus + j], j != cpu ? 0 : value,
+					       "value on specified cpu"))
+					goto out;
+			}
+		}
+	}
+
+out:
+	if (values)
+		free(values);
+}
+
+static void test_percpu_map_cpu_flag(enum bpf_map_type map_type)
+{
+	struct percpu_alloc_array *skel;
+	size_t key_sz = sizeof(int);
+	int *keys = NULL, i, err;
+	struct bpf_map *map;
+	u32 max_entries;
+
+	skel = percpu_alloc_array__open();
+	if (!ASSERT_OK_PTR(skel, "percpu_alloc_array__open"))
+		return;
+
+	map = skel->maps.percpu;
+	bpf_map__set_type(map, map_type);
+
+	err = percpu_alloc_array__load(skel);
+	if (!ASSERT_OK(err, "test_percpu_alloc__load"))
+		goto out;
+
+	max_entries = bpf_map__max_entries(map);
+	keys = calloc(max_entries, key_sz);
+	if (!ASSERT_OK_PTR(keys, "calloc keys"))
+		goto out;
+
+	for (i = 0; i < max_entries; i++)
+		keys[i] = i;
+
+	test_percpu_map_op_cpu_flag(map, keys, key_sz, max_entries, true);
+out:
+	if (keys)
+		free(keys);
+	percpu_alloc_array__destroy(skel);
+}
+
+static void test_percpu_array_cpu_flag(void)
+{
+	test_percpu_map_cpu_flag(BPF_MAP_TYPE_PERCPU_ARRAY);
+}
+
+static void test_percpu_hash_cpu_flag(void)
+{
+	test_percpu_map_cpu_flag(BPF_MAP_TYPE_PERCPU_HASH);
+}
+
+static void test_lru_percpu_hash_cpu_flag(void)
+{
+	test_percpu_map_cpu_flag(BPF_MAP_TYPE_LRU_PERCPU_HASH);
+}
+
+static void test_percpu_cgroup_storage_cpu_flag(void)
+{
+	struct bpf_cgroup_storage_key key;
+	struct percpu_alloc_array *skel;
+	int cgroup = -1, prog_fd, err;
+	struct bpf_map *map;
+
+	skel = percpu_alloc_array__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "percpu_alloc_array__open_and_load"))
+		return;
+
+	cgroup = create_and_get_cgroup("/cg_percpu");
+	if (!ASSERT_GE(cgroup, 0, "create_and_get_cgroup"))
+		goto out;
+
+	err = join_cgroup("/cg_percpu");
+	if (!ASSERT_OK(err, "join_cgroup"))
+		goto out;
+
+	prog_fd = bpf_program__fd(skel->progs.cgroup_egress);
+	err = bpf_prog_attach(prog_fd, cgroup, BPF_CGROUP_INET_EGRESS, 0);
+	if (!ASSERT_OK(err, "bpf_prog_attach"))
+		goto out;
+
+	map = skel->maps.percpu_cgroup_storage;
+	err = bpf_map_get_next_key(bpf_map__fd(map), NULL, &key);
+	if (!ASSERT_OK(err, "bpf_map_get_next_key"))
+		goto out;
+
+	test_percpu_map_op_cpu_flag(map, &key, sizeof(key), 1, false);
+out:
+	bpf_prog_detach2(-1, cgroup, BPF_CGROUP_INET_EGRESS);
+	close(cgroup);
+	cleanup_cgroup_environment();
+	percpu_alloc_array__destroy(skel);
+}
+
 void test_percpu_alloc(void)
 {
 	if (test__start_subtest("array"))
@@ -125,4 +341,12 @@ void test_percpu_alloc(void)
 		test_cgrp_local_storage();
 	if (test__start_subtest("failure_tests"))
 		test_failure();
+	if (test__start_subtest("cpu_flag_percpu_array"))
+		test_percpu_array_cpu_flag();
+	if (test__start_subtest("cpu_flag_percpu_hash"))
+		test_percpu_hash_cpu_flag();
+	if (test__start_subtest("cpu_flag_lru_percpu_hash"))
+		test_lru_percpu_hash_cpu_flag();
+	if (test__start_subtest("cpu_flag_percpu_cgroup_storage"))
+		test_percpu_cgroup_storage_cpu_flag();
 }
tools/testing/selftests/bpf/progs/percpu_alloc_array.c +32 -0
index 37c2d2608ec0b..427301909c349 100644
@@ -187,4 +187,36 @@ int BPF_PROG(test_array_map_10)
 	return 0;
 }
 
+struct {
+	__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+	__uint(max_entries, 2);
+	__type(key, int);
+	__type(value, u64);
+} percpu SEC(".maps");
+
+SEC("?fentry/bpf_fentry_test1")
+int BPF_PROG(test_percpu_array, int x)
+{
+	u64 value = 0xDEADC0DE;
+	int key = 0;
+
+	bpf_map_update_elem(&percpu, &key, &value, BPF_ANY);
+	return 0;
+}
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE);
+	__type(key, struct bpf_cgroup_storage_key);
+	__type(value, u64);
+} percpu_cgroup_storage SEC(".maps");
+
+SEC("cgroup_skb/egress")
+int cgroup_egress(struct __sk_buff *skb)
+{
+	u64 *val = bpf_get_local_storage(&percpu_cgroup_storage, 0);
+
+	__sync_fetch_and_add(val, 1);
+	return 1;
+}
+
 char _license[] SEC("license") = "GPL";
Signature · 1 line
2.50.1
#2 lore raw

Re: [PATCH bpf-next v3 1/6] bpf: Introduce internal check_map_flags helper function

In reply to Leon Hwang
Quoted 17 lines · Leon Hwang

It is to unify map flags checking for lookup, update, lookup_batch and update_batch.

Therefore, it will be convenient to check BPF_F_CPU flag in this helper function for them in next patch.

---
 kernel/bpf/syscall.c | 45 ++++++++++++++++++++++----------------------
 1 file changed, 22 insertions(+), 23 deletions(-)
kernel/bpf/syscall.c +0 -0
index 0fbfa8532c392..19f7f5de5e7dc 100644

kernel/bpf/syscall.c static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size) · line 1657

Quoted 4 lines
kernel/bpf/syscall.c +1 -0
@@ -1654,6 +1654,17 @@ static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size)
        return NULL;
 }
 
+static int check_map_flags(struct bpf_map *map, u64 flags, bool check_flag)

"check_map_flags" is super generically named... (and actually misleading, it's not map flags you are checking), so I think it should be something along the lines of "check_map_op_flag", i.e. map *operation* flag?

but also check_flag bool argument name for a function called "check flags" is so confusing... The idea here is whether we should enforce there is no *extra* flags beyond those common for all operations, right? So maybe call it "allow_extra_flags" or alternatively "strict_extra_flags", something suggesting that his is something in addition to common flags

alternatively, and perhaps best of all, I'd move that particular check outside and just maintain something like ARRAY_CREATE_FLAG_MASK for each operation, checking it explicitly where appropriate. WDYT?

pw-bot: cr

[...]

#6 lore raw

Re: [PATCH bpf-next v3 2/6] bpf: Introduce BPF_F_CPU flag for percpu_array maps

In reply to Leon Hwang
Quoted 29 lines · Leon Hwang

Introduce support for the BPF_F_ALL_CPUS flag in percpu_array maps to allow updating values for all CPUs with a single value.

Introduce support for the BPF_F_CPU flag in percpu_array maps to allow updating value for specified CPU.

This enhancement enables:

* Efficient update values across all CPUs with a single value when BPF_F_ALL_CPUS is set for update_elem and update_batch APIs.
* Targeted update or lookup for a specified CPU when BPF_F_CPU is set.

The BPF_F_CPU flag is passed via:

* map_flags of lookup_elem and update_elem APIs along with embedded cpu field.
* elem_flags of lookup_batch and update_batch APIs along with embedded cpu field.

---
 include/linux/bpf.h            |  3 +-
 include/uapi/linux/bpf.h       |  2 ++
 kernel/bpf/arraymap.c          | 56 ++++++++++++++++++++++++++--------
 kernel/bpf/syscall.c           | 27 ++++++++++------
 tools/include/uapi/linux/bpf.h |  2 ++
 5 files changed, 67 insertions(+), 23 deletions(-)

[...]

Quoted 5 lines
kernel/bpf/syscall.c +0 -0
index 19f7f5de5e7dc..6251ac9bc7e42 100644

kernel/bpf/syscall.c bool bpf_map_write_active(const struct bpf_map *map) · line 137

Quoted 9 lines
kernel/bpf/syscall.c +3 -2
@@ -131,9 +131,11 @@ bool bpf_map_write_active(const struct bpf_map *map)
        return atomic64_read(&map->writecnt) != 0;
 }
 
-static u32 bpf_map_value_size(const struct bpf_map *map)
+static u32 bpf_map_value_size(const struct bpf_map *map, u64 flags)
 {
-       if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
+       if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY && (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)))
+               return round_up(map->value_size, 8);

this doesn't depend on the PERCPU_ARRAY map type, right? Any map for which we allowed BPF_F_CPU or BPF_F_ALL_CPUS would use this formula? (and if map doesn't support those flags, you should have filtered that out earlier, no?) So maybe add this is first separate condition before all this map type specific logic?

Quoted 14 lines
kernel/bpf/syscall.c +2 -1
+       else if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
            map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
            map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY ||
            map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE)
@@ -314,7 +316,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value,
            map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) {
                err = bpf_percpu_hash_copy(map, key, value);
        } else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
-               err = bpf_percpu_array_copy(map, key, value);
+               err = bpf_percpu_array_copy(map, key, value, flags);
        } else if (map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) {
                err = bpf_percpu_cgroup_storage_copy(map, key, value);
        } else if (map->map_type == BPF_MAP_TYPE_STACK_TRACE) {

kernel/bpf/syscall.c static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size) · line 1659

Quoted 2 lines
kernel/bpf/syscall.c +0 -0
@@ -1656,12 +1658,19 @@ static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size)
 
 static int check_map_flags(struct bpf_map *map, u64 flags, bool check_flag)

you are later moving this into bpf.h header, so do it in previous patch early on, less unnecessary code churn and easier to review actual changes to that function in subsequent patches

Quoted 18 lines
kernel/bpf/syscall.c +8 -1
 {
-       if (check_flag && (flags & ~BPF_F_LOCK))
+       if (check_flag && ((u32)flags & ~(BPF_F_LOCK | BPF_F_CPU | BPF_F_ALL_CPUS)))
                return -EINVAL;
 
        if ((flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))
                return -EINVAL;
 
+       if (!(flags & BPF_F_CPU) && flags >> 32)
+               return -EINVAL;
+
+       if ((flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) &&
+               map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY)
+               return -EINVAL;
+
        return 0;
 }
 

[...]

#9 lore raw

Re: [PATCH bpf-next v3 3/6] bpf: Introduce BPF_F_CPU flag for percpu_hash and lru_percpu_hash maps

In reply to Leon Hwang
Quoted 28 lines · Leon Hwang

Introduce BPF_F_ALL_CPUS flag support for percpu_hash and lru_percpu_hash maps to allow updating values for all CPUs with a single value.

Introduce BPF_F_CPU flag support for percpu_hash and lru_percpu_hash maps to allow updating value for specified CPU.

This enhancement enables:

* Efficient update values across all CPUs with a single value when BPF_F_ALL_CPUS is set for update_elem and update_batch APIs.
* Targeted update or lookup for a specified CPU when BPF_F_CPU is set.

The BPF_F_CPU flag is passed via:

* map_flags of lookup_elem and update_elem APIs along with embedded cpu field.
* elem_flags of lookup_batch and update_batch APIs along with embedded cpu field.

---
 include/linux/bpf.h   |  54 +++++++++++++++++++-
 kernel/bpf/arraymap.c |  29 ++++-------
 kernel/bpf/hashtab.c  | 111 +++++++++++++++++++++++++++++-------------
 kernel/bpf/syscall.c  |  30 +++---------
 4 files changed, 147 insertions(+), 77 deletions(-)

[...]

Quoted 14 lines
kernel/bpf/arraymap.c +1 -6
@@ -397,22 +395,14 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
                            u64 map_flags)
 {
        struct bpf_array *array = container_of(map, struct bpf_array, map);
-       const u64 cpu_flags = BPF_F_CPU | BPF_F_ALL_CPUS;
        u32 index = *(u32 *)key;
        void __percpu *pptr;
+       int off = 0, err;
        u32 size, cpu;
-       int off = 0;
-
-       if (unlikely((u32)map_flags > BPF_F_ALL_CPUS))
-               /* unknown flags */
-               return -EINVAL;

kernel/bpf/arraymap.c int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, · line 405

Quoted 10 lines
kernel/bpf/arraymap.c +3 -6
-       if (unlikely((map_flags & cpu_flags) == cpu_flags))
-               return -EINVAL;
 
-       cpu = map_flags >> 32;
-       if (unlikely((map_flags & BPF_F_CPU) && cpu >= num_possible_cpus()))
-               /* invalid cpu */
-               return -ERANGE;
+       err = bpf_map_check_cpu_flags(map_flags, true);
+       if (unlikely(err))
+               return err;

again, unnecessary churn, why not add this function in previous patch when you add cpu flags ?

Quoted 20 lines
kernel/bpf/arraymap.c +1 -0
        if (unlikely(index >= array->map.max_entries))
                /* all elements were pre-allocated, cannot insert a new one */
@@ -432,6 +422,7 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
        rcu_read_lock();
        pptr = array->pptrs[index & array->index_mask];
        if (map_flags & BPF_F_CPU) {
+               cpu = map_flags >> 32;
                copy_map_value_long(map, per_cpu_ptr(pptr, cpu), value);
                bpf_obj_free_fields(array->map.record, per_cpu_ptr(pptr, cpu));
        } else {
kernel/bpf/hashtab.c +0 -1
index 71f9931ac64cd..34a35cdade425 100644
@@ -937,24 +937,39 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
 }
 
 static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr,
-                           void *value, bool onallcpus)

kernel/bpf/hashtab.c static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) · line 948

Quoted 10 lines
kernel/bpf/hashtab.c +6 -1
+                           void *value, bool onallcpus, u64 map_flags)
 {
+       int cpu = map_flags & BPF_F_CPU ? map_flags >> 32 : 0;
+       int current_cpu = raw_smp_processor_id();
+
        if (!onallcpus) {
                /* copy true value_size bytes */
-               copy_map_value(&htab->map, this_cpu_ptr(pptr), value);
+               copy_map_value(&htab->map, (map_flags & BPF_F_CPU) && cpu != current_cpu ?
+                              per_cpu_ptr(pptr, cpu) : this_cpu_ptr(pptr), value);

is there any benefit to this cpu == current_cpu special casing? Wouldn't per_cpu_ptr() do the right thing even if cpu == current_cpu?

Quoted 10 lines
kernel/bpf/hashtab.c +7 -1
        } else {
                u32 size = round_up(htab->map.value_size, 8);
-               int off = 0, cpu;
+               int off = 0;
+
+               if (map_flags & BPF_F_CPU) {
+                       copy_map_value_long(&htab->map, cpu != current_cpu ?
+                                           per_cpu_ptr(pptr, cpu) : this_cpu_ptr(pptr), value);
+                       return;
+               }

[...]

kernel/bpf/hashtab.c __htab_map_lookup_and_delete_batch(struct bpf_map *map, · line 1837

Quoted 9 lines
kernel/bpf/hashtab.c +1 -4
@@ -1806,10 +1834,17 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *map,
                        void __percpu *pptr;
 
                        pptr = htab_elem_get_ptr(l, map->key_size);
-                       for_each_possible_cpu(cpu) {
-                               copy_map_value_long(&htab->map, dst_val + off, per_cpu_ptr(pptr, cpu));
-                               check_and_init_map_value(&htab->map, dst_val + off);
-                               off += size;
+                       if (!do_delete && (elem_map_flags & BPF_F_CPU)) {

if do_delete is true we can't have BPF_F_CPU set, right? We checked that above, so why all these complications?

Quoted 44 lines
kernel/bpf/hashtab.c +18 -7
+                               cpu = elem_map_flags >> 32;
+                               copy_map_value_long(&htab->map, dst_val, per_cpu_ptr(pptr, cpu));
+                               check_and_init_map_value(&htab->map, dst_val);
+                       } else {
+                               for_each_possible_cpu(cpu) {
+                                       copy_map_value_long(&htab->map, dst_val + off,
+                                                           per_cpu_ptr(pptr, cpu));
+                                       check_and_init_map_value(&htab->map, dst_val + off);
+                                       off += size;
+                               }
                        }
                } else {
                        value = htab_elem_value(l, key_size);
@@ -2365,14 +2400,18 @@ static void *htab_lru_percpu_map_lookup_percpu_elem(struct bpf_map *map, void *k
        return NULL;
 }
 
-int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value)
+int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value, u64 map_flags)
 {
+       int ret, cpu, off = 0;
        struct htab_elem *l;
        void __percpu *pptr;
-       int ret = -ENOENT;
-       int cpu, off = 0;
        u32 size;
 
+       ret = bpf_map_check_cpu_flags(map_flags, false);
+       if (unlikely(ret))
+               return ret;
+       ret = -ENOENT;
+
        /* per_cpu areas are zero-filled and bpf programs can only
         * access 'value_size' of them, so copying rounded areas
         * will not leak any kernel data
@@ -2386,10 +2425,16 @@ int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value)
         * eviction heuristics when user space does a map walk.
         */
        pptr = htab_elem_get_ptr(l, map->key_size);
-       for_each_possible_cpu(cpu) {
-               copy_map_value_long(map, value + off, per_cpu_ptr(pptr, cpu));
-               check_and_init_map_value(map, value + off);
-               off += size;
+       if (map_flags & BPF_F_CPU) {

kernel/bpf/hashtab.c int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value) · line 2438

Quoted 10 lines
kernel/bpf/hashtab.c +9 -0
+               cpu = map_flags >> 32;
+               copy_map_value_long(map, value, per_cpu_ptr(pptr, cpu));
+               check_and_init_map_value(map, value);
+       } else {
+               for_each_possible_cpu(cpu) {
+                       copy_map_value_long(map, value + off, per_cpu_ptr(pptr, cpu));
+                       check_and_init_map_value(map, value + off);
+                       off += size;
+               }
        }

it feels like this whole logic of copying per-cpu value to/from user should be generic between all per-cpu maps, once we get that void __percpu * pointer, no? See if you can extract it as reusable helper (but, you know, without all the per-map type special casing, though it doesn't seem like you should need it, though I might be missing details, of course). One for bpf_percpu_copy_to_user() and another for bpf_percpu_copy_from_user(), which would take into account all these cpu flags?

[...]

#13 lore raw

Re: [PATCH bpf-next v3 5/6] libbpf: Support BPF_F_CPU for percpu maps

In reply to Leon Hwang
Quoted 59 lines · Leon Hwang

Add libbpf support for the BPF_F_CPU flag for percpu maps by embedding the cpu info into the high 32 bits of:

1. **flags**: bpf_map_lookup_elem_flags(), bpf_map__lookup_elem(), bpf_map_update_elem() and bpf_map__update_elem()
2. **opts->elem_flags**: bpf_map_lookup_batch() and bpf_map_update_batch()

And the flag can be BPF_F_ALL_CPUS, but cannot be
'BPF_F_CPU | BPF_F_ALL_CPUS'.

Behavior:

* If the flag is BPF_F_ALL_CPUS, the update is applied across all CPUs.
* If the flag is BPF_F_CPU, it updates value only to the specified CPU.
* If the flag is BPF_F_CPU, lookup value only from the specified CPU.
* lookup does not support BPF_F_ALL_CPUS.

---
 tools/lib/bpf/bpf.h    |  8 ++++++++
 tools/lib/bpf/libbpf.c | 25 +++++++++++++++++++------
 tools/lib/bpf/libbpf.h | 21 ++++++++-------------
 3 files changed, 35 insertions(+), 19 deletions(-)
tools/lib/bpf/bpf.h +8 -0
index 7252150e7ad35..28acb15e982b3 100644
@@ -286,6 +286,14 @@ LIBBPF_API int bpf_map_lookup_and_delete_batch(int fd, void *in_batch,
  *    Update spin_lock-ed map elements. This must be
  *    specified if the map value contains a spinlock.
  *
+ * **BPF_F_CPU**
+ *    As for percpu maps, update value on the specified CPU. And the cpu
+ *    info is embedded into the high 32 bits of **opts->elem_flags**.
+ *
+ * **BPF_F_ALL_CPUS**
+ *    As for percpu maps, update value across all CPUs. This flag cannot
+ *    be used with BPF_F_CPU at the same time.
+ *
  * @param fd BPF map file descriptor
  * @param keys pointer to an array of *count* keys
  * @param values pointer to an array of *count* values
tools/lib/bpf/libbpf.c +1 -1
index fe4fc5438678c..c949281984880 100644
@@ -10603,7 +10603,7 @@ bpf_object__find_map_fd_by_name(const struct bpf_object *obj, const char *name)
 }
 
 static int validate_map_op(const struct bpf_map *map, size_t key_sz,
-                          size_t value_sz, bool check_value_sz)
+                          size_t value_sz, bool check_value_sz, __u64 flags)
 {
        if (!map_is_created(map)) /* map is not yet created */
                return -ENOENT;

tools/lib/bpf/libbpf.c static int validate_map_op(const struct bpf_map *map, size_t key_sz, · line 10637

Quoted 8 lines
tools/lib/bpf/libbpf.c +5 -0
@@ -10630,6 +10630,19 @@ static int validate_map_op(const struct bpf_map *map, size_t key_sz,
                int num_cpu = libbpf_num_possible_cpus();
                size_t elem_sz = roundup(map->def.value_size, 8);
 
+               if (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) {
+                       if ((flags & BPF_F_CPU) && (flags & BPF_F_ALL_CPUS))
+                               return -EINVAL;
+                       if ((flags >> 32) >= num_cpu)
+                               return -ERANGE;

The idea of validate_map_op() is to make it easier for users to understand what's wrong with how they deal with the map, rather than just getting indiscriminate -EINVAL from the kernel.

Point being: add human-readable pr_warn() explanations for all the new conditions you are detecting, otherwise it's just meaningless.

Quoted 30 lines
tools/lib/bpf/libbpf.c +10 -2
+                       if (value_sz != elem_sz) {
+                               pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %zu\n",
+                                       map->name, value_sz, elem_sz);
+                               return -EINVAL;
+                       }
+                       break;
+               }
+
                if (value_sz != num_cpu * elem_sz) {
                        pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %d * %zu = %zd\n",
                                map->name, value_sz, num_cpu, elem_sz, num_cpu * elem_sz);
@@ -10654,7 +10667,7 @@ int bpf_map__lookup_elem(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, value_sz, true);
+       err = validate_map_op(map, key_sz, value_sz, true, flags);
        if (err)
                return libbpf_err(err);
 
@@ -10667,7 +10680,7 @@ int bpf_map__update_elem(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, value_sz, true);
+       err = validate_map_op(map, key_sz, value_sz, true, flags);
        if (err)
                return libbpf_err(err);
 

tools/lib/bpf/libbpf.c int bpf_map__delete_elem(const struct bpf_map *map, · line 10695

Quoted 5 lines
tools/lib/bpf/libbpf.c +1 -1
@@ -10679,7 +10692,7 @@ int bpf_map__delete_elem(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, 0, false /* check_value_sz */);
+       err = validate_map_op(map, key_sz, 0, false /* check_value_sz */, 0);

hard-coded 0 instead of flags, why?

tools/lib/bpf/libbpf.c int bpf_map__lookup_and_delete_elem(const struct bpf_map *map, · line 10708

Quoted 9 lines
tools/lib/bpf/libbpf.c +1 -1
        if (err)
                return libbpf_err(err);
 
@@ -10692,7 +10705,7 @@ int bpf_map__lookup_and_delete_elem(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, value_sz, true);
+       err = validate_map_op(map, key_sz, value_sz, true, 0);

same about flags

Quoted 22 lines
tools/lib/bpf/libbpf.c +1 -1
        if (err)
                return libbpf_err(err);
 
@@ -10704,7 +10717,7 @@ int bpf_map__get_next_key(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, 0, false /* check_value_sz */);
+       err = validate_map_op(map, key_sz, 0, false /* check_value_sz */, 0);
        if (err)
                return libbpf_err(err);
 
tools/lib/bpf/libbpf.h +0 -2
index 2e91148d9b44d..6a972a8d060c3 100644
@@ -1196,12 +1196,13 @@ LIBBPF_API struct bpf_map *bpf_map__inner_map(struct bpf_map *map);
  * @param key_sz size in bytes of key data, needs to match BPF map definition's **key_size**
  * @param value pointer to memory in which looked up value will be stored
  * @param value_sz size in byte of value data memory; it has to match BPF map
- * definition's **value_size**. For per-CPU BPF maps value size has to be
- * a product of BPF map value size and number of possible CPUs in the system

tools/lib/bpf/libbpf.h LIBBPF_API struct bpf_map *bpf_map__inner_map(struct bpf_map *map); · line 1204

Quoted 10 lines
tools/lib/bpf/libbpf.h +6 -4
- * (could be fetched with **libbpf_num_possible_cpus()**). Note also that for
- * per-CPU values value size has to be aligned up to closest 8 bytes for
- * alignment reasons, so expected size is: `round_up(value_size, 8)
- * * libbpf_num_possible_cpus()`.
+ * definition's **value_size**. For per-CPU BPF maps, value size can be
+ * definition's **value_size** if **BPF_F_CPU** or **BPF_F_ALL_CPUS** is
+ * specified in **flags**, otherwise a product of BPF map value size and number
+ * of possible CPUs in the system (could be fetched with
+ * **libbpf_num_possible_cpus()**). Note else that for per-CPU values value
+ * size has to be aligned up to closest 8 bytes for alignment reasons, so

nit: aligned up for alignment reasons... drop "for alignment reasons", I guess?

Quoted 22 lines
tools/lib/bpf/libbpf.h +2 -8
+ * expected size is: `round_up(value_size, 8) * libbpf_num_possible_cpus()`.
  * @flags extra flags passed to kernel for this operation
  * @return 0, on success; negative error, otherwise
  *
@@ -1219,13 +1220,7 @@ LIBBPF_API int bpf_map__lookup_elem(const struct bpf_map *map,
  * @param key pointer to memory containing bytes of the key
  * @param key_sz size in bytes of key data, needs to match BPF map definition's **key_size**
  * @param value pointer to memory containing bytes of the value
- * @param value_sz size in byte of value data memory; it has to match BPF map
- * definition's **value_size**. For per-CPU BPF maps value size has to be
- * a product of BPF map value size and number of possible CPUs in the system
- * (could be fetched with **libbpf_num_possible_cpus()**). Note also that for
- * per-CPU values value size has to be aligned up to closest 8 bytes for
- * alignment reasons, so expected size is: `round_up(value_size, 8)
- * * libbpf_num_possible_cpus()`.
+ * @param value_sz refer to **bpf_map__lookup_elem**'s description.'
  * @flags extra flags passed to kernel for this operation
  * @return 0, on success; negative error, otherwise
  *
--

2.50.1

author
#7 lore raw

Re: [PATCH bpf-next v3 2/6] bpf: Introduce BPF_F_CPU flag for percpu_array maps

In reply to Andrii Nakryiko
Quoted 41 lines · Andrii Nakryiko
Quoted 29 lines · Leon Hwang

Introduce support for the BPF_F_ALL_CPUS flag in percpu_array maps to allow updating values for all CPUs with a single value.

Introduce support for the BPF_F_CPU flag in percpu_array maps to allow updating value for specified CPU.

This enhancement enables:

* Efficient update values across all CPUs with a single value when BPF_F_ALL_CPUS is set for update_elem and update_batch APIs.
* Targeted update or lookup for a specified CPU when BPF_F_CPU is set.

The BPF_F_CPU flag is passed via:

* map_flags of lookup_elem and update_elem APIs along with embedded cpu field.
* elem_flags of lookup_batch and update_batch APIs along with embedded cpu field.

---
 include/linux/bpf.h            |  3 +-
 include/uapi/linux/bpf.h       |  2 ++
 kernel/bpf/arraymap.c          | 56 ++++++++++++++++++++++++++--------
 kernel/bpf/syscall.c           | 27 ++++++++++------
 tools/include/uapi/linux/bpf.h |  2 ++
 5 files changed, 67 insertions(+), 23 deletions(-)

[...]

Quoted 8 lines
kernel/bpf/syscall.c +0 -0
index 19f7f5de5e7dc..6251ac9bc7e42 100644
@@ -131,9 +131,11 @@ bool bpf_map_write_active(const struct bpf_map *map)
        return atomic64_read(&map->writecnt) != 0;
 }
 
Quoted 13 lines

kernel/bpf/syscall.c bool bpf_map_write_active(const struct bpf_map *map) · line 137

Quoted 6 lines
kernel/bpf/syscall.c +3 -2
-static u32 bpf_map_value_size(const struct bpf_map *map)
+static u32 bpf_map_value_size(const struct bpf_map *map, u64 flags)
 {
-       if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
+       if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY && (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)))
+               return round_up(map->value_size, 8);

this doesn't depend on the PERCPU_ARRAY map type, right? Any map for which we allowed BPF_F_CPU or BPF_F_ALL_CPUS would use this formula? (and if map doesn't support those flags, you should have filtered that out earlier, no?) So maybe add this is first separate condition before all this map type specific logic?

Right.

I will remove this map type specific logic in next revision.

Quoted 15 lines
Quoted 15 lines
kernel/bpf/syscall.c +2 -1
+       else if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
            map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
            map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY ||
            map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE)
@@ -314,7 +316,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value,
            map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) {
                err = bpf_percpu_hash_copy(map, key, value);
        } else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
-               err = bpf_percpu_array_copy(map, key, value);
+               err = bpf_percpu_array_copy(map, key, value, flags);
        } else if (map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) {
                err = bpf_percpu_cgroup_storage_copy(map, key, value);
        } else if (map->map_type == BPF_MAP_TYPE_STACK_TRACE) {
@@ -1656,12 +1658,19 @@ static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size)
 
Quoted 6 lines

kernel/bpf/syscall.c static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size) · line 1659

Quoted 1 line
kernel/bpf/syscall.c +0 -0
 static int check_map_flags(struct bpf_map *map, u64 flags, bool check_flag)

you are later moving this into bpf.h header, so do it in previous patch early on, less unnecessary code churn and easier to review actual changes to that function in subsequent patches

Ack.

Thanks,
Leon

[...]

author
#10 lore raw

Re: [PATCH bpf-next v3 3/6] bpf: Introduce BPF_F_CPU flag for percpu_hash and lru_percpu_hash maps

In reply to Andrii Nakryiko
Quoted 49 lines · Andrii Nakryiko
Quoted 28 lines · Leon Hwang

Introduce BPF_F_ALL_CPUS flag support for percpu_hash and lru_percpu_hash maps to allow updating values for all CPUs with a single value.

Introduce BPF_F_CPU flag support for percpu_hash and lru_percpu_hash maps to allow updating value for specified CPU.

This enhancement enables:

* Efficient update values across all CPUs with a single value when BPF_F_ALL_CPUS is set for update_elem and update_batch APIs.
* Targeted update or lookup for a specified CPU when BPF_F_CPU is set.

The BPF_F_CPU flag is passed via:

* map_flags of lookup_elem and update_elem APIs along with embedded cpu field.
* elem_flags of lookup_batch and update_batch APIs along with embedded cpu field.

---
 include/linux/bpf.h   |  54 +++++++++++++++++++-
 kernel/bpf/arraymap.c |  29 ++++-------
 kernel/bpf/hashtab.c  | 111 +++++++++++++++++++++++++++++-------------
 kernel/bpf/syscall.c  |  30 +++---------
 4 files changed, 147 insertions(+), 77 deletions(-)

[...]

Quoted 17 lines
kernel/bpf/arraymap.c +1 -8
@@ -397,22 +395,14 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
                            u64 map_flags)
 {
        struct bpf_array *array = container_of(map, struct bpf_array, map);
-       const u64 cpu_flags = BPF_F_CPU | BPF_F_ALL_CPUS;
        u32 index = *(u32 *)key;
        void __percpu *pptr;
+       int off = 0, err;
        u32 size, cpu;
-       int off = 0;
-
-       if (unlikely((u32)map_flags > BPF_F_ALL_CPUS))
-               /* unknown flags */
-               return -EINVAL;
-       if (unlikely((map_flags & cpu_flags) == cpu_flags))
-               return -EINVAL;
 
Quoted 11 lines

kernel/bpf/arraymap.c int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, · line 405

Quoted 7 lines
kernel/bpf/arraymap.c +3 -4
-       cpu = map_flags >> 32;
-       if (unlikely((map_flags & BPF_F_CPU) && cpu >= num_possible_cpus()))
-               /* invalid cpu */
-               return -ERANGE;
+       err = bpf_map_check_cpu_flags(map_flags, true);
+       if (unlikely(err))
+               return err;

again, unnecessary churn, why not add this function in previous patch when you add cpu flags ?

Ack.

Quoted 19 lines
Quoted 19 lines
kernel/bpf/arraymap.c +1 -0
        if (unlikely(index >= array->map.max_entries))
                /* all elements were pre-allocated, cannot insert a new one */
@@ -432,6 +422,7 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
        rcu_read_lock();
        pptr = array->pptrs[index & array->index_mask];
        if (map_flags & BPF_F_CPU) {
+               cpu = map_flags >> 32;
                copy_map_value_long(map, per_cpu_ptr(pptr, cpu), value);
                bpf_obj_free_fields(array->map.record, per_cpu_ptr(pptr, cpu));
        } else {
kernel/bpf/hashtab.c +0 -0
index 71f9931ac64cd..34a35cdade425 100644
@@ -937,24 +937,39 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
 }
 
Quoted 16 lines

kernel/bpf/hashtab.c static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) · line 948

Quoted 12 lines
kernel/bpf/hashtab.c +6 -2
 static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr,
-                           void *value, bool onallcpus)
+                           void *value, bool onallcpus, u64 map_flags)
 {
+       int cpu = map_flags & BPF_F_CPU ? map_flags >> 32 : 0;
+       int current_cpu = raw_smp_processor_id();
+
        if (!onallcpus) {
                /* copy true value_size bytes */
-               copy_map_value(&htab->map, this_cpu_ptr(pptr), value);
+               copy_map_value(&htab->map, (map_flags & BPF_F_CPU) && cpu != current_cpu ?
+                              per_cpu_ptr(pptr, cpu) : this_cpu_ptr(pptr), value);

is there any benefit to this cpu == current_cpu special casing? Wouldn't per_cpu_ptr() do the right thing even if cpu == current_cpu?

per_cpu_ptr() seems doing the same thing of this_cpu_ptr().

However, after having a deep research of per_cpu_ptr() and this_cpu_ptr(), I think this_cpu_ptr() is more efficient than per_cpu_ptr().

e.g.

DEFINE_PER_CPU(int, my_percpu_var) = 0;

int percpu_read(int cpu);
int thiscpu_read(void);
int percpu_currcpu_read(int cpu);

int percpu_read(int cpu)

{
    return *(int *)(void *)per_cpu_ptr(&my_percpu_var, cpu);
}

EXPORT_SYMBOL(percpu_read);

int thiscpu_read(void)

{
    return *(int *)(void *)this_cpu_ptr(&my_percpu_var);
}

EXPORT_SYMBOL(thiscpu_read);

int percpu_currcpu_read(int cpu)

{
    if (cpu == raw_smp_processor_id())
        return *(int *)(void *)this_cpu_ptr(&my_percpu_var);
    else
        return *(int *)(void *)per_cpu_ptr(&my_percpu_var, cpu);
}

EXPORT_SYMBOL(percpu_currcpu_read);

Source code link[0].

With disassembling these functions, the core insns of this_cpu_ptr() are:

0xffffffffc15c5076 <+6/0x6>:      48 c7 c0 04 60 03 00	movq	$0x36004, %rax
0xffffffffc15c507d <+13/0xd>:     65 48 03 05 7b 49 a5 3e	addq	%gs:0x3ea5497b(%rip), %rax
0xffffffffc15c5085 <+21/0x15>:    8b 00              	movl	(%rax), %eax

The core insns of per_cpu_ptr() are:

0xffffffffc15c501b <+11/0xb>:     49 c7 c4 04 60 03 00	movq	$0x36004, %r12
0xffffffffc15c5022 <+18/0x12>:    53                 	pushq	%rbx
0xffffffffc15c5023 <+19/0x13>:    48 63 df           	movslq	%edi, %rbx
0xffffffffc15c5026 <+22/0x16>:    48 81 fb 00 20 00 00	cmpq	$0x2000, %rbx
0xffffffffc15c502d <+29/0x1d>:    73 19              	jae	0xffffffffc15c5048	; percpu_read+0x38
0xffffffffc15c502f <+31/0x1f>:    48 8b 04 dd 80 fc 24 a8	movq	-0x57db0380(, %rbx, 8), %rax
0xffffffffc15c5037 <+39/0x27>:    5b                 	popq	%rbx
0xffffffffc15c5038 <+40/0x28>:    42 8b 04 20        	movl	(%rax, %r12), %eax
0xffffffffc15c503c <+44/0x2c>:    41 5c              	popq	%r12
0xffffffffc15c503e <+46/0x2e>:    5d                 	popq	%rbp
0xffffffffc15c503f <+47/0x2f>:    31 f6              	xorl	%esi, %esi
0xffffffffc15c5041 <+49/0x31>:    31 ff              	xorl	%edi, %edi
0xffffffffc15c5043 <+51/0x33>:    c3                 	retq
0xffffffffc15c5044 <+52/0x34>:    cc                 	int3
0xffffffffc15c5045 <+53/0x35>:    cc                 	int3
0xffffffffc15c5046 <+54/0x36>:    cc                 	int3
0xffffffffc15c5047 <+55/0x37>:    cc                 	int3
0xffffffffc15c5048 <+56/0x38>:    48 89 de           	movq	%rbx, %rsi
0xffffffffc15c504b <+59/0x3b>:    48 c7 c7 a0 70 5c c1	movq	$18446744072658645152, %rdi
0xffffffffc15c5052 <+66/0x42>:    e8 d9 87 ac e5     	callq	0xffffffffa708d830	; __ubsan_handle_out_of_bounds+0x0 lib/ubsan.c:336
0xffffffffc15c5057 <+71/0x47>:    eb d6              	jmp	0xffffffffc15c502f	; percpu_read+0x1f

Disasm log link[1].

As we can see, per_cpu_ptr() calls __ubsan_handle_out_of_bounds() to check index range of percpu offset array. The callq insn is much slower than the addq insn.

Therefore, cpu == current_cpu + this_cpu_ptr() is much better than per_cpu_ptr().

Links:
[0] https://github.com/Asphaltt/kernel-module-fun/blob/master/percpu-ptr.c
[1] https://github.com/Asphaltt/kernel-module-fun/blob/master/percpu_ptr.md

Quoted 16 lines
Quoted 10 lines
kernel/bpf/hashtab.c +7 -1
        } else {
                u32 size = round_up(htab->map.value_size, 8);
-               int off = 0, cpu;
+               int off = 0;
+
+               if (map_flags & BPF_F_CPU) {
+                       copy_map_value_long(&htab->map, cpu != current_cpu ?
+                                           per_cpu_ptr(pptr, cpu) : this_cpu_ptr(pptr), value);
+                       return;
+               }

[...]

Quoted 3 lines
kernel/bpf/hashtab.c +0 -0
@@ -1806,10 +1834,17 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *map,
                        void __percpu *pptr;
 
Quoted 10 lines

kernel/bpf/hashtab.c __htab_map_lookup_and_delete_batch(struct bpf_map *map, · line 1837

Quoted 6 lines
kernel/bpf/hashtab.c +1 -4
                        pptr = htab_elem_get_ptr(l, map->key_size);
-                       for_each_possible_cpu(cpu) {
-                               copy_map_value_long(&htab->map, dst_val + off, per_cpu_ptr(pptr, cpu));
-                               check_and_init_map_value(&htab->map, dst_val + off);
-                               off += size;
+                       if (!do_delete && (elem_map_flags & BPF_F_CPU)) {

if do_delete is true we can't have BPF_F_CPU set, right? We checked that above, so why all these complications?

Ack.

This !do_delete is unnecessary. I'll drop it in next revision.

Quoted 55 lines
Quoted 44 lines
kernel/bpf/hashtab.c +18 -7
+                               cpu = elem_map_flags >> 32;
+                               copy_map_value_long(&htab->map, dst_val, per_cpu_ptr(pptr, cpu));
+                               check_and_init_map_value(&htab->map, dst_val);
+                       } else {
+                               for_each_possible_cpu(cpu) {
+                                       copy_map_value_long(&htab->map, dst_val + off,
+                                                           per_cpu_ptr(pptr, cpu));
+                                       check_and_init_map_value(&htab->map, dst_val + off);
+                                       off += size;
+                               }
                        }
                } else {
                        value = htab_elem_value(l, key_size);
@@ -2365,14 +2400,18 @@ static void *htab_lru_percpu_map_lookup_percpu_elem(struct bpf_map *map, void *k
        return NULL;
 }
 
-int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value)
+int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value, u64 map_flags)
 {
+       int ret, cpu, off = 0;
        struct htab_elem *l;
        void __percpu *pptr;
-       int ret = -ENOENT;
-       int cpu, off = 0;
        u32 size;
 
+       ret = bpf_map_check_cpu_flags(map_flags, false);
+       if (unlikely(ret))
+               return ret;
+       ret = -ENOENT;
+
        /* per_cpu areas are zero-filled and bpf programs can only
         * access 'value_size' of them, so copying rounded areas
         * will not leak any kernel data
@@ -2386,10 +2425,16 @@ int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value)
         * eviction heuristics when user space does a map walk.
         */
        pptr = htab_elem_get_ptr(l, map->key_size);
-       for_each_possible_cpu(cpu) {
-               copy_map_value_long(map, value + off, per_cpu_ptr(pptr, cpu));
-               check_and_init_map_value(map, value + off);
-               off += size;
+       if (map_flags & BPF_F_CPU) {

kernel/bpf/hashtab.c int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value) · line 2438

Quoted 10 lines
kernel/bpf/hashtab.c +9 -0
+               cpu = map_flags >> 32;
+               copy_map_value_long(map, value, per_cpu_ptr(pptr, cpu));
+               check_and_init_map_value(map, value);
+       } else {
+               for_each_possible_cpu(cpu) {
+                       copy_map_value_long(map, value + off, per_cpu_ptr(pptr, cpu));
+                       check_and_init_map_value(map, value + off);
+                       off += size;
+               }
        }
Quoted 9 lines

it feels like this whole logic of copying per-cpu value to/from user should be generic between all per-cpu maps, once we get that void __percpu * pointer, no? See if you can extract it as reusable helper (but, you know, without all the per-map type special casing, though it doesn't seem like you should need it, though I might be missing details, of course). One for bpf_percpu_copy_to_user() and another for bpf_percpu_copy_from_user(), which would take into account all these cpu flags?

Good idea.

Let's introduce these two helper functions.

Thanks,
Leon

[...]

author
#3 lore raw

Re: [PATCH bpf-next v3 1/6] bpf: Introduce internal check_map_flags helper function

In reply to Andrii Nakryiko
Quoted 35 lines · Andrii Nakryiko
Quoted 17 lines · Leon Hwang

It is to unify map flags checking for lookup, update, lookup_batch and update_batch.

Therefore, it will be convenient to check BPF_F_CPU flag in this helper function for them in next patch.

---
 kernel/bpf/syscall.c | 45 ++++++++++++++++++++++----------------------
 1 file changed, 22 insertions(+), 23 deletions(-)
kernel/bpf/syscall.c +0 -0
index 0fbfa8532c392..19f7f5de5e7dc 100644

kernel/bpf/syscall.c static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size) · line 1657

Quoted 4 lines
kernel/bpf/syscall.c +1 -0
@@ -1654,6 +1654,17 @@ static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size)
        return NULL;
 }
 
+static int check_map_flags(struct bpf_map *map, u64 flags, bool check_flag)

"check_map_flags" is super generically named... (and actually misleading, it's not map flags you are checking), so I think it should be something along the lines of "check_map_op_flag", i.e. map *operation* flag?

but also check_flag bool argument name for a function called "check flags" is so confusing... The idea here is whether we should enforce there is no *extra* flags beyond those common for all operations, right? So maybe call it "allow_extra_flags" or alternatively "strict_extra_flags", something suggesting that his is something in addition to common flags

Quoted 4 lines

alternatively, and perhaps best of all, I'd move that particular check outside and just maintain something like ARRAY_CREATE_FLAG_MASK for each operation, checking it explicitly where appropriate. WDYT?

Ack.

Following this idea, the checking functions will be

static inline bool bpf_map_check_op_flags(struct bpf_map *map, u64 flags, bool strict_extra_flags,
u64 extra_flags_mask)

{
        if (strict_extra_flags && ((u32)flags & extra_flags_mask))
                return -EINVAL;

        if ((flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))
                return -EINVAL;

        if (!(flags & BPF_F_CPU) && flags >> 32)
                return -EINVAL;

        if ((flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) && !bpf_map_supports_cpu_flags(map->map_type))
                return -EINVAL;

        return 0;
}

#define BPF_MAP_LOOKUP_ELEM_EXTRA_FLAGS_MASK (~(BPF_F_LOCK | BPF_F_CPU | BPF_F_ALL_CPUS))

static inline bool bpf_map_check_update_flags(struct bpf_map *map, u64 flags)

{
        return bpf_map_check_op_flags(map, flags, false, 0);
}

static inline bool bpf_map_check_lookup_flags(struct bpf_map *map, u64 flags)

{
        return bpf_map_check_op_flags(map, flags, true, BPF_MAP_LOOKUP_ELEM_EXTRA_FLAGS_MASK);
}

static inline bool bpf_map_check_batch_flags(struct bpf_map *map, u64 flags)

{
        return bpf_map_check_op_flags(map, flags, true, BPF_MAP_LOOKUP_ELEM_EXTRA_FLAGS_MASK);
}

These functions are better than check_map_flags().

Thanks,
Leon

Quoted 3 lines

pw-bot: cr

[...]

author
#14 lore raw

Re: [PATCH bpf-next v3 5/6] libbpf: Support BPF_F_CPU for percpu maps

In reply to Andrii Nakryiko
Quoted 2 lines · Andrii Nakryiko
Quoted 1 line · Leon Hwang

[...]

Quoted 14 lines

tools/lib/bpf/libbpf.c static int validate_map_op(const struct bpf_map *map, size_t key_sz, · line 10637

Quoted 9 lines
tools/lib/bpf/libbpf.c +5 -0
@@ -10630,6 +10630,19 @@ static int validate_map_op(const struct bpf_map *map, size_t key_sz,
                int num_cpu = libbpf_num_possible_cpus();
                size_t elem_sz = roundup(map->def.value_size, 8);
 
+               if (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) {
+                       if ((flags & BPF_F_CPU) && (flags & BPF_F_ALL_CPUS))
+                               return -EINVAL;
+                       if ((flags >> 32) >= num_cpu)
+                               return -ERANGE;

The idea of validate_map_op() is to make it easier for users to understand what's wrong with how they deal with the map, rather than just getting indiscriminate -EINVAL from the kernel.

Quoted 3 lines

Point being: add human-readable pr_warn() explanations for all the new conditions you are detecting, otherwise it's just meaningless.

Ack.

I'll add these pr_warn() explanations in next revision.

Quoted 29 lines
Quoted 29 lines
tools/lib/bpf/libbpf.c +10 -2
+                       if (value_sz != elem_sz) {
+                               pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %zu\n",
+                                       map->name, value_sz, elem_sz);
+                               return -EINVAL;
+                       }
+                       break;
+               }
+
                if (value_sz != num_cpu * elem_sz) {
                        pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %d * %zu = %zd\n",
                                map->name, value_sz, num_cpu, elem_sz, num_cpu * elem_sz);
@@ -10654,7 +10667,7 @@ int bpf_map__lookup_elem(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, value_sz, true);
+       err = validate_map_op(map, key_sz, value_sz, true, flags);
        if (err)
                return libbpf_err(err);
 
@@ -10667,7 +10680,7 @@ int bpf_map__update_elem(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, value_sz, true);
+       err = validate_map_op(map, key_sz, value_sz, true, flags);
        if (err)
                return libbpf_err(err);
 
Quoted 9 lines

tools/lib/bpf/libbpf.c int bpf_map__delete_elem(const struct bpf_map *map, · line 10695

Quoted 6 lines
tools/lib/bpf/libbpf.c +1 -1
@@ -10679,7 +10692,7 @@ int bpf_map__delete_elem(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, 0, false /* check_value_sz */);
+       err = validate_map_op(map, key_sz, 0, false /* check_value_sz */, 0);

hard-coded 0 instead of flags, why?

It should be flags.

However, delete op does not support the introduced cpu flags.

I think it's OK to use 0 here.

Quoted 12 lines

tools/lib/bpf/libbpf.c int bpf_map__lookup_and_delete_elem(const struct bpf_map *map, · line 10708

Quoted 9 lines
tools/lib/bpf/libbpf.c +1 -1
        if (err)
                return libbpf_err(err);
 
@@ -10692,7 +10705,7 @@ int bpf_map__lookup_and_delete_elem(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, value_sz, true);
+       err = validate_map_op(map, key_sz, value_sz, true, 0);

same about flags

Ack.

Quoted 26 lines
Quoted 26 lines
tools/lib/bpf/libbpf.c +1 -1
        if (err)
                return libbpf_err(err);
 
@@ -10704,7 +10717,7 @@ int bpf_map__get_next_key(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, 0, false /* check_value_sz */);
+       err = validate_map_op(map, key_sz, 0, false /* check_value_sz */, 0);
        if (err)
                return libbpf_err(err);
 
tools/lib/bpf/libbpf.h +0 -6
index 2e91148d9b44d..6a972a8d060c3 100644
@@ -1196,12 +1196,13 @@ LIBBPF_API struct bpf_map *bpf_map__inner_map(struct bpf_map *map);
  * @param key_sz size in bytes of key data, needs to match BPF map definition's **key_size**
  * @param value pointer to memory in which looked up value will be stored
  * @param value_sz size in byte of value data memory; it has to match BPF map
- * definition's **value_size**. For per-CPU BPF maps value size has to be
- * a product of BPF map value size and number of possible CPUs in the system
- * (could be fetched with **libbpf_num_possible_cpus()**). Note also that for
- * per-CPU values value size has to be aligned up to closest 8 bytes for
- * alignment reasons, so expected size is: `round_up(value_size, 8)
- * * libbpf_num_possible_cpus()`.
Quoted 9 lines

tools/lib/bpf/libbpf.h LIBBPF_API struct bpf_map *bpf_map__inner_map(struct bpf_map *map); · line 1204

Quoted 6 lines
tools/lib/bpf/libbpf.h +6 -0
+ * definition's **value_size**. For per-CPU BPF maps, value size can be
+ * definition's **value_size** if **BPF_F_CPU** or **BPF_F_ALL_CPUS** is
+ * specified in **flags**, otherwise a product of BPF map value size and number
+ * of possible CPUs in the system (could be fetched with
+ * **libbpf_num_possible_cpus()**). Note else that for per-CPU values value
+ * size has to be aligned up to closest 8 bytes for alignment reasons, so

nit: aligned up for alignment reasons... drop "for alignment reasons", I guess?

It is "for alignment reasons", because percpu maps use bpf_long_memcpy() to copy data.

static inline void bpf_long_memcpy(void *dst, const void *src, u32 size)

{
        const long *lsrc = src;
        long *ldst = dst;

        size /= sizeof(long);
        while (size--)
                data_race(*ldst++ = *lsrc++);
}

Thanks,
Leon

[...]

#15 lore raw

Re: [PATCH bpf-next v3 5/6] libbpf: Support BPF_F_CPU for percpu maps

Quoted 125 lines · Leon Hwang
Quoted 2 lines · Andrii Nakryiko
Quoted 1 line · Leon Hwang

[...]

Quoted 14 lines

tools/lib/bpf/libbpf.c static int validate_map_op(const struct bpf_map *map, size_t key_sz, · line 10637

Quoted 9 lines
tools/lib/bpf/libbpf.c +5 -0
@@ -10630,6 +10630,19 @@ static int validate_map_op(const struct bpf_map *map, size_t key_sz,
                int num_cpu = libbpf_num_possible_cpus();
                size_t elem_sz = roundup(map->def.value_size, 8);
 
+               if (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) {
+                       if ((flags & BPF_F_CPU) && (flags & BPF_F_ALL_CPUS))
+                               return -EINVAL;
+                       if ((flags >> 32) >= num_cpu)
+                               return -ERANGE;

The idea of validate_map_op() is to make it easier for users to understand what's wrong with how they deal with the map, rather than just getting indiscriminate -EINVAL from the kernel.

Quoted 3 lines

Point being: add human-readable pr_warn() explanations for all the new conditions you are detecting, otherwise it's just meaningless.

Ack.

I'll add these pr_warn() explanations in next revision.

Quoted 29 lines
Quoted 29 lines
tools/lib/bpf/libbpf.c +10 -2
+                       if (value_sz != elem_sz) {
+                               pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %zu\n",
+                                       map->name, value_sz, elem_sz);
+                               return -EINVAL;
+                       }
+                       break;
+               }
+
                if (value_sz != num_cpu * elem_sz) {
                        pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %d * %zu = %zd\n",
                                map->name, value_sz, num_cpu, elem_sz, num_cpu * elem_sz);
@@ -10654,7 +10667,7 @@ int bpf_map__lookup_elem(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, value_sz, true);
+       err = validate_map_op(map, key_sz, value_sz, true, flags);
        if (err)
                return libbpf_err(err);
 
@@ -10667,7 +10680,7 @@ int bpf_map__update_elem(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, value_sz, true);
+       err = validate_map_op(map, key_sz, value_sz, true, flags);
        if (err)
                return libbpf_err(err);
 
Quoted 9 lines

tools/lib/bpf/libbpf.c int bpf_map__delete_elem(const struct bpf_map *map, · line 10695

Quoted 6 lines
tools/lib/bpf/libbpf.c +1 -1
@@ -10679,7 +10692,7 @@ int bpf_map__delete_elem(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, 0, false /* check_value_sz */);
+       err = validate_map_op(map, key_sz, 0, false /* check_value_sz */, 0);

hard-coded 0 instead of flags, why?

It should be flags.

However, delete op does not support the introduced cpu flags.

I think it's OK to use 0 here.

Quoted 12 lines

tools/lib/bpf/libbpf.c int bpf_map__lookup_and_delete_elem(const struct bpf_map *map, · line 10708

Quoted 9 lines
tools/lib/bpf/libbpf.c +1 -1
        if (err)
                return libbpf_err(err);
 
@@ -10692,7 +10705,7 @@ int bpf_map__lookup_and_delete_elem(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, value_sz, true);
+       err = validate_map_op(map, key_sz, value_sz, true, 0);

same about flags

Ack.

Quoted 26 lines
Quoted 26 lines
tools/lib/bpf/libbpf.c +1 -1
        if (err)
                return libbpf_err(err);
 
@@ -10704,7 +10717,7 @@ int bpf_map__get_next_key(const struct bpf_map *map,
 {
        int err;
 
-       err = validate_map_op(map, key_sz, 0, false /* check_value_sz */);
+       err = validate_map_op(map, key_sz, 0, false /* check_value_sz */, 0);
        if (err)
                return libbpf_err(err);
 
tools/lib/bpf/libbpf.h +0 -6
index 2e91148d9b44d..6a972a8d060c3 100644
@@ -1196,12 +1196,13 @@ LIBBPF_API struct bpf_map *bpf_map__inner_map(struct bpf_map *map);
  * @param key_sz size in bytes of key data, needs to match BPF map definition's **key_size**
  * @param value pointer to memory in which looked up value will be stored
  * @param value_sz size in byte of value data memory; it has to match BPF map
- * definition's **value_size**. For per-CPU BPF maps value size has to be
- * a product of BPF map value size and number of possible CPUs in the system
- * (could be fetched with **libbpf_num_possible_cpus()**). Note also that for
- * per-CPU values value size has to be aligned up to closest 8 bytes for
- * alignment reasons, so expected size is: `round_up(value_size, 8)
- * * libbpf_num_possible_cpus()`.
Quoted 9 lines

tools/lib/bpf/libbpf.h LIBBPF_API struct bpf_map *bpf_map__inner_map(struct bpf_map *map); · line 1204

Quoted 6 lines
tools/lib/bpf/libbpf.h +6 -0
+ * definition's **value_size**. For per-CPU BPF maps, value size can be
+ * definition's **value_size** if **BPF_F_CPU** or **BPF_F_ALL_CPUS** is
+ * specified in **flags**, otherwise a product of BPF map value size and number
+ * of possible CPUs in the system (could be fetched with
+ * **libbpf_num_possible_cpus()**). Note else that for per-CPU values value
+ * size has to be aligned up to closest 8 bytes for alignment reasons, so

nit: aligned up for alignment reasons... drop "for alignment reasons", I guess?

Quoted 3 lines

It is "for alignment reasons", because percpu maps use bpf_long_memcpy() to copy data.

my complaint is just wording, if you say "aligned up" it implies that there is some alignment reason. So I'd just say "size has to be aligned up to 8 bytes." and be done with it

Quoted 14 lines

static inline void bpf_long_memcpy(void *dst, const void *src, u32 size)

{
        const long *lsrc = src;
        long *ldst = dst;

        size /= sizeof(long);
        while (size--)
                data_race(*ldst++ = *lsrc++);
}

Thanks,
Leon

[...]

#4 lore raw

Re: [PATCH bpf-next v3 1/6] bpf: Introduce internal check_map_flags helper function

In reply to Leon Hwang
Quoted 44 lines · Leon Hwang
Quoted 35 lines · Andrii Nakryiko
Quoted 17 lines · Leon Hwang

It is to unify map flags checking for lookup, update, lookup_batch and update_batch.

Therefore, it will be convenient to check BPF_F_CPU flag in this helper function for them in next patch.

---
 kernel/bpf/syscall.c | 45 ++++++++++++++++++++++----------------------
 1 file changed, 22 insertions(+), 23 deletions(-)
kernel/bpf/syscall.c +0 -0
index 0fbfa8532c392..19f7f5de5e7dc 100644

kernel/bpf/syscall.c static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size) · line 1657

Quoted 4 lines
kernel/bpf/syscall.c +1 -0
@@ -1654,6 +1654,17 @@ static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size)
        return NULL;
 }
 
+static int check_map_flags(struct bpf_map *map, u64 flags, bool check_flag)

"check_map_flags" is super generically named... (and actually misleading, it's not map flags you are checking), so I think it should be something along the lines of "check_map_op_flag", i.e. map *operation* flag?

but also check_flag bool argument name for a function called "check flags" is so confusing... The idea here is whether we should enforce there is no *extra* flags beyond those common for all operations, right? So maybe call it "allow_extra_flags" or alternatively "strict_extra_flags", something suggesting that his is something in addition to common flags

Quoted 4 lines

alternatively, and perhaps best of all, I'd move that particular check outside and just maintain something like ARRAY_CREATE_FLAG_MASK for each operation, checking it explicitly where appropriate. WDYT?

Ack.

Quoted 7 lines

Following this idea, the checking functions will be

static inline bool bpf_map_check_op_flags(struct bpf_map *map, u64 flags, bool strict_extra_flags,
u64 extra_flags_mask)

{
        if (strict_extra_flags && ((u32)flags & extra_flags_mask))
                return -EINVAL;

with this implementation strict_extra_flags argument is superficial, you can just pass extra_flags_mask == 0 to disable this check, effectively

Quoted 38 lines
        if ((flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))
                return -EINVAL;

        if (!(flags & BPF_F_CPU) && flags >> 32)
                return -EINVAL;

        if ((flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) && !bpf_map_supports_cpu_flags(map->map_type))
                return -EINVAL;

        return 0;
}

#define BPF_MAP_LOOKUP_ELEM_EXTRA_FLAGS_MASK (~(BPF_F_LOCK | BPF_F_CPU | BPF_F_ALL_CPUS))

static inline bool bpf_map_check_update_flags(struct bpf_map *map, u64 flags)

{
        return bpf_map_check_op_flags(map, flags, false, 0);
}

static inline bool bpf_map_check_lookup_flags(struct bpf_map *map, u64 flags)

{
        return bpf_map_check_op_flags(map, flags, true, BPF_MAP_LOOKUP_ELEM_EXTRA_FLAGS_MASK);
}

static inline bool bpf_map_check_batch_flags(struct bpf_map *map, u64 flags)

{
        return bpf_map_check_op_flags(map, flags, true, BPF_MAP_LOOKUP_ELEM_EXTRA_FLAGS_MASK);
}

These functions are better than check_map_flags().

Thanks,
Leon

Quoted 3 lines

pw-bot: cr

[...]