Skip to contentSkip to conversation

bpf: Introduce global percpu data

RFC PATCH In review Leon Hwang 3 participants 9 messages Jan 13, 2025 – Jan 17, 2025

Subsystems (5)
Kselftest BPF BPF selftests BPF core +1 more subsystems
lore
Share & tools
lore

Leon Hwang

author
#0 lore raw

cover bpf: Introduce global percpu data

This patch set introduces global per-CPU data, similar to commit 6316f78306c1 ("Merge branch 'support-global-data'"), to reduce restrictions in C for BPF programs.

With this enhancement, it becomes possible to define and use global per-CPU variables, much like the DEFINE_PER_CPU() macro in the kernel[0].

The idea stems from the bpflbr project[1], which itself was inspired by retsnoop[2]. During testing of bpflbr on the v6.6 kernel, two LBR (Last Branch Record) entries were observed related to the bpf_get_smp_processor_id() helper.

Since commit 1ae6921009e5 ("bpf: inline bpf_get_smp_processor_id() helper"), the bpf_get_smp_processor_id() helper has been inlined on x86_64, reducing the overhead and consequently minimizing these two LBR records.

However, the introduction of global per-CPU data offers a more robust solution. By leveraging the percpu_array map and percpu instructions, global per-CPU data can be implemented intrinsically.

This feature also facilitates sharing per-CPU information between tail callers and callees or between freplace callers and callees through a shared global per-CPU variable. Previously, this was achieved using a 1-entry percpu map, which this patch set aims to improve upon.

Links:
[0] https://github.com/torvalds/linux/blob/fbfd64d25c7af3b8695201ebc85efe90be28c5a3/include/linux/percpu-defs.h#L114
[1] https://github.com/Asphaltt/bpflbr
[2] https://github.com/anakryiko/retsnoop

Leon Hwang (2):
bpf: Introduce global percpu data selftests/bpf: Add a case to test global percpu data

 kernel/bpf/arraymap.c                         |  39 +++++-
 kernel/bpf/verifier.c                         |  45 +++++++
 tools/lib/bpf/libbpf.c                        | 112 ++++++++++++++----
 .../bpf/prog_tests/global_data_init.c         |  83 ++++++++++++-
 .../bpf/progs/test_global_percpu_data.c       |  21 ++++
 5 files changed, 274 insertions(+), 26 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c
Signature · 1 line
2.47.1
author
#1 lore raw

RFC PATCH 1/2 bpf: Introduce global percpu data

This patch introduces global per-CPU data, inspired by commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global per-CPU variables in BPF, similar to the DEFINE_PER_CPU() macro in the kernel[0].

For example, in BPF, it is able to define a global per-CPU variable like this:

int percpu_data SEC(".data..percpu");

With this patch, tools like retsnoop[1] and bpflbr[2] can simplify their BPF code for handling LBRs. The code can be updated from

static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");

to

static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".data..percpu.lbrs");

This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper.

Additionally, by reusing global per-CPU variables, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to using percpu_array maps.

Links:
[0] https://github.com/torvalds/linux/blob/fbfd64d25c7af3b8695201ebc85efe90be28c5a3/include/linux/percpu-defs.h#L114
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/Asphaltt/bpflbr

---
 kernel/bpf/arraymap.c  |  39 +++++++++++++-
 kernel/bpf/verifier.c  |  45 +++++++++++++++++
 tools/lib/bpf/libbpf.c | 112 ++++++++++++++++++++++++++++++++---------
 3 files changed, 171 insertions(+), 25 deletions(-)
kernel/bpf/arraymap.c +38 -1
index eb28c0f219ee4..f8c60d8331975 100644
@@ -249,6 +249,40 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)
 	return this_cpu_ptr(array->pptrs[index & array->index_mask]);
 }
 
+static int percpu_array_map_direct_value_addr(const struct bpf_map *map,
+					      u64 *imm, u32 off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (off >= map->value_size)
+		return -EINVAL;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+
+	*imm = (u64) array->pptrs[0];
+	return 0;
+}
+
+static int percpu_array_map_direct_value_meta(const struct bpf_map *map,
+					      u64 imm, u32 *off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	u64 base = (u64) array->pptrs[0];
+	u64 range = array->elem_size;
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (imm < base || imm >= base + range)
+		return -ENOENT;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+
+	*off = imm - base;
+	return 0;
+}
+
 /* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */
 static int percpu_array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
 {
@@ -534,7 +568,8 @@ static int array_map_check_btf(const struct bpf_map *map,
 
 	/* One exception for keyless BTF: .bss/.data/.rodata map */
 	if (btf_type_is_void(key_type)) {
-		if (map->map_type != BPF_MAP_TYPE_ARRAY ||
+		if ((map->map_type != BPF_MAP_TYPE_ARRAY &&
+		     map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) ||
 		    map->max_entries != 1)
 			return -EINVAL;
 
@@ -815,6 +850,8 @@ const struct bpf_map_ops percpu_array_map_ops = {
 	.map_get_next_key = array_map_get_next_key,
 	.map_lookup_elem = percpu_array_map_lookup_elem,
 	.map_gen_lookup = percpu_array_map_gen_lookup,
+	.map_direct_value_addr = percpu_array_map_direct_value_addr,
+	.map_direct_value_meta = percpu_array_map_direct_value_meta,
 	.map_update_elem = array_map_update_elem,
 	.map_delete_elem = array_map_delete_elem,
 	.map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,
kernel/bpf/verifier.c +45 -0

75 lines not on this page · raw message

tools/lib/bpf/libbpf.c +88 -24

197 lines not on this page · raw message

Signature · 1 line
2.47.1
#2 lore raw
Quoted 32 lines · Leon Hwang

This patch introduces global per-CPU data, inspired by commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global per-CPU variables in BPF, similar to the DEFINE_PER_CPU() macro in the kernel[0].

For example, in BPF, it is able to define a global per-CPU variable like this:

int percpu_data SEC(".data..percpu");

With this patch, tools like retsnoop[1] and bpflbr[2] can simplify their BPF code for handling LBRs. The code can be updated from

static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");

to

static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".data..percpu.lbrs");

This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper.

Additionally, by reusing global per-CPU variables, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to using percpu_array maps.

Links:
[0] https://github.com/torvalds/linux/blob/fbfd64d25c7af3b8695201ebc85efe90be28c5a3/include/linux/percpu-defs.h#L114
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/Asphaltt/bpflbr

Quoted 7 lines
---
 kernel/bpf/arraymap.c  |  39 +++++++++++++-
 kernel/bpf/verifier.c  |  45 +++++++++++++++++
 tools/lib/bpf/libbpf.c | 112 ++++++++++++++++++++++++++++++++---------
 3 files changed, 171 insertions(+), 25 deletions(-)

So I think the feature overall makes sense, but we need to think through at least libbpf's side of things some more. Unlike .data, per-cpu .data section is not mmapable, and so that has implication on BPF skeleton and we should make sure all that makes sense on BPF skeleton side. In that sense, per-cpu global data is more akin to struct_ops initialization image, which can be accessed by user before skeleton is loaded to initialize the image.

There are a few things to consider. What's the BPF skeleton interface? Do we expose it as single struct and use that struct as initial image for each CPU (which means user won't be able to initialize different CPU data differently, at least not through BPF skeleton facilities)? Or do we expose this as an array of structs and let user set each CPU data independently?

I feel like keeping it simple and having one image for all CPUs would cover most cases. And users can still access the underlying PERCPU_ARRAY map if they need more control.

But either way, we need tests for skeleton, making sure we NULL-out this per-cpu global data, but take it into account before the load.

Also, this huge calloc for possible CPUs, I'd like to avoid it altogether for the (probably very common) zero-initialized case.

So in short, needs a bit of iteration to figure out all the interfacing issues, but makes sense overall. See some more low-level remarks below.

pw-bot: cr

[...]

Quoted 14 lines
kernel/bpf/arraymap.c +2 -0
@@ -815,6 +850,8 @@ const struct bpf_map_ops percpu_array_map_ops = {
        .map_get_next_key = array_map_get_next_key,
        .map_lookup_elem = percpu_array_map_lookup_elem,
        .map_gen_lookup = percpu_array_map_gen_lookup,
+       .map_direct_value_addr = percpu_array_map_direct_value_addr,
+       .map_direct_value_meta = percpu_array_map_direct_value_meta,
        .map_update_elem = array_map_update_elem,
        .map_delete_elem = array_map_delete_elem,
        .map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,
kernel/bpf/verifier.c +0 -0
index b8ca227c78af1..94ce02a48ddc1 100644

kernel/bpf/verifier.c static int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val, · line 6813

Quoted 5 lines
kernel/bpf/verifier.c +2 -0
@@ -6809,6 +6809,8 @@ static int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
        u64 addr;
        int err;
 
+       if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+               return -EINVAL;

I'd invert the condition and reject anything by BPF_MAP_TYPE_ARRAY. I don't see how any other possible map type can support this magic that we do with ARRAY. So to be on the safe side, let's just reject everything that's non-ARRAY (here and elsewhere)?

Quoted 3 lines
kernel/bpf/verifier.c +0 -0
        err = map->ops->map_direct_value_addr(map, &addr, off);
        if (err)
                return err;

[...]

tools/lib/bpf/libbpf.c

Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
index 6c262d0152f81..881174f4f90a4 100644

definitely let's split out libbpf changes from kernel-side changes

tools/lib/bpf/libbpf.c struct bpf_struct_ops { · line 519

Quoted 5 lines
tools/lib/bpf/libbpf.c +1 -0
@@ -516,6 +516,7 @@ struct bpf_struct_ops {
 };
 
 #define DATA_SEC ".data"
+#define PERCPU_DATA_SEC ".data..percpu"

I don't like this prefix, even if that's what we have in the kernel. Something like just ".percpu" or ".percpu_data" or ".data_percpu" is better, IMO.

Quoted 13 lines
tools/lib/bpf/libbpf.c +2 -0
 #define BSS_SEC ".bss"
 #define RODATA_SEC ".rodata"
 #define KCONFIG_SEC ".kconfig"
@@ -562,6 +563,8 @@ struct bpf_map {
        __u32 btf_value_type_id;
        __u32 btf_vmlinux_value_type_id;
        enum libbpf_map_type libbpf_type;
+       int num_cpus;
+       void *data;
        void *mmaped;
        struct bpf_struct_ops *st_ops;
        struct bpf_map *inner_map;

tools/lib/bpf/libbpf.c static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map) · line 1932

Quoted 7 lines
tools/lib/bpf/libbpf.c +4 -0
@@ -1923,11 +1926,35 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map)
        return false;
 }
 
+static bool map_is_percpu_data(struct bpf_map *map)
+{
+       return str_has_pfx(map->real_name, PERCPU_DATA_SEC);
+}

we have enum libbpf_map_type which is used to distinguish BSS vs RODATA vs DATA vs others, let's stick to that

Quoted 39 lines
tools/lib/bpf/libbpf.c +22 -1
+
+static void map_copy_data(struct bpf_map *map, const void *data)
+{
+       bool is_percpu_data = map_is_percpu_data(map);
+       size_t data_sz = map->def.value_size;
+       size_t elem_sz = roundup(data_sz, 8);
+       int i;
+
+       if (!data)
+               return;
+
+       if (!is_percpu_data)
+               memcpy(map->mmaped, data, data_sz);
+       else
+               for (i = 0; i < map->num_cpus; i++)
+                       memcpy(map->data + i*elem_sz, data, data_sz);
+}
+
 static int
 bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
                              const char *real_name, int sec_idx, void *data, size_t data_sz)
 {
+       bool is_percpu_data = str_has_pfx(real_name, PERCPU_DATA_SEC);
        struct bpf_map_def *def;
+       const char *data_desc;
        struct bpf_map *map;
        size_t mmap_sz;
        int err;
@@ -1948,7 +1975,8 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
        }
 
        def = &map->def;
-       def->type = BPF_MAP_TYPE_ARRAY;
+       def->type = is_percpu_data ? BPF_MAP_TYPE_PERCPU_ARRAY
+                                  : BPF_MAP_TYPE_ARRAY;
        def->key_size = sizeof(int);
        def->value_size = data_sz;
        def->max_entries = 1;

tools/lib/bpf/libbpf.c bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, · line 1989

Quoted 6 lines
tools/lib/bpf/libbpf.c +1 -2
@@ -1958,29 +1986,57 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
        /* failures are fine because of maps like .rodata.str1.1 */
        (void) map_fill_btf_type_info(obj, map);
 
-       if (map_is_mmapable(obj, map))
-               def->map_flags |= BPF_F_MMAPABLE;
+       data_desc = is_percpu_data ? "percpu " : "";

nit: just inline the logic in that single place you use it

tools/lib/bpf/libbpf.c bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, · line 1997

Quoted 10 lines
tools/lib/bpf/libbpf.c +7 -2
+       pr_debug("map '%s' (global %sdata): at sec_idx %d, offset %zu, flags %x.\n",
+                map->name, data_desc, map->sec_idx, map->sec_offset,
+                def->map_flags);
 
-       pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n",
-                map->name, map->sec_idx, map->sec_offset, def->map_flags);
+       if (is_percpu_data) {
+               map->num_cpus = libbpf_num_possible_cpus();
+               if (map->num_cpus < 0) {
+                       err = errno;

map->num_cpus should have actual error value already, avoid using errno unnecessarily

Quoted 10 lines
tools/lib/bpf/libbpf.c +3 -6
+                       pr_warn("failed to get possible cpus\n");
+                       goto free_name;
+               }
 
-       mmap_sz = bpf_map_mmap_sz(map);
-       map->mmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE,
-                          MAP_SHARED | MAP_ANONYMOUS, -1, 0);
-       if (map->mmaped == MAP_FAILED) {
-               err = -errno;
-               map->mmaped = NULL;

tools/lib/bpf/libbpf.c bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, · line 2006

Quoted 10 lines
tools/lib/bpf/libbpf.c +5 -5
-               pr_warn("failed to alloc map '%s' content buffer: %s\n", map->name, errstr(err));
-               zfree(&map->real_name);
-               zfree(&map->name);
-               return err;
-       }
+               map->data = calloc(map->num_cpus, roundup(data_sz, 8));
+               if (!map->data) {
+                       err = -ENOMEM;
+                       pr_warn("failed to alloc percpu map '%s' content buffer: %s\n",
+                               map->name, errstr(err));

please stick to establish reporting style (i.e., for map operations we always use "map '%s': " prefix)

Quoted 2 lines
tools/lib/bpf/libbpf.c +2 -0
+                       goto free_name;
+               }

[...]

tools/lib/bpf/libbpf.c void *bpf_map__initial_value(const struct bpf_map *map, size_t *psize) · line 10437

Quoted 6 lines
tools/lib/bpf/libbpf.c +1 -1
@@ -10370,7 +10434,7 @@ void *bpf_map__initial_value(const struct bpf_map *map, size_t *psize)
                return map->st_ops->data;
        }
 
-       if (!map->mmaped)
+       if ((!map->mmaped && !map->data))

nit: why unnecessary extra () ?

Quoted 15 lines
tools/lib/bpf/libbpf.c +1 -2
                return NULL;
 
        if (map->def.type == BPF_MAP_TYPE_ARENA)
@@ -10378,7 +10442,7 @@ void *bpf_map__initial_value(const struct bpf_map *map, size_t *psize)
        else
                *psize = map->def.value_size;
 
-       return map->mmaped;
+       return map->def.type == BPF_MAP_TYPE_PERCPU_ARRAY ? map->data : map->mmaped;
 }
 
 bool bpf_map__is_internal(const struct bpf_map *map)
--

2.47.1

author
#3 lore raw

Hi,

Quoted 56 lines · Andrii Nakryiko
Quoted 32 lines · Leon Hwang

This patch introduces global per-CPU data, inspired by commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global per-CPU variables in BPF, similar to the DEFINE_PER_CPU() macro in the kernel[0].

For example, in BPF, it is able to define a global per-CPU variable like this:

int percpu_data SEC(".data..percpu");

With this patch, tools like retsnoop[1] and bpflbr[2] can simplify their BPF code for handling LBRs. The code can be updated from

static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");

to

static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".data..percpu.lbrs");

This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper.

Additionally, by reusing global per-CPU variables, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to using percpu_array maps.

Links:
[0] https://github.com/torvalds/linux/blob/fbfd64d25c7af3b8695201ebc85efe90be28c5a3/include/linux/percpu-defs.h#L114
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/Asphaltt/bpflbr

Quoted 7 lines
---
 kernel/bpf/arraymap.c  |  39 +++++++++++++-
 kernel/bpf/verifier.c  |  45 +++++++++++++++++
 tools/lib/bpf/libbpf.c | 112 ++++++++++++++++++++++++++++++++---------
 3 files changed, 171 insertions(+), 25 deletions(-)

So I think the feature overall makes sense, but we need to think through at least libbpf's side of things some more. Unlike .data, per-cpu .data section is not mmapable, and so that has implication on BPF skeleton and we should make sure all that makes sense on BPF skeleton side. In that sense, per-cpu global data is more akin to struct_ops initialization image, which can be accessed by user before skeleton is loaded to initialize the image.

There are a few things to consider. What's the BPF skeleton interface? Do we expose it as single struct and use that struct as initial image for each CPU (which means user won't be able to initialize different CPU data differently, at least not through BPF skeleton facilities)? Or do we expose this as an array of structs and let user set each CPU data independently?

Quoted 3 lines

I feel like keeping it simple and having one image for all CPUs would cover most cases. And users can still access the underlying PERCPU_ARRAY map if they need more control.

Agree. It is necessary to keep it simple.

Quoted 6 lines

But either way, we need tests for skeleton, making sure we NULL-out this per-cpu global data, but take it into account before the load.

Also, this huge calloc for possible CPUs, I'd like to avoid it altogether for the (probably very common) zero-initialized case.

Ack.

Quoted 5 lines

So in short, needs a bit of iteration to figure out all the interfacing issues, but makes sense overall. See some more low-level remarks below.

It is challenging to figure out them. I'll do my best to achieve it.

Quoted 22 lines

pw-bot: cr

[...]

Quoted 17 lines
kernel/bpf/arraymap.c +2 -0
@@ -815,6 +850,8 @@ const struct bpf_map_ops percpu_array_map_ops = {
        .map_get_next_key = array_map_get_next_key,
        .map_lookup_elem = percpu_array_map_lookup_elem,
        .map_gen_lookup = percpu_array_map_gen_lookup,
+       .map_direct_value_addr = percpu_array_map_direct_value_addr,
+       .map_direct_value_meta = percpu_array_map_direct_value_meta,
        .map_update_elem = array_map_update_elem,
        .map_delete_elem = array_map_delete_elem,
        .map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,
kernel/bpf/verifier.c +0 -0
index b8ca227c78af1..94ce02a48ddc1 100644
@@ -6809,6 +6809,8 @@ static int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
        u64 addr;
        int err;
 
Quoted 7 lines

kernel/bpf/verifier.c static int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val, · line 6813

Quoted 2 lines
kernel/bpf/verifier.c +2 -0
+       if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+               return -EINVAL;

I'd invert the condition and reject anything by BPF_MAP_TYPE_ARRAY. I don't see how any other possible map type can support this magic that we do with ARRAY. So to be on the safe side, let's just reject everything that's non-ARRAY (here and elsewhere)?

Ack.

Quoted 5 lines

kernel/bpf/verifier.c static int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val, · line 6816

Quoted 4 lines
kernel/bpf/verifier.c +0 -0
        err = map->ops->map_direct_value_addr(map, &addr, off);
        if (err)
                return err;
Quoted 8 lines

[...]

tools/lib/bpf/libbpf.c

Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
index 6c262d0152f81..881174f4f90a4 100644

definitely let's split out libbpf changes from kernel-side changes

Ack.

Quoted 10 lines

tools/lib/bpf/libbpf.c struct bpf_struct_ops { · line 519

Quoted 6 lines
tools/lib/bpf/libbpf.c +1 -0
@@ -516,6 +516,7 @@ struct bpf_struct_ops {
 };
 
 #define DATA_SEC ".data"
+#define PERCPU_DATA_SEC ".data..percpu"

I don't like this prefix, even if that's what we have in the kernel. Something like just ".percpu" or ".percpu_data" or ".data_percpu" is better, IMO.

I tested ".percpu". It is OK to use it. But we have to update "bpftool gen" too, which relies on these section names.

Is it better to keep ".data" prefix, like ".data.percpu", ".data_percpu"? Can keeping ".data" prefix reduce some works on bpftool, go-ebpf and akin bpf loaders?

Quoted 17 lines
Quoted 17 lines
tools/lib/bpf/libbpf.c +2 -0
 #define BSS_SEC ".bss"
 #define RODATA_SEC ".rodata"
 #define KCONFIG_SEC ".kconfig"
@@ -562,6 +563,8 @@ struct bpf_map {
        __u32 btf_value_type_id;
        __u32 btf_vmlinux_value_type_id;
        enum libbpf_map_type libbpf_type;
+       int num_cpus;
+       void *data;
        void *mmaped;
        struct bpf_struct_ops *st_ops;
        struct bpf_map *inner_map;
@@ -1923,11 +1926,35 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map)
        return false;
 }
 
Quoted 8 lines

tools/lib/bpf/libbpf.c static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map) · line 1932

Quoted 4 lines
tools/lib/bpf/libbpf.c +4 -0
+static bool map_is_percpu_data(struct bpf_map *map)
+{
+       return str_has_pfx(map->real_name, PERCPU_DATA_SEC);
+}

we have enum libbpf_map_type which is used to distinguish BSS vs RODATA vs DATA vs others, let's stick to that

Ack.

Quoted 42 lines
Quoted 42 lines
tools/lib/bpf/libbpf.c +22 -1
+
+static void map_copy_data(struct bpf_map *map, const void *data)
+{
+       bool is_percpu_data = map_is_percpu_data(map);
+       size_t data_sz = map->def.value_size;
+       size_t elem_sz = roundup(data_sz, 8);
+       int i;
+
+       if (!data)
+               return;
+
+       if (!is_percpu_data)
+               memcpy(map->mmaped, data, data_sz);
+       else
+               for (i = 0; i < map->num_cpus; i++)
+                       memcpy(map->data + i*elem_sz, data, data_sz);
+}
+
 static int
 bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
                              const char *real_name, int sec_idx, void *data, size_t data_sz)
 {
+       bool is_percpu_data = str_has_pfx(real_name, PERCPU_DATA_SEC);
        struct bpf_map_def *def;
+       const char *data_desc;
        struct bpf_map *map;
        size_t mmap_sz;
        int err;
@@ -1948,7 +1975,8 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
        }
 
        def = &map->def;
-       def->type = BPF_MAP_TYPE_ARRAY;
+       def->type = is_percpu_data ? BPF_MAP_TYPE_PERCPU_ARRAY
+                                  : BPF_MAP_TYPE_ARRAY;
        def->key_size = sizeof(int);
        def->value_size = data_sz;
        def->max_entries = 1;
@@ -1958,29 +1986,57 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
        /* failures are fine because of maps like .rodata.str1.1 */
        (void) map_fill_btf_type_info(obj, map);
 
Quoted 6 lines

tools/lib/bpf/libbpf.c bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, · line 1989

Quoted 3 lines
tools/lib/bpf/libbpf.c +1 -2
-       if (map_is_mmapable(obj, map))
-               def->map_flags |= BPF_F_MMAPABLE;
+       data_desc = is_percpu_data ? "percpu " : "";

nit: just inline the logic in that single place you use it

Ack.

Quoted 11 lines

tools/lib/bpf/libbpf.c bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, · line 1997

Quoted 10 lines
tools/lib/bpf/libbpf.c +7 -2
+       pr_debug("map '%s' (global %sdata): at sec_idx %d, offset %zu, flags %x.\n",
+                map->name, data_desc, map->sec_idx, map->sec_offset,
+                def->map_flags);
 
-       pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n",
-                map->name, map->sec_idx, map->sec_offset, def->map_flags);
+       if (is_percpu_data) {
+               map->num_cpus = libbpf_num_possible_cpus();
+               if (map->num_cpus < 0) {
+                       err = errno;
Quoted 3 lines

map->num_cpus should have actual error value already, avoid using errno unnecessarily

Ack.

Quoted 21 lines
Quoted 10 lines
tools/lib/bpf/libbpf.c +3 -6
+                       pr_warn("failed to get possible cpus\n");
+                       goto free_name;
+               }
 
-       mmap_sz = bpf_map_mmap_sz(map);
-       map->mmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE,
-                          MAP_SHARED | MAP_ANONYMOUS, -1, 0);
-       if (map->mmaped == MAP_FAILED) {
-               err = -errno;
-               map->mmaped = NULL;

tools/lib/bpf/libbpf.c bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, · line 2006

Quoted 10 lines
tools/lib/bpf/libbpf.c +5 -5
-               pr_warn("failed to alloc map '%s' content buffer: %s\n", map->name, errstr(err));
-               zfree(&map->real_name);
-               zfree(&map->name);
-               return err;
-       }
+               map->data = calloc(map->num_cpus, roundup(data_sz, 8));
+               if (!map->data) {
+                       err = -ENOMEM;
+                       pr_warn("failed to alloc percpu map '%s' content buffer: %s\n",
+                               map->name, errstr(err));
Quoted 3 lines

please stick to establish reporting style (i.e., for map operations we always use "map '%s': " prefix)

Ack.

Quoted 5 lines
Quoted 2 lines
tools/lib/bpf/libbpf.c +2 -0
+                       goto free_name;
+               }

[...]

Quoted 9 lines

tools/lib/bpf/libbpf.c void *bpf_map__initial_value(const struct bpf_map *map, size_t *psize) · line 10437

Quoted 6 lines
tools/lib/bpf/libbpf.c +1 -1
@@ -10370,7 +10434,7 @@ void *bpf_map__initial_value(const struct bpf_map *map, size_t *psize)
                return map->st_ops->data;
        }
 
-       if (!map->mmaped)
+       if ((!map->mmaped && !map->data))

nit: why unnecessary extra () ?

My bad. I'll remove it.

Quoted 8 lines
Quoted 8 lines
tools/lib/bpf/libbpf.c +0 -0
                return NULL;
 
        if (map->def.type == BPF_MAP_TYPE_ARENA)
@@ -10378,7 +10442,7 @@ void *bpf_map__initial_value(const struct bpf_map *map, size_t *psize)
        else
                *psize = map->def.value_size;
 
Quoted 8 lines
Quoted 8 lines
tools/lib/bpf/libbpf.c +1 -2
-       return map->mmaped;
+       return map->def.type == BPF_MAP_TYPE_PERCPU_ARRAY ? map->data : map->mmaped;
 }
 
 bool bpf_map__is_internal(const struct bpf_map *map)
--

2.47.1

Thanks,
Leon

#4 lore raw
Quoted 86 lines · Leon Hwang

Hi,

Quoted 56 lines · Andrii Nakryiko
Quoted 32 lines · Leon Hwang

This patch introduces global per-CPU data, inspired by commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global per-CPU variables in BPF, similar to the DEFINE_PER_CPU() macro in the kernel[0].

For example, in BPF, it is able to define a global per-CPU variable like this:

int percpu_data SEC(".data..percpu");

With this patch, tools like retsnoop[1] and bpflbr[2] can simplify their BPF code for handling LBRs. The code can be updated from

static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");

to

static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".data..percpu.lbrs");

This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper.

Additionally, by reusing global per-CPU variables, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to using percpu_array maps.

Links:
[0] https://github.com/torvalds/linux/blob/fbfd64d25c7af3b8695201ebc85efe90be28c5a3/include/linux/percpu-defs.h#L114
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/Asphaltt/bpflbr

Quoted 7 lines
---
 kernel/bpf/arraymap.c  |  39 +++++++++++++-
 kernel/bpf/verifier.c  |  45 +++++++++++++++++
 tools/lib/bpf/libbpf.c | 112 ++++++++++++++++++++++++++++++++---------
 3 files changed, 171 insertions(+), 25 deletions(-)

So I think the feature overall makes sense, but we need to think through at least libbpf's side of things some more. Unlike .data, per-cpu .data section is not mmapable, and so that has implication on BPF skeleton and we should make sure all that makes sense on BPF skeleton side. In that sense, per-cpu global data is more akin to struct_ops initialization image, which can be accessed by user before skeleton is loaded to initialize the image.

There are a few things to consider. What's the BPF skeleton interface? Do we expose it as single struct and use that struct as initial image for each CPU (which means user won't be able to initialize different CPU data differently, at least not through BPF skeleton facilities)? Or do we expose this as an array of structs and let user set each CPU data independently?

Quoted 3 lines

I feel like keeping it simple and having one image for all CPUs would cover most cases. And users can still access the underlying PERCPU_ARRAY map if they need more control.

Agree. It is necessary to keep it simple.

Quoted 6 lines

But either way, we need tests for skeleton, making sure we NULL-out this per-cpu global data, but take it into account before the load.

Also, this huge calloc for possible CPUs, I'd like to avoid it altogether for the (probably very common) zero-initialized case.

Ack.

Quoted 5 lines

So in short, needs a bit of iteration to figure out all the interfacing issues, but makes sense overall. See some more low-level remarks below.

It is challenging to figure out them. I'll do my best to achieve it.

Quoted 3 lines

pw-bot: cr

[...]

Quoted 11 lines
Quoted 10 lines

tools/lib/bpf/libbpf.c struct bpf_struct_ops { · line 519

Quoted 6 lines
tools/lib/bpf/libbpf.c +1 -0
@@ -516,6 +516,7 @@ struct bpf_struct_ops {
 };
 
 #define DATA_SEC ".data"
+#define PERCPU_DATA_SEC ".data..percpu"

I don't like this prefix, even if that's what we have in the kernel. Something like just ".percpu" or ".percpu_data" or ".data_percpu" is better, IMO.

Quoted 6 lines

I tested ".percpu". It is OK to use it. But we have to update "bpftool gen" too, which relies on these section names.

Is it better to keep ".data" prefix, like ".data.percpu", ".data_percpu"? Can keeping ".data" prefix reduce some works on bpftool, go-ebpf and akin bpf loaders?

It's literally two lines of code in gen.c, and that should actually be a common array of known prefixes. Even if someone uses this new .percpu section with old bpftool nothing will break, they just won't have structure representing the initial per-CPU image. They will still have the generic map pointer in the skeleton. So I think this is acceptable.

I'd definitely go with a simple and less error-prone ".percpu" prefix.

[...]

author
#5 lore raw
Quoted 7 lines · Andrii Nakryiko
Quoted 6 lines · Leon Hwang

Hi,

Quoted 2 lines · Andrii Nakryiko
Quoted 1 line · Leon Hwang

[...]

Quoted 63 lines
Quoted 42 lines
Quoted 16 lines

So I think the feature overall makes sense, but we need to think through at least libbpf's side of things some more. Unlike .data, per-cpu .data section is not mmapable, and so that has implication on BPF skeleton and we should make sure all that makes sense on BPF skeleton side. In that sense, per-cpu global data is more akin to struct_ops initialization image, which can be accessed by user before skeleton is loaded to initialize the image.

There are a few things to consider. What's the BPF skeleton interface? Do we expose it as single struct and use that struct as initial image for each CPU (which means user won't be able to initialize different CPU data differently, at least not through BPF skeleton facilities)? Or do we expose this as an array of structs and let user set each CPU data independently?

Quoted 3 lines

I feel like keeping it simple and having one image for all CPUs would cover most cases. And users can still access the underlying PERCPU_ARRAY map if they need more control.

Agree. It is necessary to keep it simple.

Quoted 6 lines

But either way, we need tests for skeleton, making sure we NULL-out this per-cpu global data, but take it into account before the load.

Also, this huge calloc for possible CPUs, I'd like to avoid it altogether for the (probably very common) zero-initialized case.

Ack.

Quoted 5 lines

So in short, needs a bit of iteration to figure out all the interfacing issues, but makes sense overall. See some more low-level remarks below.

It is challenging to figure out them. I'll do my best to achieve it.

Quoted 3 lines

pw-bot: cr

[...]

Quoted 11 lines
Quoted 10 lines

tools/lib/bpf/libbpf.c struct bpf_struct_ops { · line 519

Quoted 6 lines
tools/lib/bpf/libbpf.c +1 -0
@@ -516,6 +516,7 @@ struct bpf_struct_ops {
 };
 
 #define DATA_SEC ".data"
+#define PERCPU_DATA_SEC ".data..percpu"

I don't like this prefix, even if that's what we have in the kernel. Something like just ".percpu" or ".percpu_data" or ".data_percpu" is better, IMO.

Quoted 6 lines

I tested ".percpu". It is OK to use it. But we have to update "bpftool gen" too, which relies on these section names.

Is it better to keep ".data" prefix, like ".data.percpu", ".data_percpu"? Can keeping ".data" prefix reduce some works on bpftool, go-ebpf and akin bpf loaders?

Quoted 9 lines

It's literally two lines of code in gen.c, and that should actually be a common array of known prefixes. Even if someone uses this new .percpu section with old bpftool nothing will break, they just won't have structure representing the initial per-CPU image. They will still have the generic map pointer in the skeleton. So I think this is acceptable.

I'd definitely go with a simple and less error-prone ".percpu" prefix.

Being simple and less error-prone is indeed important. Let's proceed with the ".percpu" prefix as suggested.

Thanks,
Leon

author
#6 lore raw

RFC PATCH 2/2 selftests/bpf: Add a case to test global percpu data

In reply to Leon Hwang

If the arch, like s390x, does not support percpu insn, this case won't test global percpu data by checking -EOPNOTSUPP when load prog.

cd tools/testing/selftests/bpf; ./test_progs -t global_percpu_data
124 global_percpu_data_init:OK
Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED

---
 .../bpf/prog_tests/global_data_init.c         | 83 ++++++++++++++++++-
 .../bpf/progs/test_global_percpu_data.c       | 21 +++++
 2 files changed, 103 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c
tools/testing/selftests/bpf/prog_tests/global_data_init.c +82 -1
index 8466332d7406f..634ff531bf5a4 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 #include <test_progs.h>
+#include "test_global_percpu_data.skel.h"
 
 void test_global_data_init(void)
 {
@@ -8,7 +9,7 @@ void test_global_data_init(void)
 	__u8 *buff = NULL, *newval = NULL;
 	struct bpf_object *obj;
 	struct bpf_map *map;
-        __u32 duration = 0;
+	__u32 duration = 0;
 	size_t sz;
 
 	obj = bpf_object__open_file(file, NULL);
@@ -60,3 +61,83 @@ void test_global_data_init(void)
 	free(newval);
 	bpf_object__close(obj);
 }
+
+void test_global_percpu_data_init(void)
+{
+	struct test_global_percpu_data *skel = NULL;
+	u64 *percpu_data = NULL;
+	struct bpf_map *map;
+	size_t init_data_sz;
+	char buff[128] = {};
+	int init_value = 2;
+	int key, value_sz;
+	int prog_fd, err;
+	int *init_data;
+	int num_cpus;
+
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		    .data_in = buff,
+		    .data_size_in = sizeof(buff),
+		    .repeat = 1,
+	);
+
+	num_cpus = libbpf_num_possible_cpus();
+	if (!ASSERT_GT(num_cpus, 0, "libbpf_num_possible_cpus"))
+		return;
+
+	percpu_data = calloc(num_cpus, sizeof(*percpu_data));
+	if (!ASSERT_FALSE(percpu_data == NULL, "calloc percpu_data"))
+		return;
+
+	value_sz = sizeof(*percpu_data) * num_cpus;
+	memset(percpu_data, 0, value_sz);
+
+	skel = test_global_percpu_data__open();
+	if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
+		goto out;
+
+	map = skel->maps.data__percpu;
+	err = bpf_map__set_initial_value(map, &init_value,
+					 sizeof(init_value));
+	if (!ASSERT_OK(err, "bpf_map__set_initial_value"))
+		goto out;
+
+	init_data = bpf_map__initial_value(map, &init_data_sz);
+	if (!ASSERT_OK_PTR(init_data, "bpf_map__initial_value"))
+		goto out;
+
+	ASSERT_EQ(*init_data, init_value, "initial_value");
+	ASSERT_EQ(init_data_sz, sizeof(init_value), "initial_value size");
+
+	err = test_global_percpu_data__load(skel);
+	if (err == -EOPNOTSUPP)
+		goto out;
+	if (!ASSERT_OK(err, "test_global_percpu_data__load"))
+		goto out;
+
+	ASSERT_EQ(bpf_map__type(map), BPF_MAP_TYPE_PERCPU_ARRAY,
+		  "bpf_map__type");
+
+	prog_fd = bpf_program__fd(skel->progs.update_percpu_data);
+	err = bpf_prog_test_run_opts(prog_fd, &topts);
+	ASSERT_OK(err, "update_percpu_data");
+	ASSERT_EQ(topts.retval, 0, "update_percpu_data retval");
+
+	key = 0;
+	err = bpf_map__lookup_elem(map, &key, sizeof(key), percpu_data,
+				   value_sz, 0);
+	if (!ASSERT_OK(err, "bpf_map__lookup_elem"))
+		goto out;
+
+	if (!ASSERT_LT(skel->bss->curr_cpu, num_cpus, "curr_cpu"))
+		goto out;
+	ASSERT_EQ(percpu_data[skel->bss->curr_cpu], 1, "percpu_data");
+	if (num_cpus > 1)
+		ASSERT_EQ(percpu_data[(skel->bss->curr_cpu+1)%num_cpus],
+			  init_value, "init_value");
+
+out:
+	test_global_percpu_data__destroy(skel);
+	if (percpu_data)
+		free(percpu_data);
+}
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +21 -0
new file mode 100644
index 0000000000000..2f01051b782c6
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright Leon Hwang */
+
+#include <linux/bpf.h>
+#include <linux/pkt_cls.h>
+
+#include <bpf/bpf_helpers.h>
+
+int percpu_data SEC(".data..percpu");
+int curr_cpu;
+
+SEC("tc")
+int update_percpu_data(struct __sk_buff *skb)
+{
+	curr_cpu = bpf_get_smp_processor_id();
+	percpu_data = 1;
+
+	return TC_ACT_OK;
+}
+
+char _license[] SEC("license") = "GPL";
Signature · 1 line
2.47.1
#7 lore raw
In reply to Leon Hwang

Hi Leon,

Quoted 16 lines · Leon Hwang

This patch set introduces global per-CPU data, similar to commit 6316f78306c1 ("Merge branch 'support-global-data'"), to reduce restrictions in C for BPF programs.

With this enhancement, it becomes possible to define and use global per-CPU variables, much like the DEFINE_PER_CPU() macro in the kernel[0].

The idea stems from the bpflbr project[1], which itself was inspired by retsnoop[2]. During testing of bpflbr on the v6.6 kernel, two LBR (Last Branch Record) entries were observed related to the bpf_get_smp_processor_id() helper.

Since commit 1ae6921009e5 ("bpf: inline bpf_get_smp_processor_id() helper"), the bpf_get_smp_processor_id() helper has been inlined on x86_64, reducing the overhead and consequently minimizing these two LBR records.

Quoted 8 lines

However, the introduction of global per-CPU data offers a more robust solution. By leveraging the percpu_array map and percpu instructions, global per-CPU data can be implemented intrinsically.

This feature also facilitates sharing per-CPU information between tail callers and callees or between freplace callers and callees through a shared global per-CPU variable. Previously, this was achieved using a 1-entry percpu map, which this patch set aims to improve upon.

I think this would be great to have. bpftrace would've liked to use this for its recent big string support, but instead had to simulate a percpu global through regular globals.

Thanks,
Daniel

author
#8 lore raw

Hi Daniel Xu,

Quoted 23 lines · Daniel Xu

Hi Leon,

Quoted 20 lines · Leon Hwang

This patch set introduces global per-CPU data, similar to commit 6316f78306c1 ("Merge branch 'support-global-data'"), to reduce restrictions in C for BPF programs.

With this enhancement, it becomes possible to define and use global per-CPU variables, much like the DEFINE_PER_CPU() macro in the kernel[0].

The idea stems from the bpflbr project[1], which itself was inspired by retsnoop[2]. During testing of bpflbr on the v6.6 kernel, two LBR (Last Branch Record) entries were observed related to the bpf_get_smp_processor_id() helper.

Since commit 1ae6921009e5 ("bpf: inline bpf_get_smp_processor_id() helper"), the bpf_get_smp_processor_id() helper has been inlined on x86_64, reducing the overhead and consequently minimizing these two LBR records.

However, the introduction of global per-CPU data offers a more robust solution. By leveraging the percpu_array map and percpu instructions, global per-CPU data can be implemented intrinsically.

Quoted 9 lines
Quoted 4 lines

This feature also facilitates sharing per-CPU information between tail callers and callees or between freplace callers and callees through a shared global per-CPU variable. Previously, this was achieved using a 1-entry percpu map, which this patch set aims to improve upon.

I think this would be great to have. bpftrace would've liked to use this for its recent big string support, but instead had to simulate a percpu global through regular globals.

Thank you for the feedback! I'm glad to hear that this feature could help simplify bpftrace, especially for use cases like the recent big string support. It's great to know this feature has potential to address more real-world challenges.

Thanks,
Leon