libbpf: move arena variables out of the zero page
Subsystems (5)
Emil Tsalapatis
reviewerv2 cover libbpf: move arena variables out of the zero page
Modify libbpf to place arena globals in a small offset inside the arena mapping instead of at the very beginning. This allows programs to leave the "zero page" of the arena unmapped, so that NULL arena pointer dereferences trigger a page fault and associated backtrace in BPF streams. In contrast, the current policy of placing global data in the zero pages means that NULL dereferences silently corrupt global data, e.g, arena qspinlock state. This makes arena bugs more difficult to debug.
The patchset adds code to libbpf to move global arena data 16 pages into the arena mapping. If this move is impossible, libbpf tries progressively smaller increments, and finally defaults to 0 if there is not enough space in the arena. At load time, libbpf adjusts each symbol's location within the arena by that offset. The patchset also adds padding to the BPF skeleton struct arena datasec to ensure the arena's fields are pointing in the right locations within the mapping.
HISTORY
-------
v1->v2
------
v1: https://lore.kernel.org/bpf/[email protected]/
- Fix ifdef guards causing unused variable errors for architectures/compilers without addr_space_cast support (CI)
Emil Tsalapatis (4):
selftests/bpf: explicitly account for globals in verifier_arena_large
libbpf: add stub for offset-related skeleton padding
libbpf: offset global arena data into the arena if possible selftests/bpf: add tests for the arena offset of globals
tools/bpf/bpftool/gen.c | 23 ++++++- tools/lib/bpf/libbpf.c | 36 ++++++++++- tools/lib/bpf/libbpf.h | 9 +++ tools/lib/bpf/libbpf.map | 1 + .../selftests/bpf/prog_tests/verifier.c | 6 ++ .../bpf/progs/verifier_arena_globals1.c | 60 ++++++++++++++++++ .../bpf/progs/verifier_arena_globals2.c | 49 +++++++++++++++ .../bpf/progs/verifier_arena_globals3.c | 61 +++++++++++++++++++ .../bpf/progs/verifier_arena_large.c | 25 ++++++-- 9 files changed, 261 insertions(+), 9 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_globals1.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_globals2.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_globals3.c
Signature · 1 line
2.49.0
Emil Tsalapatis
reviewerPATCH v2 1/4 selftests/bpf: explicitly account for globals in verifier_arena_large
The big_alloc1 test in verifier_arena_large assumes that the arena base and the first page allocated by bpf_arena_alloc_pages are identical. This is not the case, because the first page in the arena is populated by global arena data. The test still passes because the code makes the tacit assumption that the first page is on offset PAGE_SIZE instead of 0.
Make this distinction explicit in the code, and adjust the page offsets requested during the test to count from the beginning of the arena instead of using the address of the first allocated page.
--- .../selftests/bpf/progs/verifier_arena_large.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/progs/verifier_arena_large.c +11 -4
@@ -23,18 +23,25 @@ int big_alloc1(void *ctx) {#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) volatile char __arena *page1, *page2, *no_page, *page3; - void __arena *base; +u64 base; - page1 = base =bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 ); + base = (u64 )arena_base (&arena); + + page1 =bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 );if (!page1)return 1 ; + +/* Account for global arena data. */ +if ((u64 )page1 != base +PAGE_SIZE ) +return 15 ; + *page1 =1 ; - page2 =bpf_arena_alloc_pages (&arena, base +ARENA_SIZE -PAGE_SIZE *2 , + page2 = bpf_arena_alloc_pages(&arena, (void __arena *)(ARENA_SIZE - PAGE_SIZE),1 ,NUMA_NO_NODE ,0 );if (!page2)return 2 ; *page2 =2 ; - no_page =bpf_arena_alloc_pages (&arena, base +ARENA_SIZE -PAGE_SIZE , + no_page = bpf_arena_alloc_pages(&arena, (void __arena *)ARENA_SIZE,1 ,NUMA_NO_NODE ,0 );if (no_page)return 3 ;
Signature · 1 line
2.49.0
Re: [PATCH v2 1/4] selftests/bpf: explicitly account for globals in verifier_arena_large
Quoted 7 lines · Emil Tsalapatis
The
big_alloc1test inverifier_arena_largeassumes that the arena base and the first page allocated bybpf_arena_alloc_pagesare identical. This is not the case, because the first page in the arena is populated by global arena data. The test still passes because the code makes the tacit assumption that the first page is on offsetPAGE_SIZEinstead of 0.
Quoted 6 lines
Make this distinction explicit in the code, and adjust the page offsets requested during the test to count from the beginning of the arena instead of using the address of the first allocated page.
Signed-off-by Emil Tsalapatis <[email protected]>---
Seems correct.
[...]
Emil Tsalapatis
reviewerPATCH v2 2/4 libbpf: add stub for offset-related skeleton padding
In reply to Emil TsalapatisAdd a stub function for reporting in which offset within a mapping libbpf places the map's data. This will be used in a subsequent patch to support offsetting arena variables within the mapped region.
Adjust skeleton generation to account for the new arena memory layout by adding padding corresponding to the offset into the arena map. Add a libbbpf API function to get the data offset within the map's mapping during skeleton generation.
--- tools/bpf/bpftool/gen.c | 23 +++++++++++++++++++++-- tools/lib/bpf/libbpf.c | 10 ++++++++++ tools/lib/bpf/libbpf.h | 9 +++++++++ tools/lib/bpf/libbpf.map | 1 + 4 files changed, 41 insertions(+), 2 deletions(-)
tools/bpf/bpftool/gen.c +21 -2
@@ -148,7 +148,8 @@ static int codegen_datasec_def(struct bpf_object *obj,struct btf *btf, struct btf_dump *d, const struct btf_type *sec, - const char *obj_name) + const char *obj_name, + int var_off) { const char *sec_name = btf__name_by_offset(btf, sec->name_off);const struct btf_var_secinfo *sec_var =btf_var_secinfos (sec);
@@ -163,6 +164,17 @@ static int codegen_datasec_def(struct bpf_object *obj, strip_mods = true;printf (" struct %s__%s {\n" , obj_name, sec_ident); + +/* +* Arena variables may be placed in an offset within the section. +* Represent this in the skeleton using a padding struct. +*/ +if (var_off >0 ) { +printf ("\t\tchar __pad%d[%d];\n" , + pad_cnt, var_off); + pad_cnt++; + } +for (i =0 ; i < vlen; i++, sec_var++) {const struct btf_type *var =btf__type_by_id (btf, sec_var->type);const char *var_name =btf__name_by_offset (btf, var->name_off);
@@ -279,6 +291,7 @@ static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)struct bpf_map *map;const struct btf_type *sec;char map_ident[256 ]; +int var_off;int err =0 ; d =btf_dump__new (btf, codegen_btf_dump_printf,NULL ,NULL );
@@ -303,7 +316,13 @@ static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)printf (" struct %s__%s {\n" , obj_name, map_ident);printf (" } *%s;\n" , map_ident); }else { - err =codegen_datasec_def (obj, btf, d, sec, obj_name); + var_off =bpf_map__data_offset (map); +if (var_off <0 ) { +p_err ("bpf_map__data_offset called on unmapped map\n" ); + err = var_off; + goto out; + } + err =codegen_datasec_def (obj, btf, d, sec, obj_name, var_off);if (err) goto out; }
tools/lib/bpf/libbpf.c +10 -0
@@ -10552,6 +10552,16 @@ const char *bpf_map__name(const struct bpf_map *map)return map->name; } +int bpf_map__data_offset (const struct bpf_map *map) +{ +if (!map->mmaped) +return -EINVAL ; + +/* No offsetting for now. */ +return 0 ; +} + +enum bpf_map_type bpf_map__type (const struct bpf_map *map) {return map->def.type;
tools/lib/bpf/libbpf.h +9 -0
@@ -1314,6 +1314,15 @@ LIBBPF_API int bpf_map__set_exclusive_program(struct bpf_map *map, struct bpf_pr*/ LIBBPF_API struct bpf_program *bpf_map__exclusive_program(struct bpf_map *map); +/* +* @brief **bpf_map__data_offset** returns the offset of the map's data +* within the address mapping. +* @param BPF map whose variable offset we are looking into. +* @return the offset >= 0 of the map's contents within its mapping; negative +* error code, otherwise. +*/ +LIBBPF_API int bpf_map__data_offset(const struct bpf_map *map); +struct bpf_xdp_set_link_opts {size_t sz;int old_fd;
tools/lib/bpf/libbpf.map +1 -0
@@ -451,4 +451,5 @@ LIBBPF_1.7.0 { global: bpf_map__set_exclusive_program; bpf_map__exclusive_program; + bpf_map__data_offset; } LIBBPF_1.6.0;
Signature · 1 line
2.49.0
Emil Tsalapatis
reviewerPATCH v2 3/4 libbpf: offset global arena data into the arena if possible
In reply to Emil TsalapatisCurrently, libbpf places global arena data at the very beginning of the arena mapping. Stray NULL dereferences into the arena then find valid data and lead to silent corruption instead of causing an arena page fault. The data is placed in the mapping at load time, preventing us from reserving the region using bpf_arena_reserve_pages().
Adjust the arena logic to attempt placing the data from an offset within the arena (currently 16 pages in) instead of the very beginning. If placing the data at an offset would lead to an allocation failure due to global data being as large as the entire arena, progressively reduce the offset down to 0 until placement succeeds.
Adjust existing arena tests in the same commit to account for the new global data offset. New tests that explicitly consider the new feature are introduced in the next patch.
--- tools/lib/bpf/libbpf.c | 30 +++++++++++++++---- .../bpf/progs/verifier_arena_large.c | 14 +++++++-- 2 files changed, 37 insertions(+), 7 deletions(-)
tools/lib/bpf/libbpf.c +25 -5
@@ -757,6 +757,7 @@ struct bpf_object {int arena_map_idx;void *arena_data;size_t arena_data_sz; +__u32 arena_data_off;void *jumptables_data;size_t jumptables_data_sz;
@@ -2991,10 +2992,14 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map, void *data, size_t data_sz) {const long page_sz =sysconf (_SC_PAGE_SIZE); +const size_t data_alloc_sz =roundup (data_sz, page_sz); +/* default offset into the arena, may be resized */ +const long max_off_pages =16 ;size_t mmap_sz; +long off_pages; mmap_sz =bpf_map_mmap_sz (map); -if (roundup (data_sz, page_sz) > mmap_sz) { +if (data_alloc_sz > mmap_sz) {pr_warn ("elf: sec '%s': declared ARENA map size (%zu) is too small to hold global __arena variables of size %zu\n" , sec_name, mmap_sz, data_sz);return -E2BIG ;
@@ -3006,6 +3011,17 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map,memcpy (obj->arena_data, data, data_sz); obj->arena_data_sz = data_sz; +/* +* find the largest offset for global arena variables +* where they still fit in the arena +*/ +for (off_pages = max_off_pages; off_pages >0 ; off_pages >>=1 ) { +if (off_pages * page_sz + data_alloc_sz <= mmap_sz) +break ; + } + + obj->arena_data_off = off_pages * page_sz; +/* make bpf_map__init_value() work for ARENA maps */ map->mmaped = obj->arena_data;
@@ -4663,7 +4679,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog, reloc_desc->type =RELO_DATA ; reloc_desc->insn_idx = insn_idx; reloc_desc->map_idx = obj->arena_map_idx; - reloc_desc->sym_off = sym->st_value; + reloc_desc->sym_off = sym->st_value + obj->arena_data_off; map = &obj->maps[obj->arena_map_idx]; pr_debug("prog '%s': found arena map %d (%s, sec %d, off %zu) for insn %u\n",
@@ -5624,7 +5640,8 @@ bpf_object__create_maps(struct bpf_object *obj)return err; }if (obj->arena_data) { -memcpy (map->mmaped, obj->arena_data, obj->arena_data_sz); +memcpy (map->mmaped + obj->arena_data_off, obj->arena_data, + obj->arena_data_sz);zfree (&obj->arena_data); } }
@@ -10557,8 +10574,11 @@ int bpf_map__data_offset(const struct bpf_map *map)if (!map->mmaped)return -EINVAL ; -/* No offsetting for now. */ -return 0 ; +/* Only arenas have offsetting. */ +if (map->def.type !=BPF_MAP_TYPE_ARENA ) +return 0 ; + +return map->obj->arena_data_off; }
tools/testing/selftests/bpf/progs/verifier_arena_large.c +12 -2
@@ -10,6 +10,7 @@#include "bpf_arena_common.h" #define ARENA_SIZE (1ull << 32) +#define GLOBAL_PGOFF (16)struct { __uint(type, BPF_MAP_TYPE_ARENA);
@@ -31,8 +32,7 @@ int big_alloc1(void *ctx)if (!page1)return 1 ; -/* Account for global arena data. */ -if ((u64 )page1 != base +PAGE_SIZE ) +if ((u64 )page1 != base)return 15 ; *page1 =1 ;
@@ -216,6 +216,16 @@ int big_alloc2(void *ctx) __u8 __arena *pg;int i, err; +/* +* The global data is placed in a page with global offset 16. +* This test is about page allocation contiguity, so avoid +* accounting for the stray allocation by also allocating +* all pages before it. We never use the page range, so leak it. +*/ + pg =bpf_arena_alloc_pages (&arena,NULL ,GLOBAL_PGOFF ,NUMA_NO_NODE ,0 ); +if (!pg) +return 10 ; + base =bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 );if (!base)return 1 ;
Signature · 1 line
2.49.0
Eduard Zingerman
maintainerRe: [PATCH v2 3/4] libbpf: offset global arena data into the arena if possible
Quoted 18 lines · Emil Tsalapatis
Currently, libbpf places global arena data at the very beginning of the arena mapping. Stray NULL dereferences into the arena then find valid data and lead to silent corruption instead of causing an arena page fault. The data is placed in the mapping at load time, preventing us from reserving the region using
bpf_arena_reserve_pages().Adjust the arena logic to attempt placing the data from an offset within the arena (currently 16 pages in) instead of the very beginning. If placing the data at an offset would lead to an allocation failure due to global data being as large as the entire arena, progressively reduce the offset down to 0 until placement succeeds.
Adjust existing arena tests in the same commit to account for the new global data offset. New tests that explicitly consider the new feature are introduced in the next patch.
Signed-off-by Emil Tsalapatis <[email protected]>---
[...]
Quoted 15 lines
tools/lib/bpf/libbpf.c +0
-0@@ -3006,6 +3011,17 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map, memcpy(obj->arena_data, data, data_sz); obj->arena_data_sz = data_sz;tools/lib/bpf/libbpf.c +11
-0+/* +* find the largest offset for global arena variables +* where they still fit in the arena +*/ +for (off_pages = max_off_pages; off_pages >0 ; off_pages >>=1 ) { +if (off_pages * page_sz + data_alloc_sz <= mmap_sz) +break ; + } + + obj->arena_data_off = off_pages * page_sz; +
Quoted 3 lines
/* make
bpf_map__init_value()work for ARENA maps */map->mmaped=obj->arena_data;
Please correct me if I'm wrong about the goals of this change:
a. Avoid allocating global data at NULL address
b. Reserve some space to use by upcoming arena-KASAN functionality.
For (b) wouldn't it be simpler to implicitly increase the arena map size by an amount needed for KASAN functionality? Then there would be no need to guess the necessary data offset.
For (a), is there a way to move an address of first valid mmaped page (from BPF perspective) w/o physically allocating the first page?
[...]
Emil Tsalapatis
reviewerRe: [PATCH v2 3/4] libbpf: offset global arena data into the arena if possible
Quoted 27 lines · Eduard Zingerman
Quoted 18 lines · Emil Tsalapatis
Currently, libbpf places global arena data at the very beginning of the arena mapping. Stray NULL dereferences into the arena then find valid data and lead to silent corruption instead of causing an arena page fault. The data is placed in the mapping at load time, preventing us from reserving the region using
bpf_arena_reserve_pages().Adjust the arena logic to attempt placing the data from an offset within the arena (currently 16 pages in) instead of the very beginning. If placing the data at an offset would lead to an allocation failure due to global data being as large as the entire arena, progressively reduce the offset down to 0 until placement succeeds.
Adjust existing arena tests in the same commit to account for the new global data offset. New tests that explicitly consider the new feature are introduced in the next patch.
Signed-off-by Emil Tsalapatis <[email protected]>---
[...]
Quoted 4 lines
tools/lib/bpf/libbpf.c +0
-0@@ -3006,6 +3011,17 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map,memcpy (obj->arena_data, data, data_sz); obj->arena_data_sz = data_sz;
Quoted 15 lines
Quoted 4 lines
tools/lib/bpf/libbpf.c +4
-0+/* +* find the largest offset for global arena variables +* where they still fit in the arena +*/ tools/lib/bpf/libbpf.c static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map, · line 3027
Quoted 10 lines
tools/lib/bpf/libbpf.c +7
-0+for (off_pages = max_off_pages; off_pages >0 ; off_pages >>=1 ) { +if (off_pages * page_sz + data_alloc_sz <= mmap_sz) +break ; + } + + obj->arena_data_off = off_pages * page_sz; +/* make bpf_map__init_value() work for ARENA maps */ map->mmaped = obj->arena_data;
Hi Eduard,
Quoted 8 lines
Please correct me if I'm wrong about the goals of this change:
a. Avoid allocating global data at NULL address
b. Reserve some space to use by upcoming arena-KASAN functionality.For (b) wouldn't it be simpler to implicitly increase the arena map size by an amount needed for KASAN functionality? Then there would be no need to guess the necessary data offset.
b. isn't really an issue because we do not need to reserve the shadow map region.
Arena ASAN uses a global variable to denote the base of the shadow mapping, since the mapping needs to be in the last 1/8th of the address space, and the address space size is equal to the size of the arena (and thus configurable by the user).
(More details about how the ASAN region is reserved for reference:) The ASAN runtime code initializes the shadow map at load time and makes the necessary allocations for it. If the globals fall in the ASAN shadow map region, then
ASAN will fail for now. This only happens if the arena is tiny, or the size of the
globals is massive. Both cases are outliers and we support them later, but the bottom line is that this is a problem for the ASAN runtime and we don't have to worry about it in this patch.
Quoted 3 lines
For (a), is there a way to move an address of first valid mmaped page (from BPF perspective) w/o physically allocating the first page?
IIUC what you mean:
This is what this change combined with bpf_alloc_reserve_pages() amounts to. First, we adjust all the symbol addresses into the mapping to move the address out of the zero the page. Then, we call the existing bpf_alloc_reserve_pages() call to prevent the first page from ever being physically allocated.
Alternatively, if you maybe mean transparently add an offset to every arena address
in xlated/jitted code: That would also adjust the NULL pointers we're
trying to catch
to point to the first valid page we can allocate. In general, I don't think it's possible
to do this in the verifier/JIT because this change depends on adjusting only the arena globals, and that is best done at load time.
Quoted 1 line
[...]
Eduard Zingerman
maintainerRe: [PATCH v2 3/4] libbpf: offset global arena data into the arena if possible
On Mon, 2025-12-01 at 13:34 -0500, Emil Tsalapatis wrote:
[...]
Quoted 10 lines
Quoted 3 lines
For (a), is there a way to move an address of first valid mmaped page (from BPF perspective) w/o physically allocating the first page?
IIUC what you mean:
This is what this change combined with
bpf_alloc_reserve_pages()amounts to. First, we adjust all the symbol addresses into the mapping to move the address out of the zero the page. Then, we call the existingbpf_alloc_reserve_pages()call to prevent the first page from ever being physically allocated.
libbpf.c:bpf_object__create_maps() allocates arena memory region as follows:
map->mmaped = mmap(addr: (void *)(long)map->map_extra, ...);
Where 'map_extra' comes from fill_map_from_def() / parse_btf_map_def(). The latter even provides a flag:
map_def->parts |= MAP_DEF_MAP_EXTRA;
On the kernel side arena.c:arena_map_alloc() uses this 'map_extra' as a starting arena address:
arena->user_vm_start = attr->map_extra;
arena.c:arena_alloc_pages() uses 'user_vm_start' as follows:
uaddr32 = (u32)(arena->user_vm_start + pgoff * PAGE_SIZE); ... return clear_lo32(val: arena->user_vm_start) + uaddr32;
Returned value is what BPF program sees as an address, right? Is there a way to tweak arena_alloc_pages() implementation or arena creation at libbpf side, such that arena_alloc_pages() never returns value smaller then 16 * PAGE_SIZE?
E.g. by tweaking the attr->map_extra value in bpf_object__create_maps() if it was not specified directly by user?
Quoted 8 lines
Alternatively, if you maybe mean transparently add an offset to every arena address
in xlated/jitted code: That would also adjust the NULL pointers we're
trying to catch
to point to the first valid page we can allocate. In general, I don't think it's possible
to do this in the verifier/JIT because this change depends on adjusting only the arena globals, and that is best done at load time.
For sure, it does not make sense to do it on verifier side. It might even be impossible in general case.
Emil Tsalapatis
reviewerRe: [PATCH v2 3/4] libbpf: offset global arena data into the arena if possible
Quoted 37 lines · Eduard Zingerman
On Mon, 2025-12-01 at 13:34 -0500, Emil Tsalapatis wrote:
[...]
Quoted 10 lines
Quoted 3 lines
For (a), is there a way to move an address of first valid mmaped page (from BPF perspective) w/o physically allocating the first page?
IIUC what you mean:
This is what this change combined with
bpf_alloc_reserve_pages()amounts to. First, we adjust all the symbol addresses into the mapping to move the address out of the zero the page. Then, we call the existingbpf_alloc_reserve_pages()call to prevent the first page from ever being physically allocated.libbpf.c:bpf_object__create_maps() allocates arena memory region as follows:
map->mmaped = mmap(addr: (void *)(long)map->map_extra, ...);Where '
map_extra' comes fromfill_map_from_def()/parse_btf_map_def(). The latter even provides a flag:map_def->parts |= MAP_DEF_MAP_EXTRA;On the kernel side arena.c:arena_map_alloc() uses this '
map_extra' as a starting arena address:arena->user_vm_start = attr->map_extra;arena.c:arena_alloc_pages() uses '
user_vm_start' as follows:uaddr32 = (u32)(arena->user_vm_start + pgoff * PAGE_SIZE); ... return clear_lo32(val: arena->user_vm_start) + uaddr32;
Quoted 7 lines
Returned value is what BPF program sees as an address, right? Is there a way to tweak
arena_alloc_pages()implementation or arena creation at libbpf side, such thatarena_alloc_pages()never returns value smaller then 16 *PAGE_SIZE?
E.g. by tweaking theattr->map_extravalue inbpf_object__create_maps()if it was not specified directly by user?
Turns out that we can move the globals to the end of the arena by moving the shadow map, so the following is moot: But we could always call bpf_reserve_alloc_pages() from libbpf when we first set up the arena.
It turns out that it's trivial to move the ASAN shadow map around the address space so we can remove most of the issues in the patchset ( extra libbpf call, globals in the middle of the address space complicating the logic). I will send an updated version.
Quoted 11 lines
Quoted 8 lines
Alternatively, if you maybe mean transparently add an offset to every arena address
in xlated/jitted code: That would also adjust the NULL pointers we're
trying to catch
to point to the first valid page we can allocate. In general, I don't think it's possible
to do this in the verifier/JIT because this change depends on adjusting only the arena globals, and that is best done at load time.For sure, it does not make sense to do it on verifier side. It might even be impossible in general case.
Re: [PATCH v2 3/4] libbpf: offset global arena data into the arena if possible
In reply to Emil TsalapatisQuoted 7 lines · Emil Tsalapatis
Currently, libbpf places global arena data at the very beginning of the arena mapping. Stray NULL dereferences into the arena then find valid data and lead to silent corruption instead of causing an arena page fault. The data is placed in the mapping at load time, preventing us from reserving the region using
bpf_arena_reserve_pages().
Quoted 6 lines
Adjust the arena logic to attempt placing the data from an offset within the arena (currently 16 pages in) instead of the very beginning. If placing the data at an offset would lead to an allocation failure due to global data being as large as the entire arena, progressively reduce the offset down to 0 until placement succeeds.
I'm not a big fan of adding a single-purpose bpf_map__data_offset(), tbh, and the whole "let's try to place it within the first 16 pages" logic also looks a bit random...
Can't we just say that arena-based global variables are always placed at the end of the arena? Obviously, page aligned all that stuff, so it's deterministic and well-defined. Seems like we always expect BPF_MAP_TYPE_ARENA arena explicitly defined in BPF object file with max_entries set, so that should always work as expected?
And also, I don't think we need to change anything about skeleton generation logic for the arena. That padding is not necessary, libbpf should be able to point arena struct to the beginning of arena global variables, no? As you implemented patch #2, it breaks backwards compat for no good reason.
WDYT?
pw-bot: cr
Quoted 10 lines
Adjust existing arena tests in the same commit to account for the new global data offset. New tests that explicitly consider the new feature are introduced in the next patch.
Signed-off-by Emil Tsalapatis <[email protected]>--- tools/lib/bpf/libbpf.c | 30 +++++++++++++++---- .../bpf/progs/verifier_arena_large.c | 14 +++++++-- 2 files changed, 37 insertions(+), 7 deletions(-)
[...]
Emil Tsalapatis
reviewerRe: [PATCH v2 3/4] libbpf: offset global arena data into the arena if possible
Quoted 9 lines · Andrii Nakryiko
Quoted 7 lines · Emil Tsalapatis
Currently, libbpf places global arena data at the very beginning of the arena mapping. Stray NULL dereferences into the arena then find valid data and lead to silent corruption instead of causing an arena page fault. The data is placed in the mapping at load time, preventing us from reserving the region using
bpf_arena_reserve_pages().
Quoted 7 lines
Quoted 6 lines
Adjust the arena logic to attempt placing the data from an offset within the arena (currently 16 pages in) instead of the very beginning. If placing the data at an offset would lead to an allocation failure due to global data being as large as the entire arena, progressively reduce the offset down to 0 until placement succeeds.
Hi Andrii,
Quoted 4 lines
I'm not a big fan of adding a single-purpose
bpf_map__data_offset(), tbh, and the whole "let's try to place it within the first 16 pages" logic also looks a bit random...
Giving it some thought, I think we can get away with removing the new API call. The intended use is for userspace to know where the global data is placed within the arena. For example, we've had sched-ext code in the past that allocated
arena pages from userspace using read faults, and that code had to start in the had to start in the middle of the arena to avoid conflicting with globals.
I think it is reasonable to have users call mincore(2) into the mapping to find which
pages were populated by libbpf when loading the globals in the arena. I can't think
of any other use cases for the API call, so we can remove it. Does this make sense?
For the 16 pages rule I chose 16 as a sane default because we don't want users choosing
it themselves. The logic is to make sure there's no "false negatives" wrt to relocation, to add as many guard pages as we can in case the arena is small.
Is it the number or the logic for choosing the offset that looks random? How would you like to simplify it? Possible ways I see are:
1) Choose a different number, maybe just a single page. That does open up the possibility of very large offsets added to NULL pointers ending up in the globals
regions, and 1 isn't really any less arbitrary than 16 or any other choice.
2) Simplify the logic for choosing the offset. Right now it's just exponential backoff
from 16 all the way to 0, in which case we fall back to existing behavior. We can
make it that relocation either succeeds at an offset or fails, and avoid trying intermediate offsets.
Unless the code is very confusing (I don't think it is, but it's my own code so I'm
not exactly impartial :) ) I don't think removing the backoff gets us much.
Quoted 6 lines
Can't we just say that arena-based global variables are always placed at the end of the arena? Obviously, page aligned all that stuff, so it's deterministic and well-defined. Seems like we always expect
BPF_MAP_TYPE_ARENAarena explicitly defined in BPF object file withmax_entriesset, so that should always work as expected?
The main problem I see with this is that it will cause complications with the arena shadow
map. Putting the global variables at the end requires us to either forfeit the possibility of
applying ASAN to them, or to complicate the mapping function. The latter will have implications
for performance and code complexity, since it will add a conditional jump on every ASAN
check.
Quoted 8 lines
And also, I don't think we need to change anything about skeleton generation logic for the arena. That padding is not necessary, libbpf should be able to point arena struct to the beginning of arena global variables, no? As you implemented patch #2, it breaks backwards compat for no good reason.
WDYT?
IIUC you mean modifying the address returned by the arena to point where the globals are
placed instead of the guard region, right? I considered it at first but I think it has its own set
of caveats, mainly that it assumes the program never wants to access the region below the
globals.
Right now the patch makes the assumption that the program will set a guard region themselves.
It may not do so and instead use the zero page to store data. In that case, accessing the page
requires a negative offset on the arena page. It also breaks the mincore(2) trick I describe
above for finding the offset into the arena without a new API call. AFAICT expanding the
libbpf API is the bigger problem of the two, but I am not sure.
If we'd like both to avoid the extra API and to avoid the padding, we can always set up the guard
mapping by default in libbpf at load time. Would that be acceptable? It does take away control
from the user, but unless they have very good reason to constrain arena size to a handful of pages
I don't see how the extra guard mapping can be a problem.
Quoted 15 lines
pw-bot: cr
Quoted 10 lines
Adjust existing arena tests in the same commit to account for the new global data offset. New tests that explicitly consider the new feature are introduced in the next patch.
Signed-off-by Emil Tsalapatis <[email protected]>--- tools/lib/bpf/libbpf.c | 30 +++++++++++++++---- .../bpf/progs/verifier_arena_large.c | 14 +++++++-- 2 files changed, 37 insertions(+), 7 deletions(-)[...]
Re: [PATCH v2 3/4] libbpf: offset global arena data into the arena if possible
Quoted 34 lines · Emil Tsalapatis
Quoted 9 lines · Andrii Nakryiko
Quoted 7 lines · Emil Tsalapatis
Currently, libbpf places global arena data at the very beginning of the arena mapping. Stray NULL dereferences into the arena then find valid data and lead to silent corruption instead of causing an arena page fault. The data is placed in the mapping at load time, preventing us from reserving the region using
bpf_arena_reserve_pages().Quoted 7 lines
Quoted 6 lines
Adjust the arena logic to attempt placing the data from an offset within the arena (currently 16 pages in) instead of the very beginning. If placing the data at an offset would lead to an allocation failure due to global data being as large as the entire arena, progressively reduce the offset down to 0 until placement succeeds.
Hi Andrii,
Quoted 4 lines
I'm not a big fan of adding a single-purpose
bpf_map__data_offset(), tbh, and the whole "let's try to place it within the first 16 pages" logic also looks a bit random...Giving it some thought, I think we can get away with removing the new API call. The intended use is for userspace to know where the global data is placed within the arena. For example, we've had sched-ext code in the past that allocated
arena pages from userspace using read faults, and that code had to start in the had to start in the middle of the arena to avoid conflicting with globals.
Quoted 6 lines
I think it is reasonable to have users call mincore(2) into the mapping to find which
pages were populated by libbpf when loading the globals in the arena. I can't think
of any other use cases for the API call, so we can remove it. Does this make sense?
I don't know if I agree that it's reasonable to expect users to use mincore() to learn where libbpf put global variables. Most natural location is at the beginning of arena map, but if that doesn't work, then I expected that at the very end of arena would be ok.
Quoted 5 lines
For the 16 pages rule I chose 16 as a sane default because we don't want users choosing
it themselves. The logic is to make sure there's no "false negatives" wrt to relocation, to add as many guard pages as we can in case the arena is small.
I tried to understand what you are saying about false negatives, but I'm not sure I get it, sorry. I do agree that asking users to choose the offset is suboptimal, though.
Quoted 2 lines
Is it the number or the logic for choosing the offset that looks random?
The offset you chose is a bit random, plus that piece of logic where you will be shifting from 16 down to 0 pages, if you happen to fail to mmap.
Quoted 6 lines
How would you like to simplify it? Possible ways I see are:
1) Choose a different number, maybe just a single page. That does open up the possibility of very large offsets added to NULL pointers ending up in the globals
regions, and 1 isn't really any less arbitrary than 16 or any other choice.
Yeah, I was guessing you didn't choose just one page because you wanted to catch large field offsets added to NULL. (but yes, 1 is way less arbitrary than 16, but that's beside the point)
Quoted 7 lines
2) Simplify the logic for choosing the offset. Right now it's just exponential backoff
from 16 all the way to 0, in which case we fall back to existing behavior. We can
make it that relocation either succeeds at an offset or fails, and avoid trying intermediate offsets.
Yes, if we absolutely have to go with 16 page offset instead of putting globals at the end, I'd just fix the offset and it either works or not. It still feels wrong to hard-code 16 as part of API, though.
Quoted 12 lines
Unless the code is very confusing (I don't think it is, but it's my own code so I'm
not exactly impartial :) ) I don't think removing the backoff gets us much.Quoted 6 lines
Can't we just say that arena-based global variables are always placed at the end of the arena? Obviously, page aligned all that stuff, so it's deterministic and well-defined. Seems like we always expect
BPF_MAP_TYPE_ARENAarena explicitly defined in BPF object file withmax_entriesset, so that should always work as expected?
Quoted 10 lines
The main problem I see with this is that it will cause complications with the arena shadow
map. Putting the global variables at the end requires us to either forfeit the possibility of
applying ASAN to them, or to complicate the mapping function. The latter will have implications
for performance and code complexity, since it will add a conditional jump on every ASAN
check.
So maybe elaborate a bit on this? I can't say to understand why locating globals at the end of arena would change anything for your ASAN logic. Wouldn't ASAN have to track data at any possible location anyways? I'm sure you have a good answer here, but please lay it out.
Quoted 9 lines
Quoted 8 lines
And also, I don't think we need to change anything about skeleton generation logic for the arena. That padding is not necessary, libbpf should be able to point arena struct to the beginning of arena global variables, no? As you implemented patch #2, it breaks backwards compat for no good reason.
WDYT?
Quoted 7 lines
IIUC you mean modifying the address returned by the arena to point where the globals are
placed instead of the guard region, right? I considered it at first but I think it has its own set
of caveats, mainly that it assumes the program never wants to access the region below the
globals.
Hm... I guess it depends on how we look at that *arena pointer. I was thinking about it as more of pointer to arena-placed global variables, not some sort of generic way to access any place within the arena. So in that sense it's natural for that *arena pointer to point to just global variables, wherever within the arena data area it might be located.
If user needs to work with other arena pages, they can fetch memory-mapped memory for arena map itself through bpf_map__initial_value(skel->maps.arena). I.e., they take struct bpf_map object, ask libbpf where its data is, and then work with it in "untyped" way.
Quoted 10 lines
Right now the patch makes the assumption that the program will set a guard region themselves.
It may not do so and instead use the zero page to store data. In that case, accessing the page
requires a negative offset on the arena page. It also breaks the mincore(2) trick I describe
above for finding the offset into the arena without a new API call. AFAICT expanding the
libbpf API is the bigger problem of the two, but I am not sure.
It just feels like we are exposing way too much implementation details here. See my point above about using bpf_map__initial_value() (the name is not great, but it's used to get memory-mapped area for mmap-able BPF maps).
Quoted 9 lines
If we'd like both to avoid the extra API and to avoid the padding, we can always set up the guard
mapping by default in libbpf at load time. Would that be acceptable? It does take away control
from the user, but unless they have very good reason to constrain arena size to a handful of pages
I don't see how the extra guard mapping can be a problem.
Again, I'm only guessing what you are trying to say here. Please be more specific.
Quoted 15 lines
Quoted 15 lines
pw-bot: cr
Quoted 10 lines
Adjust existing arena tests in the same commit to account for the new global data offset. New tests that explicitly consider the new feature are introduced in the next patch.
Signed-off-by Emil Tsalapatis <[email protected]>--- tools/lib/bpf/libbpf.c | 30 +++++++++++++++---- .../bpf/progs/verifier_arena_large.c | 14 +++++++-- 2 files changed, 37 insertions(+), 7 deletions(-)[...]
Emil Tsalapatis
reviewerRe: [PATCH v2 3/4] libbpf: offset global arena data into the arena if possible
Quoted 112 lines · Andrii Nakryiko
Quoted 34 lines · Emil Tsalapatis
Quoted 9 lines · Andrii Nakryiko
Quoted 7 lines · Emil Tsalapatis
Currently, libbpf places global arena data at the very beginning of the arena mapping. Stray NULL dereferences into the arena then find valid data and lead to silent corruption instead of causing an arena page fault. The data is placed in the mapping at load time, preventing us from reserving the region using
bpf_arena_reserve_pages().Quoted 7 lines
Quoted 6 lines
Adjust the arena logic to attempt placing the data from an offset within the arena (currently 16 pages in) instead of the very beginning. If placing the data at an offset would lead to an allocation failure due to global data being as large as the entire arena, progressively reduce the offset down to 0 until placement succeeds.
Hi Andrii,
Quoted 4 lines
I'm not a big fan of adding a single-purpose
bpf_map__data_offset(), tbh, and the whole "let's try to place it within the first 16 pages" logic also looks a bit random...Giving it some thought, I think we can get away with removing the new API call. The intended use is for userspace to know where the global data is placed within the arena. For example, we've had sched-ext code in the past that allocated
arena pages from userspace using read faults, and that code had to start in the had to start in the middle of the arena to avoid conflicting with globals.Quoted 6 lines
I think it is reasonable to have users call mincore(2) into the mapping to find which
pages were populated by libbpf when loading the globals in the arena. I can't think
of any other use cases for the API call, so we can remove it. Does this make sense?I don't know if I agree that it's reasonable to expect users to use
mincore()to learn where libbpf put global variables. Most natural location is at the beginning of arena map, but if that doesn't work, then I expected that at the very end of arena would be ok.Quoted 5 lines
For the 16 pages rule I chose 16 as a sane default because we don't want users choosing
it themselves. The logic is to make sure there's no "false negatives" wrt to relocation, to add as many guard pages as we can in case the arena is small.I tried to understand what you are saying about false negatives, but I'm not sure I get it, sorry. I do agree that asking users to choose the offset is suboptimal, though.
Quoted 2 lines
Is it the number or the logic for choosing the offset that looks random?
The offset you chose is a bit random, plus that piece of logic where you will be shifting from 16 down to 0 pages, if you happen to fail to mmap.
Quoted 6 lines
How would you like to simplify it? Possible ways I see are:
1) Choose a different number, maybe just a single page. That does open up the possibility of very large offsets added to NULL pointers ending up in the globals
regions, and 1 isn't really any less arbitrary than 16 or any other choice.Yeah, I was guessing you didn't choose just one page because you wanted to catch large field offsets added to NULL. (but yes, 1 is way less arbitrary than 16, but that's beside the point)
Quoted 7 lines
2) Simplify the logic for choosing the offset. Right now it's just exponential backoff
from 16 all the way to 0, in which case we fall back to existing behavior. We can
make it that relocation either succeeds at an offset or fails, and avoid trying intermediate offsets.Yes, if we absolutely have to go with 16 page offset instead of putting globals at the end, I'd just fix the offset and it either works or not. It still feels wrong to hard-code 16 as part of API, though.
Quoted 12 lines
Unless the code is very confusing (I don't think it is, but it's my own code so I'm
not exactly impartial :) ) I don't think removing the backoff gets us much.Quoted 6 lines
Can't we just say that arena-based global variables are always placed at the end of the arena? Obviously, page aligned all that stuff, so it's deterministic and well-defined. Seems like we always expect
BPF_MAP_TYPE_ARENAarena explicitly defined in BPF object file withmax_entriesset, so that should always work as expected?Quoted 10 lines
The main problem I see with this is that it will cause complications with the arena shadow
map. Putting the global variables at the end requires us to either forfeit the possibility of
applying ASAN to them, or to complicate the mapping function. The latter will have implications
for performance and code complexity, since it will add a conditional jump on every ASAN
check.
Quoted 5 lines
So maybe elaborate a bit on this? I can't say to understand why locating globals at the end of arena would change anything for your ASAN logic. Wouldn't ASAN have to track data at any possible location anyways? I'm sure you have a good answer here, but please lay it out.
You're actually right, it's trivial to move. I'll send a new version of the patch
that places all globals at the end. I think this removes all problems present in the current version. The only extra bit in the new version will be a slight verifier change to handle large direct offsets into arena maps.
Quoted 41 lines
Quoted 9 lines
Quoted 8 lines
And also, I don't think we need to change anything about skeleton generation logic for the arena. That padding is not necessary, libbpf should be able to point arena struct to the beginning of arena global variables, no? As you implemented patch #2, it breaks backwards compat for no good reason.
WDYT?
Quoted 7 lines
IIUC you mean modifying the address returned by the arena to point where the globals are
placed instead of the guard region, right? I considered it at first but I think it has its own set
of caveats, mainly that it assumes the program never wants to access the region below the
globals.Hm... I guess it depends on how we look at that *arena pointer. I was thinking about it as more of pointer to arena-placed global variables, not some sort of generic way to access any place within the arena. So in that sense it's natural for that *arena pointer to point to just global variables, wherever within the arena data area it might be located.
If user needs to work with other arena pages, they can fetch memory-mapped memory for arena map itself through
bpf_map__initial_value(skel->maps.arena). I.e., they takestruct bpf_mapobject, ask libbpf where its data is, and then work with it in "untyped" way.Quoted 10 lines
Right now the patch makes the assumption that the program will set a guard region themselves.
It may not do so and instead use the zero page to store data. In that case, accessing the page
requires a negative offset on the arena page. It also breaks the mincore(2) trick I describe
above for finding the offset into the arena without a new API call. AFAICT expanding the
libbpf API is the bigger problem of the two, but I am not sure.
Quoted 5 lines
It just feels like we are exposing way too much implementation details here. See my point above about using
bpf_map__initial_value()(the name is not great, but it's used to get memory-mapped area for mmap-able BPF maps).
Wasn't aware about bpf_map__initial_value(), if we have an existing way of finding the arena's vm_user_start already then I don't see a problem with changing the arena pointer to point to the globals instead. I'll remove the additional libbpf call in the next version.
Quoted 10 lines
Quoted 9 lines
If we'd like both to avoid the extra API and to avoid the padding, we can always set up the guard
mapping by default in libbpf at load time. Would that be acceptable? It does take away control
from the user, but unless they have very good reason to constrain arena size to a handful of pages
I don't see how the extra guard mapping can be a problem.
Quoted 3 lines
Again, I'm only guessing what you are trying to say here. Please be more specific.
I meant we could do bpf_alloc_reserve_pages() on arena load times, but the point is mott.
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
pw-bot: cr
Quoted 10 lines
Adjust existing arena tests in the same commit to account for the new global data offset. New tests that explicitly consider the new feature are introduced in the next patch.
Signed-off-by Emil Tsalapatis <[email protected]>--- tools/lib/bpf/libbpf.c | 30 +++++++++++++++---- .../bpf/progs/verifier_arena_large.c | 14 +++++++-- 2 files changed, 37 insertions(+), 7 deletions(-)[...]
Emil Tsalapatis
reviewerPATCH v2 4/4 selftests/bpf: add tests for the arena offset of globals
In reply to Emil TsalapatisAdd tests for the new libbpf globals arena offset logic. The tests cover all three cases: The globals being small enough to be placed at the maximum possible offset, being as large as the arena itself and being placed at the very beginning, and requiring an intermediate offset into the arena.
--- .../selftests/bpf/prog_tests/verifier.c | 6 ++ .../bpf/progs/verifier_arena_globals1.c | 60 ++++++++++++++++++ .../bpf/progs/verifier_arena_globals2.c | 49 +++++++++++++++ .../bpf/progs/verifier_arena_globals3.c | 61 +++++++++++++++++++ 4 files changed, 176 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_globals1.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_globals2.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_globals3.c
tools/testing/selftests/bpf/prog_tests/verifier.c +6 -0
@@ -6,6 +6,9 @@#include "verifier_and.skel.h" #include "verifier_arena.skel.h" #include "verifier_arena_large.skel.h" +#include "verifier_arena_globals1.skel.h" +#include "verifier_arena_globals2.skel.h" +#include "verifier_arena_globals3.skel.h" #include "verifier_array_access.skel.h" #include "verifier_async_cb_context.skel.h" #include "verifier_basic_stack.skel.h"
@@ -147,6 +150,9 @@ static void run_tests_aux(const char *skel_name,void test_verifier_and (void ) {RUN (verifier_and); }void test_verifier_arena (void ) {RUN (verifier_arena); }void test_verifier_arena_large (void ) {RUN (verifier_arena_large); } +void test_verifier_arena_globals1 (void ) {RUN (verifier_arena_globals1); } +void test_verifier_arena_globals2 (void ) {RUN (verifier_arena_globals2); } +void test_verifier_arena_globals3 (void ) {RUN (verifier_arena_globals3); }void test_verifier_basic_stack (void ) {RUN (verifier_basic_stack); }void test_verifier_bitfield_write (void ) {RUN (verifier_bitfield_write); }void test_verifier_bounds (void ) {RUN (verifier_bounds); }
tools/testing/selftests/bpf/progs/verifier_arena_globals1.c new file +60 -0
60 lines not on this page · raw message
tools/testing/selftests/bpf/progs/verifier_arena_globals2.c new file +49 -0
49 lines not on this page · raw message
tools/testing/selftests/bpf/progs/verifier_arena_globals3.c new file +61 -0
61 lines not on this page · raw message
Signature · 1 line
2.49.0