libbpf: move arena variables out of the zero
Subsystems (5)
Emil Tsalapatis
reviewerv2 cover libbpf: move arena variables out of the zero
Modify libbpf to place arena globals in a small offset inside the arena mapping instead of the very beginning. This allows programs to leave the "zero page" of the arena unmapped, so that NULL arena pointer dereferences trigger a page fault and associated backtrace in BPF streams. In contrast, the current policy of placing global data in the zero pages means that NULL dereferences silently corrupt global data, e.g, arena qspinlock state. This makes arena bugs more difficult to debug.
The patchset adds code to libbpf to move global arena data to the end of the arena. At load time, libbpf adjusts each symbol's location within the arena to point to the right location in the arena. The patchset also adjusts the arena skeleton pointer to point to the arena globals, now that they are not in the beginning of the arena region.
CHANGESET
=========
v1->v2: (https://lore.kernel.org/bpf/[email protected])
- Moved globals to the end of the mapping: (Andrii)
- Removed extra parameter for offset and parameter picking logic
- Removed padding in the skeleton
- Removed additional libbpf call
- Added Reviewed-by from Eduard on patch 1
Emil Tsalapatis (4):
selftests/bpf: explicitly account for globals in verifier_arena_large bpf/verifier: do not limit maximum direct offset into arena map
libbpf: move arena globals to the end of the arena selftests/bpf: add tests for the arena offset of globals
kernel/bpf/verifier.c | 8 +-- tools/lib/bpf/libbpf.c | 19 ++++-- .../selftests/bpf/prog_tests/verifier.c | 4 ++ .../bpf/progs/verifier_arena_globals1.c | 58 +++++++++++++++++++ .../bpf/progs/verifier_arena_globals2.c | 49 ++++++++++++++++ .../bpf/progs/verifier_arena_large.c | 21 +++++-- 6 files changed, 147 insertions(+), 12 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_globals1.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_globals2.c
Signature · 1 line
2.49.0
Emil Tsalapatis
reviewerPATCH v2 1/4 selftests/bpf: explicitly account for globals in verifier_arena_large
The big_alloc1 test in verifier_arena_large assumes that the arena base and the first page allocated by bpf_arena_alloc_pages are identical. This is not the case, because the first page in the arena is populated by global arena data. The test still passes because the code makes the tacit assumption that the first page is on offset PAGE_SIZE instead of 0.
Make this distinction explicit in the code, and adjust the page offsets requested during the test to count from the beginning of the arena instead of using the address of the first allocated page.
--- .../selftests/bpf/progs/verifier_arena_large.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/progs/verifier_arena_large.c +11 -4
@@ -23,18 +23,25 @@ int big_alloc1(void *ctx) {#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) volatile char __arena *page1, *page2, *no_page, *page3; - void __arena *base; +u64 base; - page1 = base =bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 ); + base = (u64 )arena_base (&arena); + + page1 =bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 );if (!page1)return 1 ; + +/* Account for global arena data. */ +if ((u64 )page1 != base +PAGE_SIZE ) +return 15 ; + *page1 =1 ; - page2 =bpf_arena_alloc_pages (&arena, base +ARENA_SIZE -PAGE_SIZE *2 , + page2 = bpf_arena_alloc_pages(&arena, (void __arena *)(ARENA_SIZE - PAGE_SIZE),1 ,NUMA_NO_NODE ,0 );if (!page2)return 2 ; *page2 =2 ; - no_page =bpf_arena_alloc_pages (&arena, base +ARENA_SIZE -PAGE_SIZE , + no_page = bpf_arena_alloc_pages(&arena, (void __arena *)ARENA_SIZE,1 ,NUMA_NO_NODE ,0 );if (no_page)return 3 ;
Signature · 1 line
2.49.0
Emil Tsalapatis
reviewerPATCH v2 2/4 bpf/verifier: do not limit maximum direct offset into arena map
In reply to Emil TsalapatisThe verifier currently limits direct offsets into a map to 512MiB to avoid overflow during pointer arithmetic. However, this prevents arena maps from using direct addressing instructions to access data at the end of > 512MiB arena maps. This is necessary when moving arena globals to the end of the arena instead of the front.
Relax the limitation for direct offsets into arena maps to 4GiB, the maximum arena size.
--- kernel/bpf/verifier.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/bpf/verifier.c +4 -4
@@ -21084,13 +21084,13 @@ static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env) }else {u32 off = insn[1 ].imm; -if (off >=BPF_MAX_VAR_OFF ) { -verbose (env,"direct value offset of %u is not allowed\n" , off); +if (!map->ops->map_direct_value_addr) { +verbose (env,"no direct value access support for this map type\n" );return -EINVAL ; } -if (!map->ops->map_direct_value_addr) { -verbose (env,"no direct value access support for this map type\n" ); +if (off >=BPF_MAX_VAR_OFF && map->map_type !=BPF_MAP_TYPE_ARENA ) { +verbose (env,"direct value offset of %u is not allowed\n" , off);return -EINVAL ; }
Signature · 1 line
2.49.0
Re: [PATCH v2 2/4] bpf/verifier: do not limit maximum direct offset into arena map
Quoted 26 lines · Emil Tsalapatis
The verifier currently limits direct offsets into a map to 512MiB to avoid overflow during pointer arithmetic. However, this prevents arena maps from using direct addressing instructions to access data at the end of > 512MiB arena maps. This is necessary when moving arena globals to the end of the arena instead of the front.
Relax the limitation for direct offsets into arena maps to 4GiB, the maximum arena size.
Signed-off-by Emil Tsalapatis <[email protected]>--- kernel/bpf/verifier.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)kernel/bpf/verifier.c +1
-2@@ -21084,13 +21084,13 @@ static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env) }else {u32 off = insn[1 ].imm; -if (off >=BPF_MAX_VAR_OFF ) { -verbose (env,"direct value offset of %u is not allowed\n" , off); +if (!map->ops->map_direct_value_addr) {
kernel/bpf/verifier.c static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env) · line 21095
Quoted 10 lines
kernel/bpf/verifier.c +3
-2+verbose (env,"no direct value access support for this map type\n" );return -EINVAL ; } -if (!map->ops->map_direct_value_addr) { -verbose (env,"no direct value access support for this map type\n" ); +if (off >=BPF_MAX_VAR_OFF && map->map_type !=BPF_MAP_TYPE_ARENA ) { +verbose (env,"direct value offset of %u is not allowed\n" , off);return -EINVAL ; }
both arena and array maps validate off for correct range, not sure we even need this check here?
Quoted 4 lines
kernel/bpf/verifier.c +0
-1--2.49.0
Emil Tsalapatis
reviewerPATCH v2 3/4 libbpf: move arena globals to the end of the arena
In reply to Emil TsalapatisArena globals are currently placed at the beginning of the arena by libbpf. This is convenient, but prevents users from reserving guard pages in the beginning of the arena to identify NULL pointer dereferences. Adjust the load logic to place the globals at the end of the arena instead.
Also modify bpftool to set the arena pointer in the program's BPF skeleton to point to the globals. Users now call bpf_map__initial_value() to find the beginning of the arena mapping and use the arena pointer in the skeleton to determine which part of the mapping holds the arena globals and which part is free.
--- tools/lib/bpf/libbpf.c | 19 +++++++++++++++---- .../bpf/progs/verifier_arena_large.c | 12 +++++++++--- 2 files changed, 24 insertions(+), 7 deletions(-)
tools/lib/bpf/libbpf.c +15 -4
@@ -757,6 +757,7 @@ struct bpf_object {int arena_map_idx;void *arena_data;size_t arena_data_sz; +__u32 arena_data_off;void *jumptables_data;size_t jumptables_data_sz;
@@ -2991,10 +2992,11 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map, void *data, size_t data_sz) {const long page_sz =sysconf (_SC_PAGE_SIZE); +const size_t data_alloc_sz =roundup (data_sz, page_sz);size_t mmap_sz; mmap_sz =bpf_map_mmap_sz (map); -if (roundup (data_sz, page_sz) > mmap_sz) { +if (data_alloc_sz > mmap_sz) {pr_warn ("elf: sec '%s': declared ARENA map size (%zu) is too small to hold global __arena variables of size %zu\n" , sec_name, mmap_sz, data_sz);return -E2BIG ;
@@ -3006,6 +3008,9 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map,memcpy (obj->arena_data, data, data_sz); obj->arena_data_sz = data_sz; +/* place globals at the end of the arena */ + obj->arena_data_off = mmap_sz - data_alloc_sz; +/* make bpf_map__init_value() work for ARENA maps */ map->mmaped = obj->arena_data;
@@ -4663,7 +4668,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog, reloc_desc->type =RELO_DATA ; reloc_desc->insn_idx = insn_idx; reloc_desc->map_idx = obj->arena_map_idx; - reloc_desc->sym_off = sym->st_value; + reloc_desc->sym_off = sym->st_value + obj->arena_data_off; map = &obj->maps[obj->arena_map_idx]; pr_debug("prog '%s': found arena map %d (%s, sec %d, off %zu) for insn %u\n",
@@ -5624,7 +5629,8 @@ bpf_object__create_maps(struct bpf_object *obj)return err; }if (obj->arena_data) { -memcpy (map->mmaped, obj->arena_data, obj->arena_data_sz); +memcpy (map->mmaped + obj->arena_data_off, obj->arena_data, + obj->arena_data_sz);zfree (&obj->arena_data); } }
@@ -14387,6 +14393,7 @@ void bpf_object__destroy_subskeleton(struct bpf_object_subskeleton *s)int bpf_object__load_skeleton (struct bpf_object_skeleton *s) { +void *mmaped;int i, err; err =bpf_object__load (*s->obj);
@@ -14402,7 +14409,11 @@ int bpf_object__load_skeleton(struct bpf_object_skeleton *s)if (!map_skel->mmaped)continue ; - *map_skel->mmaped = map->mmaped; + mmaped = map->mmaped; +if (map->def.type ==BPF_MAP_TYPE_ARENA ) + mmaped += map->obj->arena_data_off; + + *map_skel->mmaped = mmaped; }return 0 ;
tools/testing/selftests/bpf/progs/verifier_arena_large.c +9 -3
@@ -31,16 +31,22 @@ int big_alloc1(void *ctx)if (!page1)return 1 ; -/* Account for global arena data. */ -if ((u64 )page1 != base +PAGE_SIZE ) +if ((u64 )page1 != base)return 15 ; *page1 =1 ; - page2 = bpf_arena_alloc_pages(&arena, (void __arena *)(ARENA_SIZE - PAGE_SIZE), + page2 = bpf_arena_alloc_pages(&arena, (void __arena *)(ARENA_SIZE - 2 * PAGE_SIZE),1 ,NUMA_NO_NODE ,0 );if (!page2)return 2 ; *page2 =2 ; + +/* Test for the guard region at the end of the arena. */ + no_page = bpf_arena_alloc_pages(&arena, (void __arena *)ARENA_SIZE - PAGE_SIZE, +1 ,NUMA_NO_NODE ,0 ); +if (no_page) +return 16 ; + no_page = bpf_arena_alloc_pages(&arena, (void __arena *)ARENA_SIZE,1 ,NUMA_NO_NODE ,0 );if (no_page)
Signature · 1 line
2.49.0
[email protected]
AI reviewRe: [PATCH v2 3/4] libbpf: move arena globals to the end of the arena
Here obj->arena_data_off is a __u32, but the calculation is done in size_t (64-bit). For a 4GB arena (which the test verifier_arena_large.c uses), mmap_sz is 4GB and the result can approach 4GB…
Quoted 4 lines
tools/lib/bpf/libbpf.c +0
-0
[ ... ]
Quoted 14 lines
tools/lib/bpf/libbpf.c +2
-1@@ -2991,10 +2992,11 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map, void *data, size_t data_sz) {const long page_sz =sysconf (_SC_PAGE_SIZE); +const size_t data_alloc_sz =roundup (data_sz, page_sz);size_t mmap_sz; mmap_sz =bpf_map_mmap_sz (map); -if (roundup (data_sz, page_sz) > mmap_sz) { +if (data_alloc_sz > mmap_sz) {pr_warn ("elf: sec '%s': declared ARENA map size (%zu) is too small to hold global __arena variables of size %zu\n" , sec_name, mmap_sz, data_sz);return -E2BIG ;
tools/lib/bpf/libbpf.c static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map, · line 3013
Quoted 6 lines
tools/lib/bpf/libbpf.c +3
-0@@ -3006,6 +3008,9 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map,memcpy (obj->arena_data, data, data_sz); obj->arena_data_sz = data_sz; +/* place globals at the end of the arena */ + obj->arena_data_off = mmap_sz - data_alloc_sz; +
Here obj->arena_data_off is a __u32, but the calculation is done in size_t (64-bit). For a 4GB arena (which the test verifier_arena_large.c uses), mmap_sz is 4GB and the result can approach 4GB - PAGE_SIZE. This fits in u32 correctly.
tools/lib/bpf/libbpf.c static int bpf_program__record_reloc(struct bpf_program *prog, · line 4671
Quoted 6 lines
tools/lib/bpf/libbpf.c +1
-1@@ -4663,7 +4668,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog, reloc_desc->type =RELO_DATA ; reloc_desc->insn_idx = insn_idx; reloc_desc->map_idx = obj->arena_map_idx; - reloc_desc->sym_off = sym->st_value; + reloc_desc->sym_off = sym->st_value + obj->arena_data_off;
^^^^^^^^^^^^^^^^^^^^^^
Can this overflow when storing to reloc_desc->sym_off? The sym_off field is 'int' (signed 32-bit), but arena_data_off can be larger than INT_MAX for arenas approaching 4GB in size.
For example, with a 4GB arena and one page of globals:
- mmap_sz = 4GB = 0x100000000
- data_alloc_sz = 4096
- arena_data_off = 0xFFFFF000 = 4294963200
This value (4294963200) exceeds INT_MAX (2147483647), so when stored in the signed int sym_off field, it becomes negative.
Later in bpf_object__relocate_data(), this is used as:
insn[1].imm = insn[0].imm + relo->sym_off;
With sym_off being negative due to overflow, this produces an incorrect offset for accessing arena globals.
Should sym_off be changed to an unsigned type to handle large arena offsets correctly?
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/19901383089
Re: [PATCH v2 3/4] libbpf: move arena globals to the end of the arena
Quoted 54 lines · On Wed, Dec 3, 2025 at 8:52 AM <[email protected]> wrote:
Quoted 5 lines
tools/lib/bpf/libbpf.c +0
-0[ ... ]
Quoted 14 lines
tools/lib/bpf/libbpf.c +2
-1@@ -2991,10 +2992,11 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map, void *data, size_t data_sz) {const long page_sz =sysconf (_SC_PAGE_SIZE); +const size_t data_alloc_sz =roundup (data_sz, page_sz);size_t mmap_sz; mmap_sz =bpf_map_mmap_sz (map); -if (roundup (data_sz, page_sz) > mmap_sz) { +if (data_alloc_sz > mmap_sz) {pr_warn ("elf: sec '%s': declared ARENA map size (%zu) is too small to hold global __arena variables of size %zu\n" , sec_name, mmap_sz, data_sz);return -E2BIG ;tools/lib/bpf/libbpf.c static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map, · line 3013
Quoted 6 lines
tools/lib/bpf/libbpf.c +3
-0@@ -3006,6 +3008,9 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map,memcpy (obj->arena_data, data, data_sz); obj->arena_data_sz = data_sz; +/* place globals at the end of the arena */ + obj->arena_data_off = mmap_sz - data_alloc_sz; +Here
obj->arena_data_offis a__u32, but the calculation is done insize_t(64-bit). For a 4GB arena (which the testverifier_arena_large.cuses),mmap_szis 4GB and the result can approach 4GB -PAGE_SIZE. This fits in u32 correctly.tools/lib/bpf/libbpf.c static int bpf_program__record_reloc(struct bpf_program *prog, · line 4671
Quoted 6 lines
tools/lib/bpf/libbpf.c +1
-1@@ -4663,7 +4668,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog, reloc_desc->type =RELO_DATA ; reloc_desc->insn_idx = insn_idx; reloc_desc->map_idx = obj->arena_map_idx; - reloc_desc->sym_off = sym->st_value; + reloc_desc->sym_off = sym->st_value + obj->arena_data_off;^^^^^^^^^^^^^^^^^^^^^^Can this overflow when storing to
reloc_desc->sym_off? Thesym_offfield is 'int' (signed 32-bit), butarena_data_offcan be larger thanINT_MAXfor arenas approaching 4GB in size.For example, with a 4GB arena and one page of globals:
-mmap_sz= 4GB =0x100000000
-data_alloc_sz= 4096
-arena_data_off=0xFFFFF000= 4294963200This value (4294963200) exceeds
INT_MAX(2147483647), so when stored in the signed intsym_offfield, it becomes negative.
Quoted 8 lines
Later in
bpf_object__relocate_data(), this is used as:insn[1].imm = insn[0].imm + relo->sym_off;With
sym_offbeing negative due to overflow, this produces an incorrect offset for accessing arena globals.Should
sym_offbe changed to an unsigned type to handle large arena offsets correctly?
AI has good a point, valid offset with 4GB arena will overflow int... But then ldimm64 instructions that allow direct access to arena have insn[1].imm as s32, so they cannot even address more than 2GB? Can you please double check this, maybe we actually treat insn[1].imm as u32 on the verifier side for this use case?
Quoted 9 lines
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/19901383089
Emil Tsalapatis
reviewerRe: [PATCH v2 3/4] libbpf: move arena globals to the end of the arena
Quoted 62 lines · Andrii Nakryiko
Quoted 60 lines · On Wed, Dec 3, 2025 at 8:52 AM <[email protected]> wrote:
Quoted 5 lines
tools/lib/bpf/libbpf.c +0
-0[ ... ]
Quoted 14 lines
tools/lib/bpf/libbpf.c +2
-1@@ -2991,10 +2992,11 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map, void *data, size_t data_sz) {const long page_sz =sysconf (_SC_PAGE_SIZE); +const size_t data_alloc_sz =roundup (data_sz, page_sz);size_t mmap_sz; mmap_sz =bpf_map_mmap_sz (map); -if (roundup (data_sz, page_sz) > mmap_sz) { +if (data_alloc_sz > mmap_sz) {pr_warn ("elf: sec '%s': declared ARENA map size (%zu) is too small to hold global __arena variables of size %zu\n" , sec_name, mmap_sz, data_sz);return -E2BIG ;tools/lib/bpf/libbpf.c static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map, · line 3013
Quoted 6 lines
tools/lib/bpf/libbpf.c +3
-0@@ -3006,6 +3008,9 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map,memcpy (obj->arena_data, data, data_sz); obj->arena_data_sz = data_sz; +/* place globals at the end of the arena */ + obj->arena_data_off = mmap_sz - data_alloc_sz; +Here
obj->arena_data_offis a__u32, but the calculation is done insize_t(64-bit). For a 4GB arena (which the testverifier_arena_large.cuses),mmap_szis 4GB and the result can approach 4GB -PAGE_SIZE. This fits in u32 correctly.tools/lib/bpf/libbpf.c static int bpf_program__record_reloc(struct bpf_program *prog, · line 4671
Quoted 6 lines
tools/lib/bpf/libbpf.c +1
-1@@ -4663,7 +4668,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog, reloc_desc->type =RELO_DATA ; reloc_desc->insn_idx = insn_idx; reloc_desc->map_idx = obj->arena_map_idx; - reloc_desc->sym_off = sym->st_value; + reloc_desc->sym_off = sym->st_value + obj->arena_data_off;^^^^^^^^^^^^^^^^^^^^^^Can this overflow when storing to
reloc_desc->sym_off? Thesym_offfield is 'int' (signed 32-bit), butarena_data_offcan be larger thanINT_MAXfor arenas approaching 4GB in size.For example, with a 4GB arena and one page of globals:
-mmap_sz= 4GB =0x100000000
-data_alloc_sz= 4096
-arena_data_off=0xFFFFF000= 4294963200This value (4294963200) exceeds
INT_MAX(2147483647), so when stored in the signed intsym_offfield, it becomes negative.Later in
bpf_object__relocate_data(), this is used as:insn[1].imm = insn[0].imm + relo->sym_off;With
sym_offbeing negative due to overflow, this produces an incorrect offset for accessing arena globals.
Quoted 9 lines
Quoted 2 lines
Should
sym_offbe changed to an unsigned type to handle large arena offsets correctly?AI has good a point, valid offset with 4GB arena will overflow int... But then ldimm64 instructions that allow direct access to arena have insn[1].imm as s32, so they cannot even address more than 2GB? Can you please double check this, maybe we actually treat insn[1].imm as u32 on the verifier side for this use case?
I can confirm we treat insn[1].imm as a u32, but the following relocation logic in
bpf_object__relocate_data looks suspect:
case RELO_DATA:
map = &obj->maps[relo->map_idx];
insn[1].imm = insn[0].imm + relo->sym_off;
if (obj->gen_loader) {If sym_off is interpreted as a negative, then insn[1].imm should be completely wrong and typecasting
back to u32 should do nothing. I added tests with both nonzero insn[0].imm and st_value, and none
of them are failing which unless I"m missing something is honestly kind of surprising.
I don't think sym_off should actually ever be negative. The st_value from the ELF headers
should be always positive, and elf_find_func_offset_* also returns only positive numbers.
Should we change sym_off in reloc_desc to be a u32? If anything that is more accurate than having it
as an int.
For the rest of the feedback: Ack, will incorporate it into the next patch.
Quoted 9 lines
Quoted 9 lines
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/19901383089
Re: [PATCH v2 3/4] libbpf: move arena globals to the end of the arena
In reply to Emil TsalapatisQuoted 25 lines · Emil Tsalapatis
Arena globals are currently placed at the beginning of the arena by libbpf. This is convenient, but prevents users from reserving guard pages in the beginning of the arena to identify NULL pointer dereferences. Adjust the load logic to place the globals at the end of the arena instead.
Also modify bpftool to set the arena pointer in the program's BPF skeleton to point to the globals. Users now call
bpf_map__initial_value()to find the beginning of the arena mapping and use the arena pointer in the skeleton to determine which part of the mapping holds the arena globals and which part is free.Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Emil Tsalapatis <[email protected]>--- tools/lib/bpf/libbpf.c | 19 +++++++++++++++---- .../bpf/progs/verifier_arena_large.c | 12 +++++++++--- 2 files changed, 24 insertions(+), 7 deletions(-)tools/lib/bpf/libbpf.c +0
-0
tools/lib/bpf/libbpf.c struct bpf_object { · line 760
Quoted 4 lines
tools/lib/bpf/libbpf.c +1
-0@@ -757,6 +757,7 @@ struct bpf_object {int arena_map_idx;void *arena_data;size_t arena_data_sz; +__u32 arena_data_off;
nit: use size_t, please
Quoted 54 lines
tools/lib/bpf/libbpf.c +9
-3void *jumptables_data;size_t jumptables_data_sz;@@ -2991,10 +2992,11 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map, void *data, size_t data_sz) {const long page_sz =sysconf (_SC_PAGE_SIZE); +const size_t data_alloc_sz =roundup (data_sz, page_sz);size_t mmap_sz; mmap_sz =bpf_map_mmap_sz (map); -if (roundup (data_sz, page_sz) > mmap_sz) { +if (data_alloc_sz > mmap_sz) {pr_warn ("elf: sec '%s': declared ARENA map size (%zu) is too small to hold global __arena variables of size %zu\n" , sec_name, mmap_sz, data_sz);return -E2BIG ;@@ -3006,6 +3008,9 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map,memcpy (obj->arena_data, data, data_sz); obj->arena_data_sz = data_sz; +/* place globals at the end of the arena */ + obj->arena_data_off = mmap_sz - data_alloc_sz; +/* make bpf_map__init_value() work for ARENA maps */ map->mmaped = obj->arena_data;@@ -4663,7 +4668,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog, reloc_desc->type =RELO_DATA ; reloc_desc->insn_idx = insn_idx; reloc_desc->map_idx = obj->arena_map_idx; - reloc_desc->sym_off = sym->st_value; + reloc_desc->sym_off = sym->st_value + obj->arena_data_off; map = &obj->maps[obj->arena_map_idx]; pr_debug("prog '%s': found arena map %d (%s, sec %d, off %zu) for insn %u\n",@@ -5624,7 +5629,8 @@ bpf_object__create_maps(struct bpf_object *obj)return err; }if (obj->arena_data) { -memcpy (map->mmaped, obj->arena_data, obj->arena_data_sz); +memcpy (map->mmaped + obj->arena_data_off, obj->arena_data, + obj->arena_data_sz);zfree (&obj->arena_data); } }@@ -14387,6 +14393,7 @@ void bpf_object__destroy_subskeleton(struct bpf_object_subskeleton *s)int bpf_object__load_skeleton (struct bpf_object_skeleton *s) { +void *mmaped;int i, err; err =bpf_object__load (*s->obj);
tools/lib/bpf/libbpf.c int bpf_object__load_skeleton(struct bpf_object_skeleton *s) · line 14416
Quoted 9 lines
tools/lib/bpf/libbpf.c +5
-1@@ -14402,7 +14409,11 @@ int bpf_object__load_skeleton(struct bpf_object_skeleton *s)if (!map_skel->mmaped)continue ; - *map_skel->mmaped = map->mmaped; + mmaped = map->mmaped; +if (map->def.type ==BPF_MAP_TYPE_ARENA ) + mmaped += map->obj->arena_data_off; + + *map_skel->mmaped = mmaped;
this is minor, but I think doing
if (map->def.type == BPF_MAP_TYPE_ARENA)
/* arena data is placed at the end of arena memory region */
*map_skel->mmaped = map->mmaped + map->obj->arena_data_off;
else
*map_skel->mmaped = map->mmaped;would be a bit cleaner and easier to follow and no need for extra mutable variable
Quoted 36 lines
tools/lib/bpf/libbpf.c +0
-0}return 0 ;tools/testing/selftests/bpf/progs/verifier_arena_large.c +9
-4@@ -31,16 +31,22 @@ int big_alloc1(void *ctx)if (!page1)return 1 ; -/* Account for global arena data. */ -if ((u64 )page1 != base +PAGE_SIZE ) +if ((u64 )page1 != base)return 15 ; *page1 =1 ; - page2 = bpf_arena_alloc_pages(&arena, (void __arena *)(ARENA_SIZE - PAGE_SIZE), + page2 = bpf_arena_alloc_pages(&arena, (void __arena *)(ARENA_SIZE - 2 * PAGE_SIZE),1 ,NUMA_NO_NODE ,0 );if (!page2)return 2 ; *page2 =2 ; + +/* Test for the guard region at the end of the arena. */ + no_page = bpf_arena_alloc_pages(&arena, (void __arena *)ARENA_SIZE - PAGE_SIZE, +1 ,NUMA_NO_NODE ,0 ); +if (no_page) +return 16 ; + no_page = bpf_arena_alloc_pages(&arena, (void __arena *)ARENA_SIZE,1 ,NUMA_NO_NODE ,0 );if (no_page) --2.49.0
Emil Tsalapatis
reviewerPATCH v2 4/4 selftests/bpf: add tests for the arena offset of globals
In reply to Emil TsalapatisAdd tests for the new libbpf globals arena offset logic. The tests cover all three cases: The globals being small enough to be placed at the maximum possible offset, being as large as the arena itself and being placed at the very beginning, and requiring an intermediate offset into the arena.
--- .../selftests/bpf/prog_tests/verifier.c | 4 ++ .../bpf/progs/verifier_arena_globals1.c | 58 +++++++++++++++++++ .../bpf/progs/verifier_arena_globals2.c | 49 ++++++++++++++++ 3 files changed, 111 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_globals1.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_globals2.c
tools/testing/selftests/bpf/prog_tests/verifier.c +4 -0
@@ -6,6 +6,8 @@#include "verifier_and.skel.h" #include "verifier_arena.skel.h" #include "verifier_arena_large.skel.h" +#include "verifier_arena_globals1.skel.h" +#include "verifier_arena_globals2.skel.h" #include "verifier_array_access.skel.h" #include "verifier_async_cb_context.skel.h" #include "verifier_basic_stack.skel.h"
@@ -147,6 +149,8 @@ static void run_tests_aux(const char *skel_name,void test_verifier_and (void ) {RUN (verifier_and); }void test_verifier_arena (void ) {RUN (verifier_arena); }void test_verifier_arena_large (void ) {RUN (verifier_arena_large); } +void test_verifier_arena_globals1 (void ) {RUN (verifier_arena_globals1); } +void test_verifier_arena_globals2 (void ) {RUN (verifier_arena_globals2); }void test_verifier_basic_stack (void ) {RUN (verifier_basic_stack); }void test_verifier_bitfield_write (void ) {RUN (verifier_bitfield_write); }void test_verifier_bounds (void ) {RUN (verifier_bounds); }
tools/testing/selftests/bpf/progs/verifier_arena_globals1.c new file +58 -0
58 lines not on this page · raw message
tools/testing/selftests/bpf/progs/verifier_arena_globals2.c new file +49 -0
49 lines not on this page · raw message
Signature · 1 line
2.49.0
Quoted 3 lines · Emil Tsalapatis
Modify libbpf to place arena globals in a small offset inside the arena mapping instead of the very beginning. This allows programs to leave
Description is now out of sync with the actual approach?
Quoted 44 lines
the "zero page" of the arena unmapped, so that NULL arena pointer dereferences trigger a page fault and associated backtrace in BPF streams. In contrast, the current policy of placing global data in the zero pages means that NULL dereferences silently corrupt global data, e.g, arena qspinlock state. This makes arena bugs more difficult to debug.
The patchset adds code to libbpf to move global arena data to the end of the arena. At load time, libbpf adjusts each symbol's location within the arena to point to the right location in the arena. The patchset also adjusts the arena skeleton pointer to point to the arena globals, now that they are not in the beginning of the arena region.
CHANGESET
=========
v1->v2: (https://lore.kernel.org/bpf/[email protected])- Moved globals to the end of the mapping: (Andrii)
- Removed extra parameter for offset and parameter picking logic
- Removed padding in the skeleton
- Removed additional libbpf call
- Added Reviewed-by from Eduard on patch 1Signed-off-by Emil Tsalapatis <[email protected]>Emil Tsalapatis (4):
selftests/bpf: explicitly account for globals inverifier_arena_largebpf/verifier: do not limit maximum direct offset into arena map
libbpf: move arena globals to the end of the arena selftests/bpf: add tests for the arena offset of globalskernel/bpf/verifier.c | 8 +-- tools/lib/bpf/libbpf.c | 19 ++++-- .../selftests/bpf/prog_tests/verifier.c | 4 ++ .../bpf/progs/verifier_arena_globals1.c | 58 +++++++++++++++++++ .../bpf/progs/verifier_arena_globals2.c | 49 ++++++++++++++++ .../bpf/progs/verifier_arena_large.c | 21 +++++-- 6 files changed, 147 insertions(+), 12 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_globals1.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_globals2.c--
2.49.0