bpf: Fix alignment of memory allocator objects on 32-bit archs
PATCH 1/1 bpf: Fix alignment of memory allocator objects on 32-bit archs
The BPF memory allocator prefixes each object with a header that holds struct llist_node while the object sits on a free list, and a pointer to the owning bpf_mem_cache while it is allocated. The header size was sizeof(struct llist_node), so on 32-bit architectures the memory handed out was offset by 4 bytes from the 8-byte aligned kmalloc allocation.
The verifier assumes that allocated objects are 8-byte aligned and allows BPF_DW atomic instructions on their u64 fields, e.g. on values of non-preallocated hash maps, objects from bpf_obj_new() and local storage. On arm32 the JIT does not implement atomics, and the interpreter executes them via atomic64_*(), which are implemented with LDREXD/STREXD on ARMv6K and later. Those instructions require 8-byte aligned addresses; a misaligned address raises an alignment fault and crashes the kernel.
Pad the header to 8 bytes to preserve the alignment provided by kmalloc. Store the percpu pointer of percpu objects at the new header offset instead of assuming the header is pointer-sized, and update the copy of the header size used by the hash map memory usage accounting.
--- kernel/bpf/hashtab.c | 2 +- kernel/bpf/memalloc.c | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-)
kernel/bpf/hashtab.c +1 -1
@@ -2337,7 +2337,7 @@ static u64 htab_map_mem_usage(const struct bpf_map *map)else if (!lru) usage +=sizeof (struct htab_elem *) *num_possible_cpus (); }else { -#define LLIST_NODE_SZ sizeof(struct llist_node) +#define LLIST_NODE_SZ ALIGN(sizeof(struct llist_node), 8) num_entries = htab->use_percpu_counter ? percpu_counter_sum(&htab->pcount) :
kernel/bpf/memalloc.c +4 -4
@@ -33,7 +33,7 @@* Every allocated objected is padded with extra 8 bytes that contains * struct llist_node. */ -#define LLIST_NODE_SZ sizeof(struct llist_node) +#define LLIST_NODE_SZ ALIGN(sizeof(struct llist_node), 8)#define BPF_MEM_ALLOC_SIZE_MAX 4096
@@ -142,7 +142,7 @@ static struct llist_node notrace *__llist_del_first(struct llist_head *head)static void *__alloc (struct bpf_mem_cache *c,int node,gfp_t flags) {if (c->percpu_size) { -void __percpu **obj =kmalloc_node (c->percpu_size, flags, node); +void *obj =kmalloc_node (c->percpu_size, flags, node);void __percpu *pptr =__alloc_percpu_gfp (c->unit_size,8 , flags);if (!obj || !pptr) {
@@ -150,7 +150,7 @@ static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags)kfree (obj);return NULL ; } - obj[1 ] = pptr; + *(void __percpu **)(obj +LLIST_NODE_SZ ) = pptr;return obj; }
@@ -257,7 +257,7 @@ static void alloc_bulk(struct bpf_mem_cache *c, int cnt, int node, bool atomic)static void free_one (void *obj,bool percpu) {if (percpu) -free_percpu (((void __percpu **)obj)[1 ]); +free_percpu (*(void __percpu **)(obj +LLIST_NODE_SZ ));kfree (obj); }
Signature · 1 line
2.51.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] On 32-bit architectures, the verifier's hardcoded 8-byte load (BPF_DW) for bpf_this_cpu_ptr()/bpf_per_cpu_ptr() causes a KASAN out-of-bounds read past the 4-byte percpu pointer and passes garbage data on Big Endian systems. --
--- Patch [1]: [PATCH] bpf: Fix alignment of memory allocator objects on 32-bit archs --- commit ed5f9ae93bc0fc6463cab51b598b3ca3395812da
Author: Felix Fietkau <[email protected]>
bpf: Fix alignment of memory allocator objects on 32-bit archs
The BPF memory allocator prefixes each object with a header. On 32-bit architectures, the memory handed out was offset by 4 bytes. This patch pads the header to 8 bytes to preserve alignment and updates the percpu pointer offset accordingly.
Quoted 4 lines
kernel/bpf/memalloc.c +0
-0
[ ... ]
Quoted 9 lines
kernel/bpf/memalloc.c +1
-1@@ -142,7 +142,7 @@ static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags) {if (c->percpu_size) { -void __percpu **obj =kmalloc_node (c->percpu_size, flags, node); +void *obj =kmalloc_node (c->percpu_size, flags, node);void __percpu *pptr =__alloc_percpu_gfp (c->unit_size,8 , flags);if (!obj || !pptr) {
kernel/bpf/memalloc.c static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags) · line 153
Quoted 5 lines
kernel/bpf/memalloc.c +1
-1@@ -150,7 +150,7 @@ static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags)kfree (obj);return NULL ; } - obj[1 ] = pptr; + *(void __percpu **)(obj +LLIST_NODE_SZ ) = pptr;
[Severity: High]
This is a pre-existing issue, but does this 32-bit pointer write lead to an out-of-bounds read when accessed by the verifier?
On 32-bit systems, the allocation size for percpu objects is typically 12 bytes (LLIST_NODE_SZ + sizeof(void *)). The percpu pointer is stored at offset 8 using a 32-bit store.
However, the verifier unconditionally patches bpf_per_cpu_ptr() and bpf_this_cpu_ptr() to read this pointer using an 8-byte load:
kernel/bpf/fixups.c:fixup_bpf_calls() {
...
insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0);
...
}Could this 64-bit read access bytes 8-15 of the 12-byte allocation, causing a KASAN slab-out-of-bounds read?
Furthermore, on 32-bit Big Endian systems, does this 8-byte load place the 4-byte pointer into the upper 32 bits and the out-of-bounds garbage into the lower 32 bits, leading to memory corruption or a kernel panic when the garbage is later dereferenced?
Quoted 2 lines
kernel/bpf/memalloc.c +0
-0return obj; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1