Skip to contentSkip to conversation

bpf: Fix alignment of memory allocator objects on 32-bit archs

PATCH New Felix Fietkau 2 participants 2 messages Jul 17 BPF

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bpf: Fix alignment of memory allocator objects on 32-bit archs

The BPF memory allocator prefixes each object with a header that holds struct llist_node while the object sits on a free list, and a pointer to the owning bpf_mem_cache while it is allocated. The header size was sizeof(struct llist_node), so on 32-bit architectures the memory handed out was offset by 4 bytes from the 8-byte aligned kmalloc allocation.

The verifier assumes that allocated objects are 8-byte aligned and allows BPF_DW atomic instructions on their u64 fields, e.g. on values of non-preallocated hash maps, objects from bpf_obj_new() and local storage. On arm32 the JIT does not implement atomics, and the interpreter executes them via atomic64_*(), which are implemented with LDREXD/STREXD on ARMv6K and later. Those instructions require 8-byte aligned addresses; a misaligned address raises an alignment fault and crashes the kernel.

Pad the header to 8 bytes to preserve the alignment provided by kmalloc. Store the percpu pointer of percpu objects at the new header offset instead of assuming the header is pointer-sized, and update the copy of the header size used by the hash map memory usage accounting.

Fixes 7c8199e24fa0 ("bpf: Introduce any context BPF specific memory allocator.")
---
 kernel/bpf/hashtab.c  | 2 +-
 kernel/bpf/memalloc.c | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)
kernel/bpf/hashtab.c +1 -1
index 3dd9b4924ae4..900c1851d264 100644
@@ -2337,7 +2337,7 @@ static u64 htab_map_mem_usage(const struct bpf_map *map)
 		else if (!lru)
 			usage += sizeof(struct htab_elem *) * num_possible_cpus();
 	} else {
-#define LLIST_NODE_SZ sizeof(struct llist_node)
+#define LLIST_NODE_SZ ALIGN(sizeof(struct llist_node), 8)
 
 		num_entries = htab->use_percpu_counter ?
 					  percpu_counter_sum(&htab->pcount) :
kernel/bpf/memalloc.c +4 -4
index e9662db7198f..ca7555b1da34 100644
@@ -33,7 +33,7 @@
  * Every allocated objected is padded with extra 8 bytes that contains
  * struct llist_node.
  */
-#define LLIST_NODE_SZ sizeof(struct llist_node)
+#define LLIST_NODE_SZ ALIGN(sizeof(struct llist_node), 8)
 
 #define BPF_MEM_ALLOC_SIZE_MAX 4096
 
@@ -142,7 +142,7 @@ static struct llist_node notrace *__llist_del_first(struct llist_head *head)
 static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags)
 {
 	if (c->percpu_size) {
-		void __percpu **obj = kmalloc_node(c->percpu_size, flags, node);
+		void *obj = kmalloc_node(c->percpu_size, flags, node);
 		void __percpu *pptr = __alloc_percpu_gfp(c->unit_size, 8, flags);
 
 		if (!obj || !pptr) {
@@ -150,7 +150,7 @@ static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags)
 			kfree(obj);
 			return NULL;
 		}
-		obj[1] = pptr;
+		*(void __percpu **)(obj + LLIST_NODE_SZ) = pptr;
 		return obj;
 	}
 
@@ -257,7 +257,7 @@ static void alloc_bulk(struct bpf_mem_cache *c, int cnt, int node, bool atomic)
 static void free_one(void *obj, bool percpu)
 {
 	if (percpu)
-		free_percpu(((void __percpu **)obj)[1]);
+		free_percpu(*(void __percpu **)(obj + LLIST_NODE_SZ));
 
 	kfree(obj);
 }
Signature · 1 line
2.51.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] On 32-bit architectures, the verifier's hardcoded 8-byte load (BPF_DW) for bpf_this_cpu_ptr()/bpf_per_cpu_ptr() causes a KASAN out-of-bounds read past the 4-byte percpu pointer and passes garbage data on Big Endian systems. --

--- Patch [1]: [PATCH] bpf: Fix alignment of memory allocator objects on 32-bit archs --- commit ed5f9ae93bc0fc6463cab51b598b3ca3395812da
Author: Felix Fietkau <[email protected]>

bpf: Fix alignment of memory allocator objects on 32-bit archs

The BPF memory allocator prefixes each object with a header. On 32-bit architectures, the memory handed out was offset by 4 bytes. This patch pads the header to 8 bytes to preserve alignment and updates the percpu pointer offset accordingly.

Quoted 4 lines
kernel/bpf/memalloc.c +0 -0
index e9662db7198fe..ca7555b1da34f 100644

[ ... ]

Quoted 9 lines
kernel/bpf/memalloc.c +1 -1
@@ -142,7 +142,7 @@ static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags)
 {
 	if (c->percpu_size) {
-		void __percpu **obj = kmalloc_node(c->percpu_size, flags, node);
+		void *obj = kmalloc_node(c->percpu_size, flags, node);
 		void __percpu *pptr = __alloc_percpu_gfp(c->unit_size, 8, flags);
 
 		if (!obj || !pptr) {

kernel/bpf/memalloc.c static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags) · line 153

Quoted 5 lines
kernel/bpf/memalloc.c +1 -1
@@ -150,7 +150,7 @@ static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags)
 			kfree(obj);
 			return NULL;
 		}
-		obj[1] = pptr;
+		*(void __percpu **)(obj + LLIST_NODE_SZ) = pptr;

[Severity: High]
This is a pre-existing issue, but does this 32-bit pointer write lead to an out-of-bounds read when accessed by the verifier?

On 32-bit systems, the allocation size for percpu objects is typically 12 bytes (LLIST_NODE_SZ + sizeof(void *)). The percpu pointer is stored at offset 8 using a 32-bit store.

However, the verifier unconditionally patches bpf_per_cpu_ptr() and bpf_this_cpu_ptr() to read this pointer using an 8-byte load:

kernel/bpf/fixups.c:fixup_bpf_calls() {
    ...
    insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0);
    ...
}

Could this 64-bit read access bytes 8-15 of the 12-byte allocation, causing a KASAN slab-out-of-bounds read?

Furthermore, on 32-bit Big Endian systems, does this 8-byte load place the 4-byte pointer into the upper 32 bits and the out-of-bounds garbage into the lower 32 bits, leading to memory corruption or a kernel panic when the garbage is later dereferenced?

Quoted 2 lines
kernel/bpf/memalloc.c +0 -0
 		return obj;
 	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1