bpf_prog_pack followup

Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page).

Signed-off-by Song Liu <[email protected]>

---
 arch/x86/kernel/module.c     | 21 +++++++++++++++++++++
 include/linux/moduleloader.h |  5 +++++
 kernel/module/main.c         |  8 ++++++++
 3 files changed, 34 insertions(+)

arch/x86/kernel/module.c +10 -0

index b98ffcf4d250..63f6a16c70dc 100644
@@ -86,6 +86,27 @@ void *module_alloc(unsigned long size)
 	return p;
 }
 
+void *module_alloc_huge(unsigned long size)
+{
+	gfp_t gfp_mask = GFP_KERNEL;
+	void *p;
+
+	if (PAGE_ALIGN(size) > MODULES_LEN)
+		return NULL;
+
+	p = __vmalloc_node_range(size, MODULE_ALIGN,
+				 MODULES_VADDR + get_module_load_offset(),

arch/x86/kernel/module.c +1 -0

+
 #ifdef CONFIG_X86_32
 int apply_relocate(Elf32_Shdr *sechdrs,
 		   const char *strtab,

include/linux/moduleloader.h +5 -0

index 9e09d11ffe5b..d34743a88938 100644
@@ -26,6 +26,11 @@ unsigned int arch_mod_section_prepend(struct module *mod, unsigned int section);
    sections.  Returns NULL on failure. */
 void *module_alloc(unsigned long size);
 
+/* Allocator used for allocating memory in module memory space. If size is
+ * greater than PMD_SIZE, allow using huge pages. Returns NULL on failure.
+ */
+void *module_alloc_huge(unsigned long size);
+
 /* Free memory returned from module_alloc. */
 void module_memfree(void *module_region);
 

kernel/module/main.c +8 -0

index fed58d30725d..349b2a8bd20f 100644
@@ -1613,6 +1613,14 @@ void * __weak module_alloc(unsigned long size)
 			NUMA_NO_NODE, __builtin_return_address(0));
 }
 
+void * __weak module_alloc_huge(unsigned long size)
+{
+	return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END,
+				    GFP_KERNEL, PAGE_KERNEL_EXEC,
+				    VM_FLUSH_RESET_PERMS | VM_ALLOW_HUGE_VMAP,
+				    NUMA_NO_NODE, __builtin_return_address(0));
+}
+
 bool __weak module_init_section(const char *name)
 {
 	return strstarts(name, ".init");

Signature · 1 line

2.30.2

Quoted 5 lines · Song Liu

Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page).

Signed-off-by Song Liu <[email protected]>

I see mm not Cc'd. I'd like review from them.

Quoted 30 lines

---
arch/x86/kernel/module.c | 21 +++++++++++++++++++++ include/linux/moduleloader.h | 5 +++++ kernel/module/main.c | 8 ++++++++

3 files changed, 34 insertions(+)

arch/x86/kernel/module.c +15 -0

index b98ffcf4d250..63f6a16c70dc 100644
@@ -86,6 +86,27 @@ void *module_alloc(unsigned long size)
 	return p;
 }
 
+void *module_alloc_huge(unsigned long size)
+{
+	gfp_t gfp_mask = GFP_KERNEL;
+	void *p;
+
+	if (PAGE_ALIGN(size) > MODULES_LEN)
+		return NULL;
+
+	p = __vmalloc_node_range(size, MODULE_ALIGN,
+				 MODULES_VADDR + get_module_load_offset(),
+				 MODULES_END, gfp_mask, PAGE_KERNEL,
+				 VM_DEFER_KMEMLEAK | VM_ALLOW_HUGE_VMAP,
+				 NUMA_NO_NODE, __builtin_return_address(0));
+	if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) {
+		vfree(p);

arch/x86/kernel/module.c void *module_alloc(unsigned long size) · line 108

Quoted 5 lines

arch/x86/kernel/module.c +5 -0

+		return NULL;
+	}
+
+	return p;
+}

1) When things like kernel/bpf/core.c start using a module alloc it is time to consider genearlizing this.

2) How we free is important, and each arch does something funky for this. This is not addressed here.

And yes I welcome generalizing generic module_alloc() too as suggested before. The concern on my part is the sloppiness this enables.

Quoted 17 lines · Luis Chamberlain

Quoted 15 lines · Song Liu

arch/x86/kernel/module.c +15 -0

+void *module_alloc_huge(unsigned long size)
+{
+	gfp_t gfp_mask = GFP_KERNEL;
+	void *p;
+
+	if (PAGE_ALIGN(size) > MODULES_LEN)
+		return NULL;
+
+	p = __vmalloc_node_range(size, MODULE_ALIGN,
+				 MODULES_VADDR + get_module_load_offset(),
+				 MODULES_END, gfp_mask, PAGE_KERNEL,
+				 VM_DEFER_KMEMLEAK | VM_ALLOW_HUGE_VMAP,
+				 NUMA_NO_NODE, __builtin_return_address(0));
+	if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) {
+		vfree(p);

Quoted 8 lines

arch/x86/kernel/module.c void *module_alloc(unsigned long size) · line 108

Quoted 5 lines

arch/x86/kernel/module.c +5 -0

+		return NULL;
+	}
+
+	return p;
+}

1) When things like kernel/bpf/core.c start using a module alloc it is time to consider genearlizing this.

I am not quite sure what the generalization would look like. IMHO, the ideal case would have:
a) A kernel_text_rw_allocator, similar to current module_alloc.
b) A kernel_text_ro_allocator, similar to current bpf_prog_pack_alloc. This is built on top of kernel_text_rw_allocator. Different allocations could share a page, thus it requires text_poke like support from the arch.
c) If the arch supports text_poke, kprobes, ftrace trampolines, and bpf trampolines should use kernel_text_ro_allocator.
d) Major archs should support CONFIG_ARCH_WANTS_MODULES_DATA_IN_VMALLOC, and they should use kernel_text_ro_allocator for module text.

Does this sound reasonable to you?

I tried to enable CONFIG_ARCH_WANTS_MODULES_DATA_IN_VMALLOC for x86_64, but that doesn't really work. Do we have plan to make this combination work?

Quoted 2 lines

2) How we free is important, and each arch does something funky for this. This is not addressed here.

How should we address this? IIUC, x86_64 just calls vfree.

Quoted 2 lines

And yes I welcome generalizing generic module_alloc() too as suggested before. The concern on my part is the sloppiness this enables.

One question I have is, does module_alloc (or kernel_text_*_allocator above) belong to module code, or mm code (maybe vmalloc)?

I am planning to let BPF trampoline use bpf_prog_pack on x86_64, which is another baby step of c) above.

This set is the second half of v4 [1].

Changes v4 => v5:
1. Rebase and resolve conflicts due to module.c split.
2. Update experiment results (below).

For our web service production benchmark, bpf_prog_pack on 4kB pages gives 0.5% to 0.7% more throughput than not using bpf_prog_pack. bpf_prog_pack on 2MB pages 0.6% to 0.9% more throughput than not using bpf_prog_pack. Note that 0.5% is a huge improvement for our fleet. I believe this is also significant for other companies with many thousand servers.

Update: Further experiments (suggested by Rick Edgecombe) showed that most of benefit on the web service benchmark came from less direct map fragmentation. The experiment is as follows:

Side A: 2MB bpf prog pack on a single 2MB page;
Side B: 2MB bpf prog pack on 512x 4kB pages;

The system only uses about 200kB for BPF programs, but 2MB is allocated for bpf_prog_pack (for both A and B). Therefore, direct map fragmentation caused by BPF programs is elminated, and we are only measuring the performance difference of 1x 2MB page vs. ~50 4kB pages (we only use about 50 out of the 512 pages). For these two sides, the difference in system throughput is within the noise. I also measured iTLB-load-misses caused by bpf programs, which is ~300/s for case A, and ~1600/s for case B. The overall iTLB-load-misses is about 1.5M/s on these hosts. Therefore, we can clearly see 2MB page reduces iTLB misses, but the difference is not enough to have visible impact on system throughput.

Of course, the impact of iTLB miss will be more significant for systems with more BPF programs loaded.

[1] https://lore.kernel.org/bpf/[email protected]/

Song Liu (5):
module: introduce module_alloc_huge
bpf: use module_alloc_huge for bpf_prog_pack
vmalloc: WARN for set_vm_flush_reset_perms() on huge pages
vmalloc: introduce huge_vmalloc_supported
bpf: simplify select_bpf_prog_pack_size

arch/x86/kernel/module.c     | 21 +++++++++++++++++++++
include/linux/moduleloader.h |  5 +++++
include/linux/vmalloc.h      |  7 +++++++
kernel/bpf/core.c            | 25 ++++++++++---------------
kernel/module/main.c         |  8 ++++++++
mm/vmalloc.c                 |  5 +++++
6 files changed, 56 insertions(+), 15 deletions(-)

--
2.30.2