vmalloc: bpf: introduce VM_ALLOW_HUGE_VMAP

mm/vmalloc.c +5 -0

+
+/**
+ * __vmalloc_huge - allocate virtually contiguous memory, allow huge pages
+ * @size:      allocation size
+ * @gfp_mask:  flags for the page level allocator

Use __vmalloc_huge() in alloc_large_system_hash() so that large system hash (>= PMD_SIZE) could benefit from huge pages. Note that __vmalloc_huge only allocates huge pages for systems with HAVE_ARCH_HUGE_VMALLOC.

Quoted 3 lines · Song Liu

Use __vmalloc_huge() in alloc_large_system_hash() so that large system hash (>= PMD_SIZE) could benefit from huge pages. Note that __vmalloc_huge only allocates huge pages for systems with HAVE_ARCH_HUGE_VMALLOC.

Looks good (modulo the possible naming chane suggested in patch 1):

Reviewed-by Christoph Hellwig <[email protected]>

Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page).

Signed-off-by Song Liu <[email protected]>

Quoted 4 lines · Song Liu

Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page).

Quoted 1 line

Signed-off-by Song Liu <[email protected]>

See modules-next [0], as modules.c has been chopped up as of late. So if you want this to go throug modules this will need to rebased on that tree. fortunately the amount of code in question does not seem like much.

[0] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git/log/?h=modules-next

Hi Luis,

Quoted 13 lines · Luis Chamberlain

Quoted 5 lines · Song Liu

Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page).

Signed-off-by Song Liu <[email protected]>

See modules-next [0], as modules.c has been chopped up as of late. So if you want this to go throug modules this will need to rebased on that tree. fortunately the amount of code in question does not seem like much.

Quoted 1 line

[0] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git/log/?h=modules-next

We are hoping to ship this with to 5.18, as the set addresses some issue with huge page backed vmalloc. I guess we cannot ship it via modules-next branch.

How about we ship module_alloc_huge() to 5.18 in module.c for now, and once we update modules-next branch, I will send another patch to clean it up?

Quoted 16 lines · Song Liu

Hi Luis,

Quoted 13 lines · Luis Chamberlain

Quoted 5 lines · Song Liu

Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page).

Signed-off-by Song Liu <[email protected]>

See modules-next [0], as modules.c has been chopped up as of late. So if you want this to go throug modules this will need to rebased on that tree. fortunately the amount of code in question does not seem like much.

Quoted 5 lines

Quoted 1 line

[0] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git/log/?h=modules-next

We are hoping to ship this with to 5.18, as the set addresses some issue with huge page backed vmalloc. I guess we cannot ship it via modules-next branch.

Huh, you intend this to go in as a fix for v5.18 (already released) once properly reviewed? This seems quite large... for a fix.

Quoted 2 lines

How about we ship module_alloc_huge() to 5.18 in module.c for now, and once we update modules-next branch, I will send another patch to clean it up?

I rather set the expectations right about getting such a large fix in for v5.18. I haven't even sat down to review all the changes in light of this, but a cursorary glance seems to me it's rather "large" for a fix.

Quoted 30 lines · Luis Chamberlain

Quoted 16 lines · Song Liu

Hi Luis,

Quoted 13 lines · Luis Chamberlain

Quoted 5 lines · Song Liu

Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page).

Signed-off-by Song Liu <[email protected]>

See modules-next [0], as modules.c has been chopped up as of late. So if you want this to go throug modules this will need to rebased on that tree. fortunately the amount of code in question does not seem like much.

Quoted 5 lines

Quoted 1 line

[0] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git/log/?h=modules-next

We are hoping to ship this with to 5.18, as the set addresses some issue with huge page backed vmalloc. I guess we cannot ship it via modules-next branch.

Huh, you intend this to go in as a fix for v5.18 (already released) once properly reviewed? This seems quite large... for a fix.

Quoted 2 lines

How about we ship module_alloc_huge() to 5.18 in module.c for now, and once we update modules-next branch, I will send another patch to clean it up?

Quoted 3 lines

I rather set the expectations right about getting such a large fix in for v5.18. I haven't even sat down to review all the changes in light of this, but a cursorary glance seems to me it's rather "large" for a fix.

Yes, I agree this is a little too big for a fix. I guess we can discuss whether some of the set need to wait until 5.19.

Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page).

Signed-off-by Song Liu <[email protected]>

---
 arch/x86/kernel/module.c     | 21 +++++++++++++++++++++
 include/linux/moduleloader.h |  5 +++++
 kernel/module.c              |  5 +++++
 3 files changed, 31 insertions(+)

arch/x86/kernel/module.c +21 -0

index b98ffcf4d250..63f6a16c70dc 100644
@@ -86,6 +86,27 @@ void *module_alloc(unsigned long size)
 	return p;
 }
 
+void *module_alloc_huge(unsigned long size)
+{
+	gfp_t gfp_mask = GFP_KERNEL;
+	void *p;
+
+	if (PAGE_ALIGN(size) > MODULES_LEN)
+		return NULL;
+
+	p = __vmalloc_node_range(size, MODULE_ALIGN,
+				 MODULES_VADDR + get_module_load_offset(),
+				 MODULES_END, gfp_mask, PAGE_KERNEL,
+				 VM_DEFER_KMEMLEAK | VM_ALLOW_HUGE_VMAP,
+				 NUMA_NO_NODE, __builtin_return_address(0));
+	if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) {
+		vfree(p);
+		return NULL;
+	}
+
+	return p;
+}
+
 #ifdef CONFIG_X86_32
 int apply_relocate(Elf32_Shdr *sechdrs,
 		   const char *strtab,

include/linux/moduleloader.h +5 -0

index 9e09d11ffe5b..d34743a88938 100644
@@ -26,6 +26,11 @@ unsigned int arch_mod_section_prepend(struct module *mod, unsigned int section);
    sections.  Returns NULL on failure. */
 void *module_alloc(unsigned long size);
 
+/* Allocator used for allocating memory in module memory space. If size is
+ * greater than PMD_SIZE, allow using huge pages. Returns NULL on failure.
+ */
+void *module_alloc_huge(unsigned long size);
+
 /* Free memory returned from module_alloc. */
 void module_memfree(void *module_region);
 

kernel/module.c +0 -0

index 6cea788fd965..b2c6cb682a7d 100644

Quoted 67 lines · Song Liu

Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page).

Signed-off-by Song Liu <[email protected]>

---
arch/x86/kernel/module.c | 21 +++++++++++++++++++++ include/linux/moduleloader.h | 5 +++++ kernel/module.c | 5 +++++

3 files changed, 31 insertions(+)

arch/x86/kernel/module.c +21 -0

index b98ffcf4d250..63f6a16c70dc 100644
@@ -86,6 +86,27 @@ void *module_alloc(unsigned long size)
 	return p;
 }
 
+void *module_alloc_huge(unsigned long size)
+{
+	gfp_t gfp_mask = GFP_KERNEL;
+	void *p;
+
+	if (PAGE_ALIGN(size) > MODULES_LEN)
+		return NULL;
+
+	p = __vmalloc_node_range(size, MODULE_ALIGN,
+				 MODULES_VADDR + get_module_load_offset(),
+				 MODULES_END, gfp_mask, PAGE_KERNEL,
+				 VM_DEFER_KMEMLEAK | VM_ALLOW_HUGE_VMAP,
+				 NUMA_NO_NODE, __builtin_return_address(0));
+	if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) {
+		vfree(p);
+		return NULL;
+	}
+
+	return p;
+}
+
 #ifdef CONFIG_X86_32
 int apply_relocate(Elf32_Shdr *sechdrs,
 		 const char *strtab,

include/linux/moduleloader.h +0 -0

index 9e09d11ffe5b..d34743a88938 100644
@@ -26,6 +26,11 @@ unsigned int arch_mod_section_prepend(struct module *mod, unsigned int section);

sections. Returns NULL on failure. */
void *module_alloc(unsigned long size);

include/linux/moduleloader.h +5 -0

+/* Allocator used for allocating memory in module memory space. If size is
+ * greater than PMD_SIZE, allow using huge pages. Returns NULL on failure.
+ */
+void *module_alloc_huge(unsigned long size);
+

/* Free memory returned from module_alloc. */
void module_memfree(void *module_region);

kernel/module.c +0 -0

index 6cea788fd965..b2c6cb682a7d 100644
@@ -2839,6 +2839,11 @@ void * __weak module_alloc(unsigned long size)
 			NUMA_NO_NODE, __builtin_return_address(0));
 }
 

kernel/module.c void * __weak module_alloc(unsigned long size) · line 2845

Quoted 4 lines

kernel/module.c +4 -0

+void * __weak module_alloc_huge(unsigned long size)
+{
+	return vmalloc_huge(size);
+}

Umm. This should use the same parameters as module_alloc except for also passing the new huge page flag.