vmalloc: bpf: introduce VM_ALLOW_HUGE_VMAP

PATCH Superseded Song Liu 2 participants 6 messages Apr 11, 2022 – Apr 13, 2022

Subsystems (11)

Networking Network drivers Memory management x86 +7 more subsystems

lore

Share & tools

lore

Song Liu

author reviewer

2022-04-11 23:35 UTC #0 lore raw

v2 cover vmalloc: bpf: introduce VM_ALLOW_HUGE_VMAP

Changes v1 => v2:
1. Add vmalloc_huge(). (Christoph Hellwig)
2. Add module_alloc_huge(). (Christoph Hellwig)
3. Add Fixes tag and Link tag. (Thorsten Leemhuis)

Enabling HAVE_ARCH_HUGE_VMALLOC on x86_64 and use it for bpf_prog_pack has caused some issues [1], as many users of vmalloc are not yet ready to handle huge pages. To enable a more smooth transition to use huge page backed vmalloc memory, this set replaces VM_NO_HUGE_VMAP flag with an new opt-in flag, VM_ALLOW_HUGE_VMAP. More discussions about this topic can be found at [2].

Patch 1 removes VM_NO_HUGE_VMAP and adds VM_ALLOW_HUGE_VMAP. Patch 2 uses VM_ALLOW_HUGE_VMAP in bpf_prog_pack.

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/

Song Liu (3):
vmalloc: replace VM_NO_HUGE_VMAP with VM_ALLOW_HUGE_VMAP
module: introduce module_alloc_huge
bpf: use vmalloc with VM_ALLOW_HUGE_VMAP for bpf_prog_pack

 arch/Kconfig                 |  6 ++----
 arch/powerpc/kernel/module.c |  2 +-
 arch/s390/kvm/pv.c           |  2 +-
 arch/x86/kernel/module.c     | 21 +++++++++++++++++++
 include/linux/moduleloader.h |  5 +++++
 include/linux/vmalloc.h      |  4 ++--
 kernel/bpf/core.c            |  9 +++++----
 kernel/module.c              |  8 ++++++++
 mm/vmalloc.c                 | 39 +++++++++++++++++++-----------------
 9 files changed, 66 insertions(+), 30 deletions(-)

--
2.30.2

Song Liu

author reviewer

2022-04-11 23:35 UTC #1 lore raw

PATCH v2 1/3 vmalloc: replace VM_NO_HUGE_VMAP with VM_ALLOW_HUGE_VMAP

Huge page backed vmalloc memory could benefit performance in many cases. Since some users of vmalloc may not be ready to handle huge pages, VM_NO_HUGE_VMAP was introduced to allow vmalloc users to opt-out huge pages. However, it is not easy to add VM_NO_HUGE_VMAP to all the users that may try to allocate >= PMD_SIZE pages, but are not ready to handle huge pages properly.

Replace VM_NO_HUGE_VMAP with an opt-in flag, VM_ALLOW_HUGE_VMAP, so that users that benefit from huge pages could ask specificially.

Also, replace vmalloc_no_huge() with opt-in helper vmalloc_huge().

Fixes fac54e2bfb5b ("x86/Kconfig: Select HAVE_ARCH_HUGE_VMALLOC with

HAVE_ARCH_HUGE_VMAP")

Link https://lore.kernel.org/netdev/[email protected]/" open here

Signed-off-by Song Liu <[email protected]>

---
 arch/Kconfig                 |  6 ++----
 arch/powerpc/kernel/module.c |  2 +-
 arch/s390/kvm/pv.c           |  2 +-
 include/linux/vmalloc.h      |  4 ++--
 mm/vmalloc.c                 | 39 +++++++++++++++++++-----------------
 5 files changed, 27 insertions(+), 26 deletions(-)

arch/Kconfig +2 -4

index 29b0167c088b..31c4fdc4a4ba 100644
@@ -854,10 +854,8 @@ config HAVE_ARCH_HUGE_VMAP
 
 #
 #  Archs that select this would be capable of PMD-sized vmaps (i.e.,
-#  arch_vmap_pmd_supported() returns true), and they must make no assumptions
-#  that vmalloc memory is mapped with PAGE_SIZE ptes. The VM_NO_HUGE_VMAP flag
-#  can be used to prohibit arch-specific allocations from using hugepages to
-#  help with this (e.g., modules may require it).
+#  arch_vmap_pmd_supported() returns true). The VM_ALLOW_HUGE_VMAP flag
+#  must be used to enable allocations to use hugepages.
 #
 config HAVE_ARCH_HUGE_VMALLOC
 	depends on HAVE_ARCH_HUGE_VMAP

arch/powerpc/kernel/module.c +1 -1

index 40a583e9d3c7..97a76a8619fb 100644
@@ -101,7 +101,7 @@ __module_alloc(unsigned long size, unsigned long start, unsigned long end, bool
 	 * too.
 	 */
 	return __vmalloc_node_range(size, 1, start, end, gfp, prot,
-				    VM_FLUSH_RESET_PERMS | VM_NO_HUGE_VMAP,
+				    VM_FLUSH_RESET_PERMS,
 				    NUMA_NO_NODE, __builtin_return_address(0));
 }

arch/s390/kvm/pv.c +1 -1

index 7f7c0d6af2ce..8afede243903 100644
@@ -142,7 +142,7 @@ static int kvm_s390_pv_alloc_vm(struct kvm *kvm)
 	 * using large pages for the virtual memory area.
 	 * This is a hardware limitation.
 	 */
-	kvm->arch.pv.stor_var = vmalloc_no_huge(vlen);
+	kvm->arch.pv.stor_var = vmalloc(vlen);
 	if (!kvm->arch.pv.stor_var)
 		goto out_err;
 	return 0;

include/linux/vmalloc.h +2 -2

index 3b1df7da402d..1024517d7937 100644
@@ -26,7 +26,7 @@ struct notifier_block;		/* in notifier.h */
 #define VM_KASAN		0x00000080      /* has allocated kasan shadow memory */
 #define VM_FLUSH_RESET_PERMS	0x00000100	/* reset direct map and flush TLB on unmap, can't be freed in atomic context */
 #define VM_MAP_PUT_PAGES	0x00000200	/* put pages and free array in vfree */
-#define VM_NO_HUGE_VMAP		0x00000400	/* force PAGE_SIZE pte mapping */
+#define VM_ALLOW_HUGE_VMAP	0x00000400      /* Allow for huge pages on archs with HAVE_ARCH_HUGE_VMALLOC */
 
 #if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
 	!defined(CONFIG_KASAN_VMALLOC)

@@ -153,7 +153,7 @@ extern void *__vmalloc_node_range(unsigned long size, unsigned long align,
 			const void *caller) __alloc_size(1);
 void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask,
 		int node, const void *caller) __alloc_size(1);
-void *vmalloc_no_huge(unsigned long size) __alloc_size(1);
+extern void *vmalloc_huge(unsigned long size) __alloc_size(1);
 
 extern void *__vmalloc_array(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2);
 extern void *vmalloc_array(size_t n, size_t size) __alloc_size(1, 2);

mm/vmalloc.c +21 ~~-18~~

index e163372d3967..7cc2be6a7554 100644
@@ -3106,7 +3106,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align,
 		return NULL;
 	}
 
-	if (vmap_allow_huge && !(vm_flags & VM_NO_HUGE_VMAP)) {
+	if (vmap_allow_huge && (vm_flags & VM_ALLOW_HUGE_VMAP)) {
 		unsigned long size_per_node;
 
 		/*

@@ -3272,23 +3272,6 @@ void *vmalloc(unsigned long size)
 }
 EXPORT_SYMBOL(vmalloc);
 
-/**
- * vmalloc_no_huge - allocate virtually contiguous memory using small pages
- * @size:    allocation size
- *
- * Allocate enough non-huge pages to cover @size from the page level
- * allocator and map them into contiguous kernel virtual space.
- *
- * Return: pointer to the allocated memory or %NULL on error
- */
-void *vmalloc_no_huge(unsigned long size)
-{
-	return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END,
-				    GFP_KERNEL, PAGE_KERNEL, VM_NO_HUGE_VMAP,
-				    NUMA_NO_NODE, __builtin_return_address(0));
-}
-EXPORT_SYMBOL(vmalloc_no_huge);
-
 /**
  * vzalloc - allocate virtually contiguous memory with zero fill
  * @size:    allocation size

@@ -3347,6 +3330,26 @@ void *vmalloc_node(unsigned long size, int node)
 }
 EXPORT_SYMBOL(vmalloc_node);
 
+/**
+ * vmalloc_huge - allocate virtually contiguous memory, allow huge pages
+ * @size:    allocation size
+ *
+ * Allocate enough pages to cover @size from the page level
+ * allocator and map them into contiguous kernel virtual space.
+ * If @size is greater than or equal to PMD_SIZE, allow using
+ * huge pages for the memory
+ *
+ * Return: pointer to the allocated memory or %NULL on error
+ */
+void *vmalloc_huge(unsigned long size)
+{
+	return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END,
+				    GFP_KERNEL, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP,
+				    NUMA_NO_NODE, __builtin_return_address(0));
+
+}
+EXPORT_SYMBOL(vmalloc_huge);
+
 /**
  * vzalloc_node - allocate memory on a specific node with zero fill
  * @size:	allocation size

Signature · 1 line

2.30.2

Song Liu

author reviewer

2022-04-11 23:35 UTC #2 lore raw

PATCH v2 2/3 module: introduce module_alloc_huge

In reply to Song Liu

Introduce module_alloc_huge, which allocates huge page backed memory in module memory space. The primary user of this memory is bpf_prog_pack (multiple BPF programs sharing a huge page).

Signed-off-by Song Liu <[email protected]>

---
 arch/x86/kernel/module.c     | 21 +++++++++++++++++++++
 include/linux/moduleloader.h |  5 +++++
 kernel/module.c              |  8 ++++++++
 3 files changed, 34 insertions(+)

arch/x86/kernel/module.c +21 -0

index b98ffcf4d250..63f6a16c70dc 100644
@@ -86,6 +86,27 @@ void *module_alloc(unsigned long size)
 	return p;
 }
 
+void *module_alloc_huge(unsigned long size)
+{
+	gfp_t gfp_mask = GFP_KERNEL;
+	void *p;
+
+	if (PAGE_ALIGN(size) > MODULES_LEN)
+		return NULL;
+
+	p = __vmalloc_node_range(size, MODULE_ALIGN,
+				 MODULES_VADDR + get_module_load_offset(),
+				 MODULES_END, gfp_mask, PAGE_KERNEL,
+				 VM_DEFER_KMEMLEAK | VM_ALLOW_HUGE_VMAP,
+				 NUMA_NO_NODE, __builtin_return_address(0));
+	if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) {
+		vfree(p);
+		return NULL;
+	}
+
+	return p;
+}
+
 #ifdef CONFIG_X86_32
 int apply_relocate(Elf32_Shdr *sechdrs,
 		   const char *strtab,

include/linux/moduleloader.h +5 -0

index 9e09d11ffe5b..d34743a88938 100644
@@ -26,6 +26,11 @@ unsigned int arch_mod_section_prepend(struct module *mod, unsigned int section);
    sections.  Returns NULL on failure. */
 void *module_alloc(unsigned long size);
 
+/* Allocator used for allocating memory in module memory space. If size is
+ * greater than PMD_SIZE, allow using huge pages. Returns NULL on failure.
+ */
+void *module_alloc_huge(unsigned long size);
+
 /* Free memory returned from module_alloc. */
 void module_memfree(void *module_region);

kernel/module.c +8 -0

index 6cea788fd965..2af20ac3209c 100644
@@ -2839,6 +2839,14 @@ void * __weak module_alloc(unsigned long size)
 			NUMA_NO_NODE, __builtin_return_address(0));
 }
 
+void * __weak module_alloc_huge(unsigned long size)
+{
+	return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END,
+				    GFP_KERNEL, PAGE_KERNEL_EXEC,
+				    VM_FLUSH_RESET_PERMS | VM_ALLOW_HUGE_VMAP,
+				    NUMA_NO_NODE, __builtin_return_address(0));
+}
+
 bool __weak module_init_section(const char *name)
 {
 	return strstarts(name, ".init");

Signature · 1 line

2.30.2

Song Liu

author reviewer

2022-04-11 23:35 UTC #3 lore raw

PATCH v2 3/3 bpf: use vmalloc with VM_ALLOW_HUGE_VMAP for bpf_prog_pack

In reply to Song Liu

Use __vmalloc_node_range with VM_ALLOW_HUGE_VMAP for bpf_prog_pack so that BPF programs sit on PMD_SIZE pages. This benefits system performance by reducing iTLB miss rate.

Signed-off-by Song Liu <[email protected]>

---
 kernel/bpf/core.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

kernel/bpf/core.c +5 -4

index 13e9dbeeedf3..fd45bdd80a75 100644
@@ -857,7 +857,7 @@ static size_t select_bpf_prog_pack_size(void)
 	void *ptr;
 
 	size = BPF_HPAGE_SIZE * num_online_nodes();
-	ptr = module_alloc(size);
+	ptr = module_alloc_huge(size);
 
 	/* Test whether we can get huge pages. If not just use PAGE_SIZE
 	 * packs.

@@ -881,7 +881,7 @@ static struct bpf_prog_pack *alloc_new_pack(void)
 		       GFP_KERNEL);
 	if (!pack)
 		return NULL;
-	pack->ptr = module_alloc(bpf_prog_pack_size);
+	pack->ptr = module_alloc_huge(bpf_prog_pack_size);
 	if (!pack->ptr) {
 		kfree(pack);
 		return NULL;

@@ -889,7 +889,6 @@ static struct bpf_prog_pack *alloc_new_pack(void)
 	bitmap_zero(pack->bitmap, bpf_prog_pack_size / BPF_PROG_CHUNK_SIZE);
 	list_add_tail(&pack->list, &pack_list);
 
-	set_vm_flush_reset_perms(pack->ptr);
 	set_memory_ro((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE);
 	set_memory_x((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE);
 	return pack;

@@ -970,7 +969,9 @@ static void bpf_prog_pack_free(struct bpf_binary_header *hdr)
 	if (bitmap_find_next_zero_area(pack->bitmap, bpf_prog_chunk_count(), 0,
 				       bpf_prog_chunk_count(), 0) == 0) {
 		list_del(&pack->list);
-		module_memfree(pack->ptr);
+		set_memory_nx((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE);
+		set_memory_rw((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE);
+		vfree(pack->ptr);
 		kfree(pack);
 	}
 out:

Signature · 1 line

2.30.2

Luis Chamberlain

maintainer

2022-04-12 15:38 UTC #4 lore raw

In reply to Song Liu

Quoted 23 lines · Song Liu

Changes v1 => v2:
1. Add vmalloc_huge(). (Christoph Hellwig)
2. Add module_alloc_huge(). (Christoph Hellwig)
3. Add Fixes tag and Link tag. (Thorsten Leemhuis)
Enabling HAVE_ARCH_HUGE_VMALLOC on x86_64 and use it for bpf_prog_pack has caused some issues [1], as many users of vmalloc are not yet ready to handle huge pages. To enable a more smooth transition to use huge page backed vmalloc memory, this set replaces VM_NO_HUGE_VMAP flag with an new opt-in flag, VM_ALLOW_HUGE_VMAP. More discussions about this topic can be found at [2].
Patch 1 removes VM_NO_HUGE_VMAP and adds VM_ALLOW_HUGE_VMAP. Patch 2 uses VM_ALLOW_HUGE_VMAP in bpf_prog_pack.
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
Song Liu (3):
vmalloc: replace VM_NO_HUGE_VMAP with VM_ALLOW_HUGE_VMAP
module: introduce module_alloc_huge
bpf: use vmalloc with VM_ALLOW_HUGE_VMAP for bpf_prog_pack

Quoted 8 lines

 arch/Kconfig                 |  6 ++----
 arch/powerpc/kernel/module.c |  2 +-
 arch/s390/kvm/pv.c           |  2 +-
 arch/x86/kernel/module.c     | 21 +++++++++++++++++++
 include/linux/moduleloader.h |  5 +++++
 include/linux/vmalloc.h      |  4 ++--
 kernel/bpf/core.c            |  9 +++++----
 kernel/module.c              |  8 ++++++++

Please use modules-next [0] as that has queued up changes which change kernel/module.c quite a bit.

[0] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git/log/?h=modules-next

Luis

Song Liu

author reviewer

2022-04-13 00:35 UTC #5 lore raw

Hi Luis,

Quoted 34 lines · Luis Chamberlain

Quoted 23 lines · Song Liu
Changes v1 => v2:
1. Add vmalloc_huge(). (Christoph Hellwig)
2. Add module_alloc_huge(). (Christoph Hellwig)
3. Add Fixes tag and Link tag. (Thorsten Leemhuis)
Enabling HAVE_ARCH_HUGE_VMALLOC on x86_64 and use it for bpf_prog_pack has caused some issues [1], as many users of vmalloc are not yet ready to handle huge pages. To enable a more smooth transition to use huge page backed vmalloc memory, this set replaces VM_NO_HUGE_VMAP flag with an new opt-in flag, VM_ALLOW_HUGE_VMAP. More discussions about this topic can be found at [2].
Patch 1 removes VM_NO_HUGE_VMAP and adds VM_ALLOW_HUGE_VMAP. Patch 2 uses VM_ALLOW_HUGE_VMAP in bpf_prog_pack.
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
Song Liu (3):
vmalloc: replace VM_NO_HUGE_VMAP with VM_ALLOW_HUGE_VMAP
module: introduce module_alloc_huge
bpf: use vmalloc with VM_ALLOW_HUGE_VMAP for bpf_prog_pack
Quoted 8 lines
 arch/Kconfig                 |  6 ++----
 arch/powerpc/kernel/module.c |  2 +-
 arch/s390/kvm/pv.c           |  2 +-
 arch/x86/kernel/module.c     | 21 +++++++++++++++++++
 include/linux/moduleloader.h |  5 +++++
 include/linux/vmalloc.h      |  4 ++--
 kernel/bpf/core.c            |  9 +++++----
 kernel/module.c              |  8 ++++++++

Quoted 4 lines

Please use modules-next [0] as that has queued up changes which change kernel/module.c quite a bit.
[0] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git/log/?h=modules-next

We are hoping to ship this set to fix some issues with 5.18. So I guess it shouldn't go through modules-next branch? Would this work for you? We are adding a new API module_alloc_huge(), so it shouldn't break existing features.

Thanks,
Song