Skip to contentSkip to conversation

Type aware module allocator

PATCH In review Song Liu 4 participants 22 messages May 26, 2023 – Jun 1, 2023

Subsystems (11)
Memory management x86 BPF MM core +7 more subsystems
lore
Share & tools
lore

Song Liu

author reviewer
#0 lore raw

cover Type aware module allocator

This set implements the second part of module type aware allocator (module_alloc_type), which was discussed in [1]. This part contains the interface of the new allocator, as well as changes in x86 code to use the new allocator (modules, BPF, ftrace, kprobe).

The set does not contain a binpack allocator to enable sharing huge pages among different allocations. But this set defines the interface used by the binpack allocator. [2] has some discussion on different options of the binpack allocator.

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Song Liu (3):
module: Introduce module_alloc_type
ftrace: Add swap_func to ftrace_process_locs()
x86/module: Use module_alloc_type

 arch/x86/kernel/alternative.c  |  37 ++++--
 arch/x86/kernel/ftrace.c       |  44 ++++---
 arch/x86/kernel/kprobes/core.c |   8 +-
 arch/x86/kernel/module.c       | 114 +++++++++++------
 arch/x86/kernel/unwind_orc.c   |  13 +-
 arch/x86/net/bpf_jit_comp.c    |  22 +++-
 include/linux/ftrace.h         |   2 +
 include/linux/module.h         |   6 +
 include/linux/moduleloader.h   |  75 ++++++++++++
 init/main.c                    |   1 +
 kernel/bpf/bpf_struct_ops.c    |  10 +-
 kernel/bpf/core.c              |  26 ++--
 kernel/bpf/trampoline.c        |   6 +-
 kernel/kprobes.c               |   6 +-
 kernel/module/internal.h       |   3 +
 kernel/module/main.c           | 217 +++++++++++++++++++++++++++++++--
 kernel/module/strict_rwx.c     |   4 +
 kernel/trace/ftrace.c          |  13 +-
 18 files changed, 493 insertions(+), 114 deletions(-)

--
2.34.1

author reviewer
#1 lore raw

PATCH 1/3 module: Introduce module_alloc_type

Introduce memory type aware module_alloc_type, which provides a unified allocator for all different archs. This work was discussed in [1].

Each arch can configure the allocator to do the following:

1. Specify module_vaddr and module_end
2. Random module start address for KASLR
3. kasan_alloc_module_shadow()
4. kasan_reset_tag()
5. Preferred and secondary module address ranges

enum mod_alloc_params_flags are used to control the behavior of module_alloc_type. Specifically: MOD_ALLOC_FALLBACK let module_alloc_type fallback to existing module_alloc. MOD_ALLOC_SET_MEMORY let module_alloc_type to protect the memory before returning to the user.

module_allocator_init() call is added to start_kernel() to initialize module_alloc_type.

[1] https://lore.kernel.org/linux-mm/[email protected]/

---
 include/linux/module.h       |   6 +
 include/linux/moduleloader.h |  75 ++++++++++++
 init/main.c                  |   1 +
 kernel/bpf/bpf_struct_ops.c  |  10 +-
 kernel/bpf/core.c            |  20 ++--
 kernel/bpf/trampoline.c      |   6 +-
 kernel/kprobes.c             |   6 +-
 kernel/module/internal.h     |   3 +
 kernel/module/main.c         | 217 +++++++++++++++++++++++++++++++++--
 kernel/module/strict_rwx.c   |   4 +
 10 files changed, 319 insertions(+), 29 deletions(-)
include/linux/module.h +6 -0
index 9e56763dff81..948b8132a742 100644
@@ -752,6 +752,8 @@ static inline bool is_livepatch_module(struct module *mod)
 
 void set_module_sig_enforced(void);
 
+void __init module_allocator_init(void);
+
 #else /* !CONFIG_MODULES... */
 
 static inline struct module *__module_address(unsigned long addr)
@@ -855,6 +857,10 @@ void *dereference_module_function_descriptor(struct module *mod, void *ptr)
 	return ptr;
 }
 
+static inline void __init module_allocator_init(void)
+{
+}
+
 #endif /* CONFIG_MODULES */
 
 #ifdef CONFIG_SYSFS
include/linux/moduleloader.h +75 -0

81 lines not on this page · raw message

init/main.c +1 -0
index af50044deed5..e05228cabde8 100644
@@ -936,6 +936,7 @@ asmlinkage __visible void __init __no_sanitize_address __noreturn start_kernel(v
 	sort_main_extable();
 	trap_init();
 	mm_core_init();
+	module_allocator_init();
 	poking_init();
 	ftrace_init();
 
kernel/bpf/bpf_struct_ops.c +5 -5
index d3f0a4825fa6..e4ec4be866cc 100644
@@ -12,6 +12,7 @@
 #include <linux/mutex.h>
 #include <linux/btf_ids.h>
 #include <linux/rcupdate_wait.h>
+#include <linux/moduleloader.h>
 
 enum bpf_struct_ops_state {
 	BPF_STRUCT_OPS_STATE_INIT,
@@ -512,7 +513,8 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
 		err = st_ops->validate(kdata);
 		if (err)
 			goto reset_unlock;
-		set_memory_rox((long)st_map->image, 1);
+		module_memory_protect(st_map->image, PAGE_SIZE, MOD_TEXT);
+
 		/* Let bpf_link handle registration & unregistration.
 		 *
 		 * Pair with smp_load_acquire() during lookup_elem().
@@ -521,7 +523,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
 		goto unlock;
 	}
 
-	set_memory_rox((long)st_map->image, 1);
+	module_memory_protect(st_map->image, PAGE_SIZE, MOD_TEXT);
 	err = st_ops->reg(kdata);
 	if (likely(!err)) {
 		/* This refcnt increment on the map here after
@@ -544,8 +546,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
 	 * there was a race in registering the struct_ops (under the same name) to
 	 * a sub-system through different struct_ops's maps.
 	 */
-	set_memory_nx((long)st_map->image, 1);
-	set_memory_rw((long)st_map->image, 1);
+	module_memory_unprotect(st_map->image, PAGE_SIZE, MOD_TEXT);
 
 reset_unlock:
 	bpf_struct_ops_map_put_progs(st_map);
@@ -907,4 +908,3 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
 	kfree(link);
 	return err;
 }
-
kernel/bpf/core.c +10 -10
index 7421487422d4..4c989a8fe8b8 100644
@@ -860,7 +860,7 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins
 		       GFP_KERNEL);
 	if (!pack)
 		return NULL;
-	pack->ptr = module_alloc(BPF_PROG_PACK_SIZE);
+	pack->ptr = module_alloc_type(BPF_PROG_PACK_SIZE, MOD_TEXT);
 	if (!pack->ptr) {
 		kfree(pack);
 		return NULL;
@@ -869,8 +869,7 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins
 	bitmap_zero(pack->bitmap, BPF_PROG_PACK_SIZE / BPF_PROG_CHUNK_SIZE);
 	list_add_tail(&pack->list, &pack_list);
 
-	set_vm_flush_reset_perms(pack->ptr);
-	set_memory_rox((unsigned long)pack->ptr, BPF_PROG_PACK_SIZE / PAGE_SIZE);
+	module_memory_protect(pack->ptr, BPF_PROG_PACK_SIZE, MOD_TEXT);
 	return pack;
 }
 
@@ -884,11 +883,10 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns)
 	mutex_lock(&pack_mutex);
 	if (size > BPF_PROG_PACK_SIZE) {
 		size = round_up(size, PAGE_SIZE);
-		ptr = module_alloc(size);
+		ptr = module_alloc_type(size, MOD_TEXT);
 		if (ptr) {
 			bpf_fill_ill_insns(ptr, size);
-			set_vm_flush_reset_perms(ptr);
-			set_memory_rox((unsigned long)ptr, size / PAGE_SIZE);
+			module_memory_protect(ptr, size, MOD_TEXT);
 		}
 		goto out;
 	}
@@ -922,7 +920,8 @@ void bpf_prog_pack_free(struct bpf_binary_header *hdr)
 
 	mutex_lock(&pack_mutex);
 	if (hdr->size > BPF_PROG_PACK_SIZE) {
-		module_memfree(hdr);
+		module_memfree_type(hdr, MOD_TEXT);
+
 		goto out;
 	}
 
@@ -946,7 +945,8 @@ void bpf_prog_pack_free(struct bpf_binary_header *hdr)
 	if (bitmap_find_next_zero_area(pack->bitmap, BPF_PROG_CHUNK_COUNT, 0,
 				       BPF_PROG_CHUNK_COUNT, 0) == 0) {
 		list_del(&pack->list);
-		module_memfree(pack->ptr);
+		module_memfree_type(pack->ptr, MOD_TEXT);
+
 		kfree(pack);
 	}
 out:
@@ -997,12 +997,12 @@ void bpf_jit_uncharge_modmem(u32 size)
 
 void *__weak bpf_jit_alloc_exec(unsigned long size)
 {
-	return module_alloc(size);
+	return module_alloc_type(size, MOD_TEXT);
 }
 
 void __weak bpf_jit_free_exec(void *addr)
 {
-	module_memfree(addr);
+	module_memfree_type(addr, MOD_TEXT);
 }
 
 struct bpf_binary_header *
kernel/bpf/trampoline.c +3 -3
index ac021bc43a66..fd2d46c9a295 100644
@@ -13,6 +13,7 @@
 #include <linux/bpf_verifier.h>
 #include <linux/bpf_lsm.h>
 #include <linux/delay.h>
+#include <linux/moduleloader.h>
 
 /* dummy _ops. The verifier will operate on target program's ops. */
 const struct bpf_verifier_ops bpf_extension_verifier_ops = {
@@ -440,7 +441,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 	if (err < 0)
 		goto out;
 
-	set_memory_rox((long)im->image, 1);
+	module_memory_protect(im->image, PAGE_SIZE, MOD_TEXT);
 
 	WARN_ON(tr->cur_image && tr->selector == 0);
 	WARN_ON(!tr->cur_image && tr->selector);
@@ -462,8 +463,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 		tr->fops->trampoline = 0;
 
 		/* reset im->image memory attr for arch_prepare_bpf_trampoline */
-		set_memory_nx((long)im->image, 1);
-		set_memory_rw((long)im->image, 1);
+		module_memory_unprotect(im->image, PAGE_SIZE, MOD_TEXT);
 		goto again;
 	}
 #endif
kernel/kprobes.c +3 -3
index 00e177de91cc..daf47da3c96e 100644
@@ -113,17 +113,17 @@ enum kprobe_slot_state {
 void __weak *alloc_insn_page(void)
 {
 	/*
-	 * Use module_alloc() so this page is within +/- 2GB of where the
+	 * Use module_alloc_type() so this page is within +/- 2GB of where the
 	 * kernel image and loaded module images reside. This is required
 	 * for most of the architectures.
 	 * (e.g. x86-64 needs this to handle the %rip-relative fixups.)
 	 */
-	return module_alloc(PAGE_SIZE);
+	return module_alloc_type(PAGE_SIZE, MOD_TEXT);
 }
 
 static void free_insn_page(void *page)
 {
-	module_memfree(page);
+	module_memfree_type(page, MOD_TEXT);
 }
 
 struct kprobe_insn_cache kprobe_insn_slots = {
kernel/module/internal.h +3 -0
index dc7b0160c480..b2e136326c4c 100644
@@ -12,6 +12,7 @@
 #include <linux/mutex.h>
 #include <linux/rculist.h>
 #include <linux/rcupdate.h>
+#include <linux/moduleloader.h>
 #include <linux/mm.h>
 
 #ifndef ARCH_SHF_SMALL
@@ -392,3 +393,5 @@ static inline int same_magic(const char *amagic, const char *bmagic, bool has_cr
 	return strcmp(amagic, bmagic) == 0;
 }
 #endif /* CONFIG_MODVERSIONS */
+
+extern struct mod_allocators module_allocators;
kernel/module/main.c +209 -8

259 lines not on this page · raw message

kernel/module/strict_rwx.c +4 -0
index a2b656b4e3d2..65ff1b09dc84 100644
@@ -16,6 +16,10 @@ static void module_set_memory(const struct module *mod, enum mod_mem_type type,
 {
 	const struct module_memory *mod_mem = &mod->mem[type];
 
+	/* The allocator already called set_memory_*, skip here. */
+	if (module_allocators.types[type]->params.flags & MOD_ALLOC_SET_MEMORY)
+		return;
+
 	set_vm_flush_reset_perms(mod_mem->base);
 	set_memory((unsigned long)mod_mem->base, mod_mem->size >> PAGE_SHIFT);
 }
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH 1/3] module: Introduce module_alloc_type

Hi--

Quoted 7 lines · Song Liu

+/**
+ * struct vmalloc_params - Parameters to call __vmalloc_node_range()

+ * @start:          Address space range start
+ * @end:            Address space range end
+ * @gfp_mask:       The gfp_t mask used for this range
+ * @pgprot:         The page protection for this range
+ * @vm_flags        The vm_flag used for this range

* @vm_flags:

Quoted 8 lines

+ */

include/linux/moduleloader.h +7 -0
+struct vmalloc_params {
+	unsigned long	start;
+	unsigned long	end;
+	gfp_t		gfp_mask;
+	pgprot_t	pgprot;
+	unsigned long	vm_flags;
+};
Signature · 1 line
~Randy
#3 lore raw

Re: [PATCH 1/3] module: Introduce module_alloc_type

In reply to Song Liu
Quoted 19 lines · Song Liu

Introduce memory type aware module_alloc_type, which provides a unified allocator for all different archs. This work was discussed in [1].

Each arch can configure the allocator to do the following:

1. Specify module_vaddr and module_end
2. Random module start address for KASLR
3. kasan_alloc_module_shadow()
4. kasan_reset_tag()
5. Preferred and secondary module address ranges

enum mod_alloc_params_flags are used to control the behavior of module_alloc_type. Specifically: MOD_ALLOC_FALLBACK let module_alloc_type fallback to existing module_alloc. MOD_ALLOC_SET_MEMORY let module_alloc_type to protect the memory before returning to the user.

module_allocator_init() call is added to start_kernel() to initialize module_alloc_type.

...

Quoted 45 lines
(quoted diff) +45 -0
+/**
+ * struct vmalloc_params - Parameters to call __vmalloc_node_range()
+ * @start:          Address space range start
+ * @end:            Address space range end
+ * @gfp_mask:       The gfp_t mask used for this range
+ * @pgprot:         The page protection for this range
+ * @vm_flags        The vm_flag used for this range
+ */
+struct vmalloc_params {
+	unsigned long	start;
+	unsigned long	end;
+	gfp_t		gfp_mask;
+	pgprot_t	pgprot;
+	unsigned long	vm_flags;
+};
+
+/**
+ * struct mod_alloc_params - Parameters for module allocation type
+ * @flags:          Properties in mod_alloc_params_flags
+ * @granularity:    The allocation granularity (PAGE/PMD) in bytes
+ * @alignment:      The allocation alignment requirement
+ * @vmp:            Parameters used to call vmalloc
+ * @fill:           Function to fill allocated space. If NULL, use memcpy()
+ * @invalidate:     Function to invalidate memory space.
+ *
+ * If @granularity > @alignment the allocation can reuse free space in
+ * previously allocated pages. If they are the same, then fresh pages
+ * have to be allocated.
+ */
+struct mod_alloc_params {
+	unsigned int		flags;
+	unsigned int		granularity;
+	unsigned int		alignment;
+	struct vmalloc_params	vmp[MOD_MAX_ADDR_SPACES];
+	void *			(*fill)(void *dst, const void *src, size_t len);
+	void *			(*invalidate)(void *ptr, size_t len);
+};
+
+struct mod_type_allocator {
+	struct mod_alloc_params	params;
+};
+
+struct mod_allocators {
+	struct mod_type_allocator *types[MOD_MEM_NUM_TYPES];
+};

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+void *module_alloc_type(size_t size, enum mod_mem_type type);
+void module_memfree_type(void *ptr, enum mod_mem_type type);
+void module_memory_fill_type(void *dst, void *src, size_t len, enum mod_mem_type type);
+void module_memory_invalidate_type(void *ptr, size_t len, enum mod_mem_type type);
+void module_memory_protect(void *ptr, size_t len, enum mod_mem_type type);
+void module_memory_unprotect(void *ptr, size_t len, enum mod_mem_type type);
+void module_memory_force_protect(void *ptr, size_t len, enum mod_mem_type type);
+void module_memory_force_unprotect(void *ptr, size_t len, enum mod_mem_type type);
+void module_alloc_type_init(struct mod_allocators *allocators);

This is a pretty big and complicated interface, and I haven't been able to find any reasoning or justification for why it's needed.

Why is this going in module.h? Don't do that, this is supposed to be a general purpose allocator for executable memory so start a new file.

What is vmalloc_params doing here? Why is it needed? Can we just get rid of it?

module_memory_protect() and module_memory_unprotect() looks like a completely broken interface for supporting sub-page allocations - different threads with different allocations on the same page will race.

The text_poke() abstraction works; the exposed interface should look like that.

Please kill MOD_ALLOC_SET_MEMORY, and _only_ return memory that is read-only. You should be able to kill mod_alloc_params entirely.

Our other memory allocators don't expose kasan - why does this one? Again, that looks wrong.

I would like to see something much simpler (that doesn't encode awkward assumptions from module and bpf!): please look at the code I sketched out below and tell me why this interface won't work - or if it can, go with _that_.

commit 80ceffd6b1108afc7593bdf060954c73eaf0ffc4

Author: Kent Overstreet <[email protected]>
Date:   Wed May 17 01:22:06 2023 -0400

mm: jit/text allocator

This provides a new, very simple slab allocator for jit/text, i.e. bpf, ftrace trampolines, or bcachefs unpack functions.

With this API we can avoid ever mapping pages both writeable and executable (not implemented in this patch: need to tweak module_alloc()), and it also supports sub-page sized allocations.

include/linux/jitalloc.h new file +9 -0
new file mode 100644
index 0000000000..f1549d60e8
@@ -0,0 +1,9 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_JITALLOC_H
+#define _LINUX_JITALLOC_H
+
+void jit_update(void *buf, void *new_buf, size_t len);
+void jit_free(void *buf);
+void *jit_alloc(void *buf, size_t len);
+
+#endif /* _LINUX_JITALLOC_H */
mm/Kconfig +3 -0
index 4751031f3f..ff26a4f0c9 100644
@@ -1202,6 +1202,9 @@ config LRU_GEN_STATS
 	  This option has a per-memcg and per-node memory overhead.
 # }
 
+config JITALLOC
+	bool
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +1 -0
index c03e1e5859..25e82db9e8 100644
@@ -138,3 +138,4 @@ obj-$(CONFIG_IO_MAPPING) += io-mapping.o
 obj-$(CONFIG_HAVE_BOOTMEM_INFO_NODE) += bootmem_info.o
 obj-$(CONFIG_GENERIC_IOREMAP) += ioremap.o
 obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
+obj-$(CONFIG_JITALLOC) += jitalloc.o
mm/jitalloc.c new file +187 -0

187 lines not on this page · raw message

author reviewer
#4 lore raw

Re: [PATCH 1/3] module: Introduce module_alloc_type

Quoted 83 lines · Kent Overstreet
Quoted 19 lines · Song Liu

Introduce memory type aware module_alloc_type, which provides a unified allocator for all different archs. This work was discussed in [1].

Each arch can configure the allocator to do the following:

1. Specify module_vaddr and module_end
2. Random module start address for KASLR
3. kasan_alloc_module_shadow()
4. kasan_reset_tag()
5. Preferred and secondary module address ranges

enum mod_alloc_params_flags are used to control the behavior of module_alloc_type. Specifically: MOD_ALLOC_FALLBACK let module_alloc_type fallback to existing module_alloc. MOD_ALLOC_SET_MEMORY let module_alloc_type to protect the memory before returning to the user.

module_allocator_init() call is added to start_kernel() to initialize module_alloc_type.

...

Quoted 45 lines
(quoted diff) +45 -0
+/**
+ * struct vmalloc_params - Parameters to call __vmalloc_node_range()
+ * @start:          Address space range start
+ * @end:            Address space range end
+ * @gfp_mask:       The gfp_t mask used for this range
+ * @pgprot:         The page protection for this range
+ * @vm_flags        The vm_flag used for this range
+ */
+struct vmalloc_params {
+     unsigned long   start;
+     unsigned long   end;
+     gfp_t           gfp_mask;
+     pgprot_t        pgprot;
+     unsigned long   vm_flags;
+};
+
+/**
+ * struct mod_alloc_params - Parameters for module allocation type
+ * @flags:          Properties in mod_alloc_params_flags
+ * @granularity:    The allocation granularity (PAGE/PMD) in bytes
+ * @alignment:      The allocation alignment requirement
+ * @vmp:            Parameters used to call vmalloc
+ * @fill:           Function to fill allocated space. If NULL, use memcpy()
+ * @invalidate:     Function to invalidate memory space.
+ *
+ * If @granularity > @alignment the allocation can reuse free space in
+ * previously allocated pages. If they are the same, then fresh pages
+ * have to be allocated.
+ */
+struct mod_alloc_params {
+     unsigned int            flags;
+     unsigned int            granularity;
+     unsigned int            alignment;
+     struct vmalloc_params   vmp[MOD_MAX_ADDR_SPACES];
+     void *                  (*fill)(void *dst, const void *src, size_t len);
+     void *                  (*invalidate)(void *ptr, size_t len);
+};
+
+struct mod_type_allocator {
+     struct mod_alloc_params params;
+};
+
+struct mod_allocators {
+     struct mod_type_allocator *types[MOD_MEM_NUM_TYPES];
+};

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+void *module_alloc_type(size_t size, enum mod_mem_type type);
+void module_memfree_type(void *ptr, enum mod_mem_type type);
+void module_memory_fill_type(void *dst, void *src, size_t len, enum mod_mem_type type);
+void module_memory_invalidate_type(void *ptr, size_t len, enum mod_mem_type type);
+void module_memory_protect(void *ptr, size_t len, enum mod_mem_type type);
+void module_memory_unprotect(void *ptr, size_t len, enum mod_mem_type type);
+void module_memory_force_protect(void *ptr, size_t len, enum mod_mem_type type);
+void module_memory_force_unprotect(void *ptr, size_t len, enum mod_mem_type type);
+void module_alloc_type_init(struct mod_allocators *allocators);

This is a pretty big and complicated interface, and I haven't been able to find any reasoning or justification for why it's needed.

Quoted 2 lines

Why is this going in module.h? Don't do that, this is supposed to be a general purpose allocator for executable memory so start a new file.

The goal of this work is to build a memory allocator for modules, text, rw data, and ro data. So it is not the same as execmem_alloc or jitalloc.

Quoted 3 lines

What is vmalloc_params doing here? Why is it needed? Can we just get rid of it?

We need it to have an interface for all archs. They are all using different variations of vmalloc for module_alloc.

Quoted 4 lines

module_memory_protect() and module_memory_unprotect() looks like a completely broken interface for supporting sub-page allocations - different threads with different allocations on the same page will race.

These two APIs allow the core code work with all archs. They won't break sub-page allocations. (not all archs will have sub-page allocations)

OTOH, the "force" version of the two APIs should be removed later. In this set, we only need them for arch_prepare_bpf_trampoline(). I plan to remove this limitation for x86 soon.

Quoted 7 lines

The text_poke() abstraction works; the exposed interface should look like that.

Please kill MOD_ALLOC_SET_MEMORY, and _only_ return memory that is read-only. You should be able to kill mod_alloc_params entirely.

Quoted 7 lines

Our other memory allocators don't expose kasan - why does this one? Again, that looks wrong.

I would like to see something much simpler (that doesn't encode awkward assumptions from module and bpf!): please look at the code I sketched out below and tell me why this interface won't work - or if it can, go with _that_.

I think we need to align the goal here. PS: we did align the goal about 6 months ago when I proposed the execmem_alloc() set.

My current goal is to build an allocator for all the use cases of modules, text, data, rodata, etc. Then the same allocator can be used by other dynamic kernel text: bpf, ftrace, kprobe, bcachefs, etc.

OTOH, jit_alloc (as-is) solves a subset of the problem (only the text). We can probably use it (with some updates) instead of the vmap_area based allocator. But I guess we need to align the direction first.

Thanks,
Song

Quoted 247 lines

commit 80ceffd6b1108afc7593bdf060954c73eaf0ffc4

Author: Kent Overstreet <[email protected]>
Date:   Wed May 17 01:22:06 2023 -0400

mm: jit/text allocator

This provides a new, very simple slab allocator for jit/text, i.e. bpf, ftrace trampolines, or bcachefs unpack functions.

With this API we can avoid ever mapping pages both writeable and executable (not implemented in this patch: need to tweak module_alloc()), and it also supports sub-page sized allocations.

include/linux/jitalloc.h new file +9 -0
new file mode 100644
index 0000000000..f1549d60e8
@@ -0,0 +1,9 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_JITALLOC_H
+#define _LINUX_JITALLOC_H
+
+void jit_update(void *buf, void *new_buf, size_t len);
+void jit_free(void *buf);
+void *jit_alloc(void *buf, size_t len);
+
+#endif /* _LINUX_JITALLOC_H */
mm/Kconfig +3 -0
index 4751031f3f..ff26a4f0c9 100644
@@ -1202,6 +1202,9 @@ config LRU_GEN_STATS
          This option has a per-memcg and per-node memory overhead.
 # }
 
+config JITALLOC
+       bool
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +1 -0
index c03e1e5859..25e82db9e8 100644
@@ -138,3 +138,4 @@ obj-$(CONFIG_IO_MAPPING) += io-mapping.o
 obj-$(CONFIG_HAVE_BOOTMEM_INFO_NODE) += bootmem_info.o
 obj-$(CONFIG_GENERIC_IOREMAP) += ioremap.o
 obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
+obj-$(CONFIG_JITALLOC) += jitalloc.o
mm/jitalloc.c new file +187 -0
new file mode 100644
index 0000000000..7c4d621802
@@ -0,0 +1,187 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/gfp.h>
+#include <linux/highmem.h>
+#include <linux/jitalloc.h>
+#include <linux/mm.h>
+#include <linux/moduleloader.h>
+#include <linux/mutex.h>
+#include <linux/set_memory.h>
+#include <linux/vmalloc.h>
+
+#include <asm/text-patching.h>
+
+static DEFINE_MUTEX(jit_alloc_lock);
+
+struct jit_cache {
+       unsigned                obj_size_bits;
+       unsigned                objs_per_slab;
+       struct list_head        partial;
+};
+
+#define JITALLOC_MIN_SIZE      16
+#define NR_JIT_CACHES          ilog2(PAGE_SIZE / JITALLOC_MIN_SIZE)
+
+static struct jit_cache jit_caches[NR_JIT_CACHES];
+
+struct jit_slab {
+       unsigned long           __page_flags;
+
+       struct jit_cache        *cache;
+       void                    *executably_mapped;;
+       unsigned long           *objs_allocated; /* bitmap of free objects */
+       struct list_head        list;
+};
+
+#define folio_jit_slab(folio)          (_Generic((folio),                      \
+       const struct folio *:           (const struct jit_slab *)(folio),       \
+       struct folio *:                 (struct jit_slab *)(folio)))
+
+#define jit_slab_folio(s)              (_Generic((s),                          \
+       const struct jit_slab *:        (const struct folio *)s,                \
+       struct jit_slab *:              (struct folio *)s))
+
+static struct jit_slab *jit_slab_alloc(struct jit_cache *cache)
+{
+       void *executably_mapped = module_alloc(PAGE_SIZE);
+       struct page *page;
+       struct folio *folio;
+       struct jit_slab *slab;
+       unsigned long *objs_allocated;
+
+       if (!executably_mapped)
+               return NULL;
+
+       objs_allocated = kcalloc(BITS_TO_LONGS(cache->objs_per_slab), sizeof(unsigned long), GFP_KERNEL);
+       if (!objs_allocated ) {
+               vfree(executably_mapped);
+               return NULL;
+       }
+
+       set_vm_flush_reset_perms(executably_mapped);
+       set_memory_rox((unsigned long) executably_mapped, 1);
+
+       page = vmalloc_to_page(executably_mapped);
+       folio = page_folio(page);
+
+       __folio_set_slab(folio);
+       slab                    = folio_jit_slab(folio);
+       slab->cache             = cache;
+       slab->executably_mapped = executably_mapped;
+       slab->objs_allocated = objs_allocated;
+       INIT_LIST_HEAD(&slab->list);
+
+       return slab;
+}
+
+static void *jit_cache_alloc(void *buf, size_t len, struct jit_cache *cache)
+{
+       struct jit_slab *s =
+               list_first_entry_or_null(&cache->partial, struct jit_slab, list) ?:
+               jit_slab_alloc(cache);
+       unsigned obj_idx, nr_allocated;
+
+       if (!s)
+               return NULL;
+
+       obj_idx = find_first_zero_bit(s->objs_allocated, cache->objs_per_slab);
+
+       BUG_ON(obj_idx >= cache->objs_per_slab);
+       __set_bit(obj_idx, s->objs_allocated);
+
+       nr_allocated = bitmap_weight(s->objs_allocated, s->cache->objs_per_slab);
+
+       if (nr_allocated == s->cache->objs_per_slab) {
+               list_del_init(&s->list);
+       } else if (nr_allocated == 1) {
+               list_del(&s->list);
+               list_add(&s->list, &s->cache->partial);
+       }
+
+       return s->executably_mapped + (obj_idx << cache->obj_size_bits);
+}
+
+void jit_update(void *buf, void *new_buf, size_t len)
+{
+       text_poke_copy(buf, new_buf, len);
+}
+EXPORT_SYMBOL_GPL(jit_update);
+
+void jit_free(void *buf)
+{
+       struct page *page;
+       struct folio *folio;
+       struct jit_slab *s;
+       unsigned obj_idx, nr_allocated;
+       size_t offset;
+
+       if (!buf)
+               return;
+
+       page    = vmalloc_to_page(buf);
+       folio   = page_folio(page);
+       offset  = offset_in_folio(folio, buf);
+
+       if (!folio_test_slab(folio)) {
+               vfree(buf);
+               return;
+       }
+
+       s = folio_jit_slab(folio);
+
+       mutex_lock(&jit_alloc_lock);
+       obj_idx = offset >> s->cache->obj_size_bits;
+
+       __clear_bit(obj_idx, s->objs_allocated);
+
+       nr_allocated = bitmap_weight(s->objs_allocated, s->cache->objs_per_slab);
+
+       if (nr_allocated == 0) {
+               list_del(&s->list);
+               kfree(s->objs_allocated);
+               folio_put(folio);
+       } else if (nr_allocated + 1 == s->cache->objs_per_slab) {
+               list_del(&s->list);
+               list_add(&s->list, &s->cache->partial);
+       }
+
+       mutex_unlock(&jit_alloc_lock);
+}
+EXPORT_SYMBOL_GPL(jit_free);
+
+void *jit_alloc(void *buf, size_t len)
+{
+       unsigned jit_cache_idx = ilog2(roundup_pow_of_two(len) / 16);
+       void *p;
+
+       if (jit_cache_idx < NR_JIT_CACHES) {
+               mutex_lock(&jit_alloc_lock);
+               p = jit_cache_alloc(buf, len, &jit_caches[jit_cache_idx]);
+               mutex_unlock(&jit_alloc_lock);
+       } else {
+               p = module_alloc(len);
+               if (p) {
+                       set_vm_flush_reset_perms(p);
+                       set_memory_rox((unsigned long) p, DIV_ROUND_UP(len, PAGE_SIZE));
+               }
+       }
+
+       if (p && buf)
+               jit_update(p, buf, len);
+
+       return p;
+}
+EXPORT_SYMBOL_GPL(jit_alloc);
+
+static int __init jit_alloc_init(void)
+{
+       for (unsigned i = 0; i < ARRAY_SIZE(jit_caches); i++) {
+               jit_caches[i].obj_size_bits     = ilog2(JITALLOC_MIN_SIZE) + i;
+               jit_caches[i].objs_per_slab     = PAGE_SIZE >> jit_caches[i].obj_size_bits;
+
+               INIT_LIST_HEAD(&jit_caches[i].partial);
+       }
+
+       return 0;
+}
+core_initcall(jit_alloc_init);
#5 lore raw

Re: [PATCH 1/3] module: Introduce module_alloc_type

Quoted 92 lines · Song Liu
Quoted 83 lines · Kent Overstreet
Quoted 19 lines · Song Liu

Introduce memory type aware module_alloc_type, which provides a unified allocator for all different archs. This work was discussed in [1].

Each arch can configure the allocator to do the following:

1. Specify module_vaddr and module_end
2. Random module start address for KASLR
3. kasan_alloc_module_shadow()
4. kasan_reset_tag()
5. Preferred and secondary module address ranges

enum mod_alloc_params_flags are used to control the behavior of module_alloc_type. Specifically: MOD_ALLOC_FALLBACK let module_alloc_type fallback to existing module_alloc. MOD_ALLOC_SET_MEMORY let module_alloc_type to protect the memory before returning to the user.

module_allocator_init() call is added to start_kernel() to initialize module_alloc_type.

...

Quoted 45 lines
(quoted diff) +45 -0
+/**
+ * struct vmalloc_params - Parameters to call __vmalloc_node_range()
+ * @start:          Address space range start
+ * @end:            Address space range end
+ * @gfp_mask:       The gfp_t mask used for this range
+ * @pgprot:         The page protection for this range
+ * @vm_flags        The vm_flag used for this range
+ */
+struct vmalloc_params {
+     unsigned long   start;
+     unsigned long   end;
+     gfp_t           gfp_mask;
+     pgprot_t        pgprot;
+     unsigned long   vm_flags;
+};
+
+/**
+ * struct mod_alloc_params - Parameters for module allocation type
+ * @flags:          Properties in mod_alloc_params_flags
+ * @granularity:    The allocation granularity (PAGE/PMD) in bytes
+ * @alignment:      The allocation alignment requirement
+ * @vmp:            Parameters used to call vmalloc
+ * @fill:           Function to fill allocated space. If NULL, use memcpy()
+ * @invalidate:     Function to invalidate memory space.
+ *
+ * If @granularity > @alignment the allocation can reuse free space in
+ * previously allocated pages. If they are the same, then fresh pages
+ * have to be allocated.
+ */
+struct mod_alloc_params {
+     unsigned int            flags;
+     unsigned int            granularity;
+     unsigned int            alignment;
+     struct vmalloc_params   vmp[MOD_MAX_ADDR_SPACES];
+     void *                  (*fill)(void *dst, const void *src, size_t len);
+     void *                  (*invalidate)(void *ptr, size_t len);
+};
+
+struct mod_type_allocator {
+     struct mod_alloc_params params;
+};
+
+struct mod_allocators {
+     struct mod_type_allocator *types[MOD_MEM_NUM_TYPES];
+};

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+void *module_alloc_type(size_t size, enum mod_mem_type type);
+void module_memfree_type(void *ptr, enum mod_mem_type type);
+void module_memory_fill_type(void *dst, void *src, size_t len, enum mod_mem_type type);
+void module_memory_invalidate_type(void *ptr, size_t len, enum mod_mem_type type);
+void module_memory_protect(void *ptr, size_t len, enum mod_mem_type type);
+void module_memory_unprotect(void *ptr, size_t len, enum mod_mem_type type);
+void module_memory_force_protect(void *ptr, size_t len, enum mod_mem_type type);
+void module_memory_force_unprotect(void *ptr, size_t len, enum mod_mem_type type);
+void module_alloc_type_init(struct mod_allocators *allocators);

This is a pretty big and complicated interface, and I haven't been able to find any reasoning or justification for why it's needed.

Quoted 2 lines

Why is this going in module.h? Don't do that, this is supposed to be a general purpose allocator for executable memory so start a new file.

The goal of this work is to build a memory allocator for modules, text, rw data, and ro data. So it is not the same as execmem_alloc or jitalloc.

Quoted 1 line
Quoted 5 lines
Quoted 2 lines

What is vmalloc_params doing here? Why is it needed? Can we just get rid of it?

We need it to have an interface for all archs. They are all using different variations of vmalloc for module_alloc.

But it should be an internal implementation detail, I don't think we want the external interface tied to vmalloc -

Quoted 2 lines

These two APIs allow the core code work with all archs. They won't break sub-page allocations. (not all archs will have sub-page allocations)

So yes, text_poke() doesn't work on all architectures, and we need a fallback.

But if the fallback needs to go the unprotect/protect route, I think we need to put that in the helper, and not expose it. Part of what people are wanting is to limit or eliminate pages that are RWX, so we definitely shouldn't be creating new interfaces to flipping page
permissions: that should be pushed down to as low a level as possible.

E.g., with my jitalloc_update(), a more complete version would look like

void jitalloc_update(void *dst, void *src, size_t len)

{
	if (text_poke_available) {
		text_poke(...);
	} else {
		unprotect();
		memcpy();
		protect();
	}
}

See? We provide a simpler, safer interface, and this also lets us handle multiple threads racing to update/flip permissions on the same page in a single place (e.g. with sticking a lock somewhere in the jitalloc structures).

Or ideally, we'd have a kmap_local() variant that actually creates a new mapping on all architectures, and we'd just use that to do the update without any arch-dependent code - if we get that, having this helper means we'll only have to change a single place later.

Quoted 15 lines

OTOH, the "force" version of the two APIs should be removed later. In this set, we only need them for arch_prepare_bpf_trampoline(). I plan to remove this limitation for x86 soon.

Quoted 10 lines

The text_poke() abstraction works; the exposed interface should look like that.

Please kill MOD_ALLOC_SET_MEMORY, and _only_ return memory that is read-only. You should be able to kill mod_alloc_params entirely.

Our other memory allocators don't expose kasan - why does this one? Again, that looks wrong.

Quoted 7 lines
Quoted 4 lines

I would like to see something much simpler (that doesn't encode awkward assumptions from module and bpf!): please look at the code I sketched out below and tell me why this interface won't work - or if it can, go with _that_.

I think we need to align the goal here. PS: we did align the goal about 6 months ago when I proposed the execmem_alloc() set.

Yeah, I know what it feels like to get yanked around by different reviewers who see and join in the discussion at different times. Sorry :)

Having a rational/design doc in the header file would help a _lot_.

Quoted 3 lines

My current goal is to build an allocator for all the use cases of modules, text, data, rodata, etc. Then the same allocator can be used by other dynamic kernel text: bpf, ftrace, kprobe, bcachefs, etc.

This must have been part of the previous discussion since you started with execmem_alloc(), but I did scan through that and I'm still not seeing the justification for why this needs to handle non-text allocations.

If I had to hazard a guess it would be because of tglx wanting to solve tlb fragmentation for modules, but to me that doesn't sound like a good enough reason and it looks like we're ending up with a messy "try to be all things to all people" interface as a result :/

Do we _really_ need that?

Mike was just saying at LSF that direct map fragmentation turned out to be a non issue for performance for non-text, so maybe we don't.

Also - module_memory_fill_type(), module_memory_invalidate_type() - I know these are for BPF, but could you explain why we need them in the external interface here? Could they perhaps be small helpers in the bpf code that use something like jitalloc_update()?

Quoted 3 lines

OTOH, jit_alloc (as-is) solves a subset of the problem (only the text). We can probably use it (with some updates) instead of the vmap_area based allocator. But I guess we need to align the direction first.

Let's see if we can get tglx to chime in again, since he was pretty vocal in the last discussion.

It's also good practice to try to summarize all the relevant "whys" from the discussion that went into a feature and put that in at least the commit message - or even better, header file comments.

Also, organization: the module code is a huge mess, let's _please_ do split this out and think about organization a bit more, not add to the mess :)

Cheers,
Kent

author reviewer
#6 lore raw

Re: [PATCH 1/3] module: Introduce module_alloc_type

Quoted 1 line · Kent Overstreet

[...]

Quoted 19 lines

But it should be an internal implementation detail, I don't think we want the external interface tied to vmalloc -

Quoted 2 lines

These two APIs allow the core code work with all archs. They won't break sub-page allocations. (not all archs will have sub-page allocations)

So yes, text_poke() doesn't work on all architectures, and we need a fallback.

But if the fallback needs to go the unprotect/protect route, I think we need to put that in the helper, and not expose it. Part of what people are wanting is to limit or eliminate pages that are RWX, so we definitely shouldn't be creating new interfaces to flipping page
permissions: that should be pushed down to as low a level as possible.

E.g., with my jitalloc_update(), a more complete version would look like

void jitalloc_update(void *dst, void *src, size_t len)

Quoted 9 lines
{
        if (text_poke_available) {
                text_poke(...);
        } else {
                unprotect();
                memcpy();
                protect();
        }
}

I think this doesn't work for sub page allocation?

Quoted 5 lines

See? We provide a simpler, safer interface, and this also lets us handle multiple threads racing to update/flip permissions on the same page in a single place (e.g. with sticking a lock somewhere in the jitalloc structures).

[...]

Quoted 11 lines

This must have been part of the previous discussion since you started with execmem_alloc(), but I did scan through that and I'm still not seeing the justification for why this needs to handle non-text allocations.

If I had to hazard a guess it would be because of tglx wanting to solve tlb fragmentation for modules, but to me that doesn't sound like a good enough reason and it looks like we're ending up with a messy "try to be all things to all people" interface as a result :/

Quoted 4 lines

Do we _really_ need that?

Mike was just saying at LSF that direct map fragmentation turned out to be a non issue for performance for non-text, so maybe we don't.

At the end of all this, we will have modules running from huge pages, data and text. It will give significant performance benefit when some key driver cannot be compiled into the kernel.

Quoted 5 lines

Also - module_memory_fill_type(), module_memory_invalidate_type() - I know these are for BPF, but could you explain why we need them in the external interface here? Could they perhaps be small helpers in the bpf code that use something like jitalloc_update()?

These are used by all users, not just BPF. 1/3 uses them in kernel/module/main.c. I didn't use them in 3/3 as it is arch code, but I can use them instead of text_poke_* (and that is probably better code style). As I am writing the email, I think I should use it in ftrace.c (2/3) to avoid the __weak function.

Quoted 8 lines
Quoted 4 lines

OTOH, jit_alloc (as-is) solves a subset of the problem (only the text). We can probably use it (with some updates) instead of the vmap_area based allocator. But I guess we need to align the direction first.

Let's see if we can get tglx to chime in again, since he was pretty vocal in the last discussion.

Quoted 7 lines

It's also good practice to try to summarize all the relevant "whys" from the discussion that went into a feature and put that in at least the commit message - or even better, header file comments.

Also, organization: the module code is a huge mess, let's _please_ do split this out and think about organization a bit more, not add to the mess :)

I don't really think module code is very messy at the moment. If necessary, we can put module_alloc_type related code in a separate file.

Thanks,
Song

#7 lore raw

Re: [PATCH 1/3] module: Introduce module_alloc_type

Quoted 34 lines · Song Liu
Quoted 1 line · Kent Overstreet

[...]

Quoted 19 lines

But it should be an internal implementation detail, I don't think we want the external interface tied to vmalloc -

Quoted 2 lines

These two APIs allow the core code work with all archs. They won't break sub-page allocations. (not all archs will have sub-page allocations)

So yes, text_poke() doesn't work on all architectures, and we need a fallback.

But if the fallback needs to go the unprotect/protect route, I think we need to put that in the helper, and not expose it. Part of what people are wanting is to limit or eliminate pages that are RWX, so we definitely shouldn't be creating new interfaces to flipping page
permissions: that should be pushed down to as low a level as possible.

E.g., with my jitalloc_update(), a more complete version would look like

void jitalloc_update(void *dst, void *src, size_t len)

Quoted 9 lines
{
        if (text_poke_available) {
                text_poke(...);
        } else {
                unprotect();
                memcpy();
                protect();
        }
}
Quoted 1 line

I think this doesn't work for sub page allocation?

Perhaps I elided too much - it does if you add a single lock. You can't do that if it's not a common helper.

Quoted 3 lines

At the end of all this, we will have modules running from huge pages, data and text. It will give significant performance benefit when some key driver cannot be compiled into the kernel.

Yeah, I've seen the numbers for the perf impact of running as a module due to the extra TLB overhead - but Mike's recent data was showing that this doesn't matter nearly as much as data as it does for text.

Quoted 10 lines
Quoted 4 lines

Also - module_memory_fill_type(), module_memory_invalidate_type() - I know these are for BPF, but could you explain why we need them in the external interface here? Could they perhaps be small helpers in the bpf code that use something like jitalloc_update()?

These are used by all users, not just BPF. 1/3 uses them in kernel/module/main.c. I didn't use them in 3/3 as it is arch code, but I can use them instead of text_poke_* (and that is probably better code style). As I am writing the email, I think I should use it in ftrace.c (2/3) to avoid the __weak function.

Ok. Could we make it clearer why they're needed and what they're for?

I know bpf fills in invalid instructions initially; I didn't read through enough code to find the "why", and let's document that to save other people the same effort.

And do they really need to be callbacks in mod_alloc_params...?

Do we need the other things in mod_alloc_params/vmalloc_params?
- your granularity field says it's for specifying PAGE/PMD size: we definitely do not want that. We've had way too much code along the lines of "implement hugepages for x", we need to stop doing that. It's an internal mm/ detail.

- vmalloc_params: we need gfp_t and pgprot_t, but those should be normal arguments. start/end/vm_flags? They seem like historical module baggage, can we get rid of them?

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

OTOH, jit_alloc (as-is) solves a subset of the problem (only the text). We can probably use it (with some updates) instead of the vmap_area based allocator. But I guess we need to align the direction first.

Let's see if we can get tglx to chime in again, since he was pretty vocal in the last discussion.

It's also good practice to try to summarize all the relevant "whys" from the discussion that went into a feature and put that in at least the commit message - or even better, header file comments.

Quoted 7 lines
Quoted 3 lines

Also, organization: the module code is a huge mess, let's _please_ do split this out and think about organization a bit more, not add to the mess :)

I don't really think module code is very messy at the moment. If necessary, we can put module_alloc_type related code in a separate file.

Hey, it's been organized since I last looked at it :) But I tihink this belongs in mm/, not module code.

author reviewer
#8 lore raw

Re: [PATCH 1/3] module: Introduce module_alloc_type

Quoted 1 line · Kent Overstreet

[...]

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

void jitalloc_update(void *dst, void *src, size_t len)

{
        if (text_poke_available) {
                text_poke(...);
        } else {
                unprotect();
                memcpy();
                protect();
        }
}
Quoted 4 lines
Quoted 1 line

I think this doesn't work for sub page allocation?

Perhaps I elided too much - it does if you add a single lock. You can't do that if it's not a common helper.

Actually, sub page allocation is not the problem. The problem is with unprotect() and protect(), as they require global TLB flush.

Quoted 20 lines
Quoted 4 lines

At the end of all this, we will have modules running from huge pages, data and text. It will give significant performance benefit when some key driver cannot be compiled into the kernel.

Yeah, I've seen the numbers for the perf impact of running as a module due to the extra TLB overhead - but Mike's recent data was showing that this doesn't matter nearly as much as data as it does for text.

Quoted 10 lines
Quoted 4 lines

Also - module_memory_fill_type(), module_memory_invalidate_type() - I know these are for BPF, but could you explain why we need them in the external interface here? Could they perhaps be small helpers in the bpf code that use something like jitalloc_update()?

These are used by all users, not just BPF. 1/3 uses them in kernel/module/main.c. I didn't use them in 3/3 as it is arch code, but I can use them instead of text_poke_* (and that is probably better code style). As I am writing the email, I think I should use it in ftrace.c (2/3) to avoid the __weak function.

Quoted 7 lines

Ok. Could we make it clearer why they're needed and what they're for?

I know bpf fills in invalid instructions initially; I didn't read through enough code to find the "why", and let's document that to save other people the same effort.

And do they really need to be callbacks in mod_alloc_params...?

If we want to call them from common code, they will either be callback or some __weak functions.

Quoted 6 lines

Do we need the other things in mod_alloc_params/vmalloc_params?
- your granularity field says it's for specifying PAGE/PMD size: we definitely do not want that. We've had way too much code along the lines of "implement hugepages for x", we need to stop doing that. It's an internal mm/ detail.

We don't need "granularity" yet. We will need them with the binpack allocator.

Quoted 3 lines

- vmalloc_params: we need gfp_t and pgprot_t, but those should be normal arguments. start/end/vm_flags? They seem like historical module baggage, can we get rid of them?

All these fields are needed by some of the module_alloc()s for different archs. I am not an expert of all the archs, so I cannot say whether some of them are not needed.
[...]

Quoted 6 lines
Quoted 3 lines

I don't really think module code is very messy at the moment. If necessary, we can put module_alloc_type related code in a separate file.

Hey, it's been organized since I last looked at it :) But I tihink this belongs in mm/, not module code.

I don't have a strong preference for this (at the moment). If we agree it belongs to mm/, we sure can move it.

Thanks,
Song

author reviewer
#9 lore raw

PATCH 2/3 ftrace: Add swap_func to ftrace_process_locs()

In reply to Song Liu

ftrace_process_locs sorts module mcount, which is inside RO memory. Add a ftrace_swap_func so that archs can use RO-memory-poke function to do the sorting.

---
 include/linux/ftrace.h |  2 ++
 kernel/trace/ftrace.c  | 13 ++++++++++++-
 2 files changed, 14 insertions(+), 1 deletion(-)
include/linux/ftrace.h +2 -0
index b23bdd414394..fe443b8ed32c 100644
@@ -1166,4 +1166,6 @@ unsigned long arch_syscall_addr(int nr);
 
 #endif /* CONFIG_FTRACE_SYSCALLS */
 
+void ftrace_swap_func(void *a, void *b, int n);
+
 #endif /* _LINUX_FTRACE_H */
kernel/trace/ftrace.c +12 -1
index 764668467155..f5ddc9d4cfb6 100644
@@ -6430,6 +6430,17 @@ static void test_is_sorted(unsigned long *start, unsigned long count)
 }
 #endif
 
+void __weak ftrace_swap_func(void *a, void *b, int n)
+{
+	unsigned long t;
+
+	WARN_ON_ONCE(n != sizeof(t));
+
+	t = *((unsigned long *)a);
+	*(unsigned long *)a = *(unsigned long *)b;
+	*(unsigned long *)b = t;
+}
+
 static int ftrace_process_locs(struct module *mod,
 			       unsigned long *start,
 			       unsigned long *end)
@@ -6455,7 +6466,7 @@ static int ftrace_process_locs(struct module *mod,
 	 */
 	if (!IS_ENABLED(CONFIG_BUILDTIME_MCOUNT_SORT) || mod) {
 		sort(start, count, sizeof(*start),
-		     ftrace_cmp_ips, NULL);
+		     ftrace_cmp_ips, ftrace_swap_func);
 	} else {
 		test_is_sorted(start, count);
 	}
Signature · 1 line
2.34.1
author reviewer
#10 lore raw

PATCH 3/3 x86/module: Use module_alloc_type

In reply to Song Liu

Enable module_alloc_type to

1. Allocate ROX data for MOD_TEXT and MOD_INIT_TEXT;
2. Allocate RO data for MOD_RODATA and MOD_INIT_RODATA;
3. Allocate RW data for other types.

Also, update users of module_alloc_type (BPF, ftrace, kprobe) to handle these restrictions.

arch_prepare_bpf_trampoline() cannot jit directly into module memory yet, so we have to use module_memory_force_[un]protect() in it.

---
 arch/x86/kernel/alternative.c  |  37 +++++++----
 arch/x86/kernel/ftrace.c       |  44 +++++++------
 arch/x86/kernel/kprobes/core.c |   8 +--
 arch/x86/kernel/module.c       | 114 +++++++++++++++++++++++----------
 arch/x86/kernel/unwind_orc.c   |  13 ++--
 arch/x86/net/bpf_jit_comp.c    |  22 +++++--
 kernel/bpf/core.c              |   6 +-
 7 files changed, 160 insertions(+), 84 deletions(-)
arch/x86/kernel/alternative.c +24 -13
index f615e0cb6d93..bb4e6c3225bf 100644
@@ -122,6 +122,17 @@ extern struct alt_instr __alt_instructions[], __alt_instructions_end[];
 extern s32 __smp_locks[], __smp_locks_end[];
 void text_poke_early(void *addr, const void *opcode, size_t len);
 
+static void __init_or_module do_text_poke(void *addr, const void *opcode, size_t len)
+{
+	if (system_state < SYSTEM_RUNNING) {
+		text_poke_early(addr, opcode, len);
+	} else {
+		mutex_lock(&text_mutex);
+		text_poke(addr, opcode, len);
+		mutex_unlock(&text_mutex);
+	}
+}
+
 /*
  * Are we looking at a near JMP with a 1 or 4-byte displacement.
  */
@@ -331,7 +342,7 @@ void __init_or_module noinline apply_alternatives(struct alt_instr *start,
 
 		DUMP_BYTES(insn_buff, insn_buff_sz, "%px: final_insn: ", instr);
 
-		text_poke_early(instr, insn_buff, insn_buff_sz);
+		do_text_poke(instr, insn_buff, insn_buff_sz);
 
 next:
 		optimize_nops(instr, a->instrlen);
@@ -564,7 +575,7 @@ void __init_or_module noinline apply_retpolines(s32 *start, s32 *end)
 			optimize_nops(bytes, len);
 			DUMP_BYTES(((u8*)addr),  len, "%px: orig: ", addr);
 			DUMP_BYTES(((u8*)bytes), len, "%px: repl: ", addr);
-			text_poke_early(addr, bytes, len);
+			do_text_poke(addr, bytes, len);
 		}
 	}
 }
@@ -638,7 +649,7 @@ void __init_or_module noinline apply_returns(s32 *start, s32 *end)
 		if (len == insn.length) {
 			DUMP_BYTES(((u8*)addr),  len, "%px: orig: ", addr);
 			DUMP_BYTES(((u8*)bytes), len, "%px: repl: ", addr);
-			text_poke_early(addr, bytes, len);
+			do_text_poke(addr, bytes, len);
 		}
 	}
 }
@@ -674,7 +685,7 @@ static void poison_endbr(void *addr, bool warn)
 	 */
 	DUMP_BYTES(((u8*)addr), 4, "%px: orig: ", addr);
 	DUMP_BYTES(((u8*)&poison), 4, "%px: repl: ", addr);
-	text_poke_early(addr, &poison, 4);
+	do_text_poke(addr, &poison, 4);
 }
 
 /*
@@ -869,7 +880,7 @@ static int cfi_disable_callers(s32 *start, s32 *end)
 		if (!hash) /* nocfi callers */
 			continue;
 
-		text_poke_early(addr, jmp, 2);
+		do_text_poke(addr, jmp, 2);
 	}
 
 	return 0;
@@ -892,7 +903,7 @@ static int cfi_enable_callers(s32 *start, s32 *end)
 		if (!hash) /* nocfi callers */
 			continue;
 
-		text_poke_early(addr, mov, 2);
+		do_text_poke(addr, mov, 2);
 	}
 
 	return 0;
@@ -913,7 +924,7 @@ static int cfi_rand_preamble(s32 *start, s32 *end)
 			return -EINVAL;
 
 		hash = cfi_rehash(hash);
-		text_poke_early(addr + 1, &hash, 4);
+		do_text_poke(addr + 1, &hash, 4);
 	}
 
 	return 0;
@@ -932,9 +943,9 @@ static int cfi_rewrite_preamble(s32 *start, s32 *end)
 			 addr, addr, 5, addr))
 			return -EINVAL;
 
-		text_poke_early(addr, fineibt_preamble_start, fineibt_preamble_size);
+		do_text_poke(addr, fineibt_preamble_start, fineibt_preamble_size);
 		WARN_ON(*(u32 *)(addr + fineibt_preamble_hash) != 0x12345678);
-		text_poke_early(addr + fineibt_preamble_hash, &hash, 4);
+		do_text_poke(addr + fineibt_preamble_hash, &hash, 4);
 	}
 
 	return 0;
@@ -953,7 +964,7 @@ static int cfi_rand_callers(s32 *start, s32 *end)
 		hash = decode_caller_hash(addr);
 		if (hash) {
 			hash = -cfi_rehash(hash);
-			text_poke_early(addr + 2, &hash, 4);
+			do_text_poke(addr + 2, &hash, 4);
 		}
 	}
 
@@ -971,9 +982,9 @@ static int cfi_rewrite_callers(s32 *start, s32 *end)
 		addr -= fineibt_caller_size;
 		hash = decode_caller_hash(addr);
 		if (hash) {
-			text_poke_early(addr, fineibt_caller_start, fineibt_caller_size);
+			do_text_poke(addr, fineibt_caller_start, fineibt_caller_size);
 			WARN_ON(*(u32 *)(addr + fineibt_caller_hash) != 0x12345678);
-			text_poke_early(addr + fineibt_caller_hash, &hash, 4);
+			do_text_poke(addr + fineibt_caller_hash, &hash, 4);
 		}
 		/* rely on apply_retpolines() */
 	}
@@ -1243,7 +1254,7 @@ void __init_or_module apply_paravirt(struct paravirt_patch_site *start,
 
 		/* Pad the rest with nops */
 		add_nops(insn_buff + used, p->len - used);
-		text_poke_early(p->instr, insn_buff, p->len);
+		do_text_poke(p->instr, insn_buff, p->len);
 	}
 }
 extern struct paravirt_patch_site __start_parainstructions[],
arch/x86/kernel/ftrace.c +26 -18

118 lines not on this page · raw message

arch/x86/kernel/kprobes/core.c +1 -7
index f7f6042eb7e6..96f56e663cbe 100644
@@ -414,16 +414,10 @@ void *alloc_insn_page(void)
 {
 	void *page;
 
-	page = module_alloc(PAGE_SIZE);
+	page = module_alloc_type(PAGE_SIZE, MOD_TEXT);
 	if (!page)
 		return NULL;
 
-	/*
-	 * TODO: Once additional kernel code protection mechanisms are set, ensure
-	 * that the page was not maliciously altered and it is still zeroed.
-	 */
-	set_memory_rox((unsigned long)page, 1);
-
 	return page;
 }
 
arch/x86/kernel/module.c +80 -34

147 lines not on this page · raw message

arch/x86/kernel/unwind_orc.c +9 -4
index 3ac50b7298d1..264188ec50c9 100644
@@ -7,6 +7,7 @@
 #include <asm/unwind.h>
 #include <asm/orc_types.h>
 #include <asm/orc_lookup.h>
+#include <asm/text-patching.h>
 
 #define orc_warn(fmt, ...) \
 	printk_deferred_once(KERN_WARNING "WARNING: " fmt, ##__VA_ARGS__)
@@ -222,18 +223,22 @@ static void orc_sort_swap(void *_a, void *_b, int size)
 	struct orc_entry orc_tmp;
 	int *a = _a, *b = _b, tmp;
 	int delta = _b - _a;
+	int val;
 
 	/* Swap the .orc_unwind_ip entries: */
 	tmp = *a;
-	*a = *b + delta;
-	*b = tmp - delta;
+	val = *b + delta;
+	text_poke_copy(a, &val, sizeof(val));
+	val = tmp - delta;
+	text_poke_copy(b, &val, sizeof(val));
 
 	/* Swap the corresponding .orc_unwind entries: */
 	orc_a = cur_orc_table + (a - cur_orc_ip_table);
 	orc_b = cur_orc_table + (b - cur_orc_ip_table);
 	orc_tmp = *orc_a;
-	*orc_a = *orc_b;
-	*orc_b = orc_tmp;
+
+	text_poke_copy(orc_a, orc_b, sizeof(*orc_b));
+	text_poke_copy(orc_b, &orc_tmp, sizeof(orc_tmp));
 }
 
 static int orc_sort_cmp(const void *_a, const void *_b)
arch/x86/net/bpf_jit_comp.c +16 -6
index 1056bbf55b17..846228fb12f2 100644
@@ -11,6 +11,7 @@
 #include <linux/bpf.h>
 #include <linux/memory.h>
 #include <linux/sort.h>
+#include <linux/moduleloader.h>
 #include <asm/extable.h>
 #include <asm/ftrace.h>
 #include <asm/set_memory.h>
@@ -226,7 +227,7 @@ static u8 simple_alu_opcodes[] = {
 static void jit_fill_hole(void *area, unsigned int size)
 {
 	/* Fill whole space with INT3 instructions */
-	memset(area, 0xcc, size);
+	text_poke_set(area, 0xcc, size);
 }
 
 int bpf_arch_text_invalidate(void *dst, size_t len)
@@ -2202,6 +2203,8 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
 		orig_call += X86_PATCH_SIZE;
 	}
 
+	module_memory_force_unprotect((void *)((unsigned long)image & PAGE_MASK),
+				      PAGE_SIZE, MOD_TEXT);
 	prog = image;
 
 	EMIT_ENDBR();
@@ -2238,20 +2241,24 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
 		emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im);
 		if (emit_rsb_call(&prog, __bpf_tramp_enter, prog)) {
 			ret = -EINVAL;
-			goto cleanup;
+			goto reprotect_memory;
 		}
 	}
 
 	if (fentry->nr_links)
 		if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off,
-			       flags & BPF_TRAMP_F_RET_FENTRY_RET))
-			return -EINVAL;
+			       flags & BPF_TRAMP_F_RET_FENTRY_RET)) {
+			ret = -EINVAL;
+			goto reprotect_memory;
+		}
 
 	if (fmod_ret->nr_links) {
 		branches = kcalloc(fmod_ret->nr_links, sizeof(u8 *),
 				   GFP_KERNEL);
-		if (!branches)
-			return -ENOMEM;
+		if (!branches) {
+			ret =  -ENOMEM;
+			goto reprotect_memory;
+		}
 
 		if (invoke_bpf_mod_ret(m, &prog, fmod_ret, regs_off,
 				       run_ctx_off, branches)) {
@@ -2336,6 +2343,9 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
 
 cleanup:
 	kfree(branches);
+reprotect_memory:
+	module_memory_force_protect((void *)((unsigned long)image & PAGE_MASK),
+				    PAGE_SIZE, MOD_TEXT);
 	return ret;
 }
 
kernel/bpf/core.c +4 -2
index 4c989a8fe8b8..90f09218d30f 100644
@@ -1092,8 +1092,10 @@ bpf_jit_binary_pack_alloc(unsigned int proglen, u8 **image_ptr,
 		return NULL;
 	}
 
-	/* Fill space with illegal/arch-dep instructions. */
-	bpf_fill_ill_insns(*rw_header, size);
+	/* bpf_fill_ill_insns is used to write to RO memory, so we cannot
+	 * use it on rw_header, use memset(0) instead.
+	 */
+	memset(*rw_header, 0, size);
 	(*rw_header)->size = size;
 
 	hole = min_t(unsigned int, size - (proglen + sizeof(*ro_header)),
Signature · 1 line
2.34.1
#11 lore raw
In reply to Song Liu
Quoted 9 lines · Song Liu

This set implements the second part of module type aware allocator (module_alloc_type), which was discussed in [1]. This part contains the interface of the new allocator, as well as changes in x86 code to use the new allocator (modules, BPF, ftrace, kprobe).

The set does not contain a binpack allocator to enable sharing huge pages among different allocations. But this set defines the interface used by the binpack allocator. [2] has some discussion on different options of the binpack allocator.

I'm afraid the more I look at this patchset the more it appears to be complete nonsense.

The exposed interface appears to be both for specifying architecture dependent options (which should be hidden inside the allocator internals!) _and_ a general purpose interface for module/bpf/kprobes - but it's not very clear, and the rational appears to be completely missing.

I think this needs to back to the drawing board and we need something simpler just targeted at executable memory; architecture specific options should definitely _not_ be part of the exposed interface.

The memory protection interface also needs to go, we've got a better interface to model after (text_poke(), although that code needs work too!). And the instruction fill features need a thorough justification if they're to be included.

author reviewer
#12 lore raw
Quoted 7 lines · Kent Overstreet
Quoted 5 lines · Song Liu

This set implements the second part of module type aware allocator (module_alloc_type), which was discussed in [1]. This part contains the interface of the new allocator, as well as changes in x86 code to use the new allocator (modules, BPF, ftrace, kprobe).

Quoted 7 lines
Quoted 4 lines

The set does not contain a binpack allocator to enable sharing huge pages among different allocations. But this set defines the interface used by the binpack allocator. [2] has some discussion on different options of the binpack allocator.

I'm afraid the more I look at this patchset the more it appears to be complete nonsense.

I don't think it is nonsense, as you clearly got most of the points here. :)

Quoted 6 lines

The exposed interface appears to be both for specifying architecture dependent options (which should be hidden inside the allocator internals!) _and_ a general purpose interface for module/bpf/kprobes - but it's not very clear, and the rational appears to be completely missing.

The rationale is to have something to replace module_alloc(). Therefore, it needs to handle architecture specific requirements, and provide interface to all current users of module_alloc(). It may look a little weird at the moment, because the actual allocator logic is very thin. But that's where we will plug in the next step of the work.

Quoted 4 lines

I think this needs to back to the drawing board and we need something simpler just targeted at executable memory; architecture specific options should definitely _not_ be part of the exposed interface.

I don't think we are exposing architecture specific options to users. Some layer need to handle arch specifics. If the new allocator is built on top of module_alloc, module_alloc is handling that. If the new allocator is to replace module_alloc, it needs to handle arch specifics.

Quoted 5 lines

The memory protection interface also needs to go, we've got a better interface to model after (text_poke(), although that code needs work too!). And the instruction fill features need a thorough justification if they're to be included.

I guess the first step to use text_poke() is to make it available on all archs? That doesn't seem easy to me.

Thanks,
Song

reviewer
#13 lore raw
Quoted 11 lines · Song Liu
Quoted 4 lines · Kent Overstreet

I think this needs to back to the drawing board and we need something simpler just targeted at executable memory; architecture specific options should definitely _not_ be part of the exposed interface.

I don't think we are exposing architecture specific options to users. Some layer need to handle arch specifics. If the new allocator is built on top of module_alloc, module_alloc is handling that. If the new allocator is to replace module_alloc, it needs to handle arch specifics.

I'm for creating a new allocator that will replace module_alloc(). This will give us a clean abstraction that modules and all the rest will use and it will make easier to plug binpack or another allocator instead of vmalloc.
Another point is with a new allocator we won't have weird dependencies on CONFIG_MODULE in e.g. bpf and kprobes.

I'll have something ready to post as an RFC in a few days.

Quoted 2 lines

Thanks,
Song

Signature · 2 lines
Sincerely yours,
Mike.
author reviewer
#14 lore raw
Quoted 19 lines · Mike Rapoport
Quoted 11 lines · Song Liu
Quoted 4 lines · Kent Overstreet

I think this needs to back to the drawing board and we need something simpler just targeted at executable memory; architecture specific options should definitely _not_ be part of the exposed interface.

I don't think we are exposing architecture specific options to users. Some layer need to handle arch specifics. If the new allocator is built on top of module_alloc, module_alloc is handling that. If the new allocator is to replace module_alloc, it needs to handle arch specifics.

I'm for creating a new allocator that will replace module_alloc(). This will give us a clean abstraction that modules and all the rest will use and it will make easier to plug binpack or another allocator instead of vmalloc.

Quoted 4 lines

Another point is with a new allocator we won't have weird dependencies on CONFIG_MODULE in e.g. bpf and kprobes.

I'll have something ready to post as an RFC in a few days.

I guess this RFC is similar to unmapped_alloc()? If it replaces vmalloc, we can probably trim this set down a bit (remove mod_alloc_params and vmalloc_params, etc.).

Thanks,
Song

reviewer
#15 lore raw
Quoted 20 lines · Song Liu
Quoted 19 lines · Mike Rapoport
Quoted 11 lines · Song Liu
Quoted 4 lines · Kent Overstreet

I think this needs to back to the drawing board and we need something simpler just targeted at executable memory; architecture specific options should definitely _not_ be part of the exposed interface.

I don't think we are exposing architecture specific options to users. Some layer need to handle arch specifics. If the new allocator is built on top of module_alloc, module_alloc is handling that. If the new allocator is to replace module_alloc, it needs to handle arch specifics.

I'm for creating a new allocator that will replace module_alloc(). This will give us a clean abstraction that modules and all the rest will use and it will make easier to plug binpack or another allocator instead of vmalloc.

Quoted 8 lines
Quoted 4 lines

Another point is with a new allocator we won't have weird dependencies on CONFIG_MODULE in e.g. bpf and kprobes.

I'll have something ready to post as an RFC in a few days.

I guess this RFC is similar to unmapped_alloc()? If it replaces vmalloc, we can probably trim this set down a bit (remove mod_alloc_params and vmalloc_params, etc.).

No, it's not a new allocator. I'm trying to create an API for code allocations that can accommodate all the architectures and it won't be a part of modules code. The modules will use the new API just like every other subsystem that needs to allocate code.

I've got a core part of it here:

https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=jitalloc/v1

and I hope I'll get it ready to post this week.

Quoted 2 lines

Thanks,
Song

Signature · 2 lines
Sincerely yours,
Mike.
author reviewer
#16 lore raw
Quoted 31 lines · Mike Rapoport
Quoted 20 lines · Song Liu
Quoted 19 lines · Mike Rapoport
Quoted 11 lines · Song Liu
Quoted 4 lines · Kent Overstreet

I think this needs to back to the drawing board and we need something simpler just targeted at executable memory; architecture specific options should definitely _not_ be part of the exposed interface.

I don't think we are exposing architecture specific options to users. Some layer need to handle arch specifics. If the new allocator is built on top of module_alloc, module_alloc is handling that. If the new allocator is to replace module_alloc, it needs to handle arch specifics.

I'm for creating a new allocator that will replace module_alloc(). This will give us a clean abstraction that modules and all the rest will use and it will make easier to plug binpack or another allocator instead of vmalloc.

Quoted 8 lines
Quoted 4 lines

Another point is with a new allocator we won't have weird dependencies on CONFIG_MODULE in e.g. bpf and kprobes.

I'll have something ready to post as an RFC in a few days.

I guess this RFC is similar to unmapped_alloc()? If it replaces vmalloc, we can probably trim this set down a bit (remove mod_alloc_params and vmalloc_params, etc.).

Quoted 8 lines

No, it's not a new allocator. I'm trying to create an API for code allocations that can accommodate all the architectures and it won't be a part of modules code. The modules will use the new API just like every other subsystem that needs to allocate code.

I've got a core part of it here:

https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=jitalloc/v1

This branch looks like the same scope as this set (but with different implementation). So it will still use vmalloc, right?

Thanks,
Song

reviewer
#17 lore raw
Quoted 37 lines · Song Liu
Quoted 36 lines · Mike Rapoport
Quoted 20 lines · Song Liu
Quoted 19 lines · Mike Rapoport
Quoted 11 lines · Song Liu
Quoted 4 lines · Kent Overstreet

I think this needs to back to the drawing board and we need something simpler just targeted at executable memory; architecture specific options should definitely _not_ be part of the exposed interface.

I don't think we are exposing architecture specific options to users. Some layer need to handle arch specifics. If the new allocator is built on top of module_alloc, module_alloc is handling that. If the new allocator is to replace module_alloc, it needs to handle arch specifics.

I'm for creating a new allocator that will replace module_alloc(). This will give us a clean abstraction that modules and all the rest will use and it will make easier to plug binpack or another allocator instead of vmalloc.

Quoted 8 lines
Quoted 4 lines

Another point is with a new allocator we won't have weird dependencies on CONFIG_MODULE in e.g. bpf and kprobes.

I'll have something ready to post as an RFC in a few days.

I guess this RFC is similar to unmapped_alloc()? If it replaces vmalloc, we can probably trim this set down a bit (remove mod_alloc_params and vmalloc_params, etc.).

No, it's not a new allocator. I'm trying to create an API for code allocations that can accommodate all the architectures and it won't be a part of modules code. The modules will use the new API just like every other subsystem that needs to allocate code.

Quoted 6 lines
Quoted 3 lines

I've got a core part of it here:

https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=jitalloc/v1

This branch looks like the same scope as this set (but with different implementation). So it will still use vmalloc, right?

Yes, it still uses vmalloc. The idea is to decouple code allocations from modules from one side and make it handle all the variants expected by the architectures based on a set of parameters each architecture provides.

The first few commits essentially shuffle the code around and replace arch::module_alloc() with arch::jit_alloc_params.

The commits on top enable some bits that are not available today, like ROX executable memory and DYNAMIC_FTRACE without modules for x86.

Quoted 2 lines

Thanks,
Song

Signature · 2 lines
Sincerely yours,
Mike.
#18 lore raw
In reply to Song Liu
Quoted 4 lines · Song Liu

I don't think we are exposing architecture specific options to users. Some layer need to handle arch specifics. If the new allocator is built on top of module_alloc, module_alloc is handling that. If the new allocator is to replace module_alloc, it needs to handle arch specifics.

Ok, I went back and read more thoroughly, I got this part wrong. The actual interface is the mod_mem_type enum, not mod_alloc_params or vmalloc_params.

So this was my main complaint, but this actually looks ok now.

It would be better to have those structs in a .c file, not the header file - it looks like those are the public interface the way you have it.

Quoted 7 lines
Quoted 4 lines

The memory protection interface also needs to go, we've got a better interface to model after (text_poke(), although that code needs work too!). And the instruction fill features need a thorough justification if they're to be included.

I guess the first step to use text_poke() is to make it available on all archs? That doesn't seem easy to me.

We just need a helper that either calls text_poke() or does the page permission dance in a single place.

If we do the same thing for other mod_mem_types, we could potentially allow them to be shared on hugepages too.

author reviewer
#19 lore raw
Quoted 7 lines · Kent Overstreet
Quoted 4 lines · Song Liu

I don't think we are exposing architecture specific options to users. Some layer need to handle arch specifics. If the new allocator is built on top of module_alloc, module_alloc is handling that. If the new allocator is to replace module_alloc, it needs to handle arch specifics.

Quoted 8 lines

Ok, I went back and read more thoroughly, I got this part wrong. The actual interface is the mod_mem_type enum, not mod_alloc_params or vmalloc_params.

So this was my main complaint, but this actually looks ok now.

It would be better to have those structs in a .c file, not the header file - it looks like those are the public interface the way you have it.

Thanks for this suggestion. It makes a lot of sense. But I am not quite sure how we can avoid putting it in the header yet. I will take a closer look. OTOH, if we plan to use Mike's new allocator to replace vmalloc, we probably don't need this part.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

The memory protection interface also needs to go, we've got a better interface to model after (text_poke(), although that code needs work too!). And the instruction fill features need a thorough justification if they're to be included.

I guess the first step to use text_poke() is to make it available on all archs? That doesn't seem easy to me.

We just need a helper that either calls text_poke() or does the page permission dance in a single place.

AFAICT, we don't have a global text_poke() API yet. I can take a look into it (if it makes sense).

Quoted 3 lines

If we do the same thing for other mod_mem_types, we could potentially allow them to be shared on hugepages too.

Yeah, that's part of the goal to extend the scope from executable to all types.

Thanks,
Song

#20 lore raw
Quoted 15 lines · Song Liu
Quoted 13 lines · Kent Overstreet
Quoted 4 lines · Song Liu

I don't think we are exposing architecture specific options to users. Some layer need to handle arch specifics. If the new allocator is built on top of module_alloc, module_alloc is handling that. If the new allocator is to replace module_alloc, it needs to handle arch specifics.

Ok, I went back and read more thoroughly, I got this part wrong. The actual interface is the mod_mem_type enum, not mod_alloc_params or vmalloc_params.

So this was my main complaint, but this actually looks ok now.

Quoted 7 lines
Quoted 2 lines

It would be better to have those structs in a .c file, not the header file - it looks like those are the public interface the way you have it.

Thanks for this suggestion. It makes a lot of sense. But I am not quite sure how we can avoid putting it in the header yet. I will take a closer look. OTOH, if we plan to use Mike's new allocator to replace vmalloc, we probably don't need this part.

The architectures previously exported constants that were used by module_alloc(), why not stick with that?

Quoted 2 lines

AFAICT, we don't have a global text_poke() API yet. I can take a look into it (if it makes sense).

Great

Quoted 2 lines

Yeah, that's part of the goal to extend the scope from executable to all types.

Yeah it took me a bit to wrap my head around how this all makes sense - it started out as just a better module_alloc(), then there was lots of talk about hugepages.

I like it more now, looking forward to see how it fits together with Mike's work :)

#21 lore raw
In reply to Song Liu
Quoted 15 lines · Song Liu
Quoted 13 lines · Kent Overstreet
Quoted 4 lines · Song Liu

I don't think we are exposing architecture specific options to users. Some layer need to handle arch specifics. If the new allocator is built on top of module_alloc, module_alloc is handling that. If the new allocator is to replace module_alloc, it needs to handle arch specifics.

Ok, I went back and read more thoroughly, I got this part wrong. The actual interface is the mod_mem_type enum, not mod_alloc_params or vmalloc_params.

So this was my main complaint, but this actually looks ok now.

Quoted 7 lines
Quoted 2 lines

It would be better to have those structs in a .c file, not the header file - it looks like those are the public interface the way you have it.

Thanks for this suggestion. It makes a lot of sense. But I am not quite sure how we can avoid putting it in the header yet. I will take a closer look. OTOH, if we plan to use Mike's new allocator to replace vmalloc, we probably don't need this part.

Well, right now module_alloc() uses arch-exported constants, we could keep doing that if it makes sense.

Or the structs could go in a separate header file that better indicates what they're for. The main point is just that - when we're writing new code and creating a new interface, it's very helpful if we can have the header file basically be the documentation for what the external interface is. Put your big kernel doc "what is this thing about" comment in that file, too :)

Quoted 5 lines
Quoted 2 lines

We just need a helper that either calls text_poke() or does the page permission dance in a single place.

AFAICT, we don't have a global text_poke() API yet. I can take a look into it (if it makes sense).

I don't think anyone's working on this yet, so if you're interested I think it might be helpful.

Quoted 5 lines
Quoted 2 lines

If we do the same thing for other mod_mem_types, we could potentially allow them to be shared on hugepages too.

Yeah, that's part of the goal to extend the scope from executable to all types.

Yeah, I think the "enumerate types of allocations that want similar page permission handling" is a good thing to focus on.