x86/module: use large ROX pages for text allocations
Subsystems (44)
Mike Rapoport
maintainerv7 cover x86/module: use large ROX pages for text allocations
From: "Mike Rapoport (Microsoft)" <[email protected]>
Hi,
This is an updated version of execmem ROX caches.
v6: https://lore.kernel.org/all/[email protected]
* Fixed handling of alternatives for fineibt (kbuild bot)
* Restored usage of text_poke_early for ftrace boot time initialization (Steve)
* Made !module path in module_writable_address inline
v5: https://lore.kernel.org/all/[email protected]
* Droped check for !area in mas_for_each() loop (Kees Bakker)
* Droped externs in include/linux/vmalloc.h (Christoph)
* Fixed handling of alternatives for CFI-enabled configs (Nathan)
* Fixed interaction with kmemleak (Sergey).
It looks like execmem and kmemleak interaction should be improved further, but it's out of scope of this series.
* Added ARCH_HAS_EXECMEM_ROX configuration option to arch/Kconfig. The option serves two purposes:
- make sure architecture that uses ROX caches implements execmem_fill_trapping_insns() callback (Christoph)
- make sure entire physical memory is mapped in the direct map (Dave)
v4: https://lore.kernel.org/all/[email protected]
* Fix copy/paste error in looongarch (Huacai)
v3: https://lore.kernel.org/all/[email protected]
* Drop ftrace_swap_func(). It is not needed because mcount array lives in a data section (Peter)
* Update maple_tree usage (Liam)
* Set ->fill_trapping_insns pointer on init (Ard)
* Instead of using VM_FLUSH_RESET_PERMS for execmem cache, completely remove it from the direct map
v2: https://lore.kernel.org/all/[email protected]
* add comment why ftrace_swap_func() is needed (Steve)
Since RFC: https://lore.kernel.org/all/[email protected]
* update changelog about HUGE_VMAP allocations (Christophe)
* move module_writable_address() from x86 to modules core (Ingo)
* rename execmem_invalidate() to execmem_fill_trapping_insns() (Peter)
* call alternatives_smp_unlock() after module text in-place is up to date (Nadav)
= Original cover letter =
These patches add support for using large ROX pages for allocations of executable memory on x86.
They address Andy's comments [1] about having executable mappings for code that was not completely formed.
The approach taken is to allocate ROX memory along with writable but not executable memory and use the writable copy to perform relocations and alternatives patching. After the module text gets into its final shape, the contents of the writable memory is copied into the actual ROX location using text poking.
The allocations of the ROX memory use vmalloc(VMAP_ALLOW_HUGE_MAP) to allocate PMD aligned memory, fill that memory with invalid instructions and in the end remap it as ROX. Portions of these large pages are handed out to execmem_alloc() callers without any changes to the permissions. When the memory is freed with execmem_free() it is invalidated again so that it won't contain stale instructions.
The module memory allocation, x86 code dealing with relocations and alternatives patching take into account the existence of the two copies, the writable memory and the ROX memory at the actual allocated virtual address.
The patches are available at git:
https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=execmem/x86-rox/v6
[1] https://lore.kernel.org/all/[email protected]
Mike Rapoport (Microsoft) (8):
mm: vmalloc: group declarations depending on CONFIG_MMU together
mm: vmalloc: don't account for number of nodes for HUGE_VMAP allocations
asm-generic: introduce text-patching.h
module: prepare to handle ROX allocations for text
arch: introduce set_direct_map_valid_noflush()
x86/module: prepare module loading for ROX allocations of text
execmem: add support for cache of large ROX pages x86/module: enable ROX caches for module text on 64 bit
arch/Kconfig | 8 +
arch/alpha/include/asm/Kbuild | 1 +
arch/arc/include/asm/Kbuild | 1 +
.../include/asm/{patch.h => text-patching.h} | 0
arch/arm/kernel/ftrace.c | 2 +-
arch/arm/kernel/jump_label.c | 2 +-
arch/arm/kernel/kgdb.c | 2 +-
arch/arm/kernel/patch.c | 2 +-
arch/arm/probes/kprobes/core.c | 2 +-
arch/arm/probes/kprobes/opt-arm.c | 2 +-
arch/arm64/include/asm/set_memory.h | 1 +
.../asm/{patching.h => text-patching.h} | 0
arch/arm64/kernel/ftrace.c | 2 +-
arch/arm64/kernel/jump_label.c | 2 +-
arch/arm64/kernel/kgdb.c | 2 +-
arch/arm64/kernel/patching.c | 2 +-
arch/arm64/kernel/probes/kprobes.c | 2 +-
arch/arm64/kernel/traps.c | 2 +-
arch/arm64/mm/pageattr.c | 10 +
arch/arm64/net/bpf_jit_comp.c | 2 +-
arch/csky/include/asm/Kbuild | 1 +
arch/hexagon/include/asm/Kbuild | 1 +
arch/loongarch/include/asm/Kbuild | 1 +
arch/loongarch/include/asm/set_memory.h | 1 +
arch/loongarch/mm/pageattr.c | 19 +
arch/m68k/include/asm/Kbuild | 1 +
arch/microblaze/include/asm/Kbuild | 1 +
arch/mips/include/asm/Kbuild | 1 +
arch/nios2/include/asm/Kbuild | 1 +
arch/openrisc/include/asm/Kbuild | 1 +
.../include/asm/{patch.h => text-patching.h} | 0
arch/parisc/kernel/ftrace.c | 2 +-
arch/parisc/kernel/jump_label.c | 2 +-
arch/parisc/kernel/kgdb.c | 2 +-
arch/parisc/kernel/kprobes.c | 2 +-
arch/parisc/kernel/patch.c | 2 +-
arch/powerpc/include/asm/kprobes.h | 2 +-
.../asm/{code-patching.h => text-patching.h} | 0
arch/powerpc/kernel/crash_dump.c | 2 +-
arch/powerpc/kernel/epapr_paravirt.c | 2 +-
arch/powerpc/kernel/jump_label.c | 2 +-
arch/powerpc/kernel/kgdb.c | 2 +-
arch/powerpc/kernel/kprobes.c | 2 +-
arch/powerpc/kernel/module_32.c | 2 +-
arch/powerpc/kernel/module_64.c | 2 +-
arch/powerpc/kernel/optprobes.c | 2 +-
arch/powerpc/kernel/process.c | 2 +-
arch/powerpc/kernel/security.c | 2 +-
arch/powerpc/kernel/setup_32.c | 2 +-
arch/powerpc/kernel/setup_64.c | 2 +-
arch/powerpc/kernel/static_call.c | 2 +-
arch/powerpc/kernel/trace/ftrace.c | 2 +-
arch/powerpc/kernel/trace/ftrace_64_pg.c | 2 +-
arch/powerpc/lib/code-patching.c | 2 +-
arch/powerpc/lib/feature-fixups.c | 2 +-
arch/powerpc/lib/test-code-patching.c | 2 +-
arch/powerpc/lib/test_emulate_step.c | 2 +-
arch/powerpc/mm/book3s32/mmu.c | 2 +-
arch/powerpc/mm/book3s64/hash_utils.c | 2 +-
arch/powerpc/mm/book3s64/slb.c | 2 +-
arch/powerpc/mm/kasan/init_32.c | 2 +-
arch/powerpc/mm/mem.c | 2 +-
arch/powerpc/mm/nohash/44x.c | 2 +-
arch/powerpc/mm/nohash/book3e_pgtable.c | 2 +-
arch/powerpc/mm/nohash/tlb.c | 2 +-
arch/powerpc/mm/nohash/tlb_64e.c | 2 +-
arch/powerpc/net/bpf_jit_comp.c | 2 +-
arch/powerpc/perf/8xx-pmu.c | 2 +-
arch/powerpc/perf/core-book3s.c | 2 +-
arch/powerpc/platforms/85xx/smp.c | 2 +-
arch/powerpc/platforms/86xx/mpc86xx_smp.c | 2 +-
arch/powerpc/platforms/cell/smp.c | 2 +-
arch/powerpc/platforms/powermac/smp.c | 2 +-
arch/powerpc/platforms/powernv/idle.c | 2 +-
arch/powerpc/platforms/powernv/smp.c | 2 +-
arch/powerpc/platforms/pseries/smp.c | 2 +-
arch/powerpc/xmon/xmon.c | 2 +-
arch/riscv/errata/andes/errata.c | 2 +-
arch/riscv/errata/sifive/errata.c | 2 +-
arch/riscv/errata/thead/errata.c | 2 +-
arch/riscv/include/asm/set_memory.h | 1 +
.../include/asm/{patch.h => text-patching.h} | 0
arch/riscv/include/asm/uprobes.h | 2 +-
arch/riscv/kernel/alternative.c | 2 +-
arch/riscv/kernel/cpufeature.c | 3 +-
arch/riscv/kernel/ftrace.c | 2 +-
arch/riscv/kernel/jump_label.c | 2 +-
arch/riscv/kernel/patch.c | 2 +-
arch/riscv/kernel/probes/kprobes.c | 2 +-
arch/riscv/mm/pageattr.c | 15 +
arch/riscv/net/bpf_jit_comp64.c | 2 +-
arch/riscv/net/bpf_jit_core.c | 2 +-
arch/s390/include/asm/set_memory.h | 1 +
arch/s390/mm/pageattr.c | 11 +
arch/sh/include/asm/Kbuild | 1 +
arch/sparc/include/asm/Kbuild | 1 +
arch/um/kernel/um_arch.c | 16 +-
arch/x86/Kconfig | 1 +
arch/x86/entry/vdso/vma.c | 3 +-
arch/x86/include/asm/alternative.h | 14 +-
arch/x86/include/asm/set_memory.h | 1 +
arch/x86/include/asm/text-patching.h | 1 +
arch/x86/kernel/alternative.c | 181 ++++++----
arch/x86/kernel/ftrace.c | 30 +-
arch/x86/kernel/module.c | 45 ++-
arch/x86/mm/init.c | 37 +-
arch/x86/mm/pat/set_memory.c | 8 +
arch/xtensa/include/asm/Kbuild | 1 +
include/asm-generic/text-patching.h | 5 +
include/linux/execmem.h | 37 ++
include/linux/module.h | 16 +
include/linux/moduleloader.h | 4 +
include/linux/set_memory.h | 6 +
include/linux/text-patching.h | 15 +
include/linux/vmalloc.h | 60 ++--
kernel/module/debug_kmemleak.c | 3 +-
kernel/module/main.c | 74 +++-
kernel/module/strict_rwx.c | 3 +
mm/execmem.c | 336 +++++++++++++++++-
mm/internal.h | 1 +
mm/vmalloc.c | 14 +-
121 files changed, 885 insertions(+), 247 deletions(-)
rename arch/arm/include/asm/{patch.h => text-patching.h} (100%)
rename arch/arm64/include/asm/{patching.h => text-patching.h} (100%)
rename arch/parisc/include/asm/{patch.h => text-patching.h} (100%)
rename arch/powerpc/include/asm/{code-patching.h => text-patching.h} (100%)
rename arch/riscv/include/asm/{patch.h => text-patching.h} (100%)
create mode 100644 include/asm-generic/text-patching.h
create mode 100644 include/linux/text-patching.hbase-commit: 9852d85ec9d492ebef56dc5f229416c925758edc
Signature · 1 line
2.43.0
Mike Rapoport
maintainerPATCH v7 1/8 mm: vmalloc: group declarations depending on CONFIG_MMU together
From: "Mike Rapoport (Microsoft)" <[email protected]>
There are a couple of declarations that depend on CONFIG_MMU in include/linux/vmalloc.h spread all over the file.
Group them all together to improve code readability.
No functional changes.
--- include/linux/vmalloc.h | 60 +++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 36 deletions(-)
include/linux/vmalloc.h +24 -36
@@ -134,12 +134,6 @@ extern void vm_unmap_ram(const void *mem, unsigned int count);extern void *vm_map_ram (struct page **pages,unsigned int count,int node);extern void vm_unmap_aliases (void ); -#ifdef CONFIG_MMU -extern unsigned long vmalloc_nr_pages (void ); -#else -static inline unsigned long vmalloc_nr_pages (void ) {return 0 ; } -#endif -extern void *vmalloc_noprof (unsigned long size)__alloc_size(1) ;#define vmalloc (...) alloc_hooks(vmalloc_noprof(__VA_ARGS__))
@@ -266,12 +260,29 @@ static inline bool is_vm_area_hugepages(const void *addr)#endif } +/* for /proc/kcore */ +long vread_iter (struct iov_iter *iter,const char *addr,size_t count); + +/* +* Internals. Don't use.. +*/ +__init void vm_area_add_early (struct vm_struct *vm); +__init void vm_area_register_early (struct vm_struct *vm,size_t align); + +int register_vmap_purge_notifier (struct notifier_block *nb); +int unregister_vmap_purge_notifier (struct notifier_block *nb); +#ifdef CONFIG_MMU +#define VMALLOC_TOTAL (VMALLOC_END - VMALLOC_START) + +unsigned long vmalloc_nr_pages (void ); +int vm_area_map_pages (struct vm_struct *area,unsigned long start,unsigned long end,struct page **pages);void vm_area_unmap_pages (struct vm_struct *area,unsigned long start,unsigned long end);void vunmap_range (unsigned long addr,unsigned long end); +static inline void set_vm_flush_reset_perms (void *addr) {struct vm_struct *vm =find_vm_area (addr);
@@ -279,24 +290,14 @@ static inline void set_vm_flush_reset_perms(void *addr)if (vm) vm->flags |= VM_FLUSH_RESET_PERMS; } +#else /* !CONFIG_MMU */ +#define VMALLOC_TOTAL 0UL -#else -static inline void set_vm_flush_reset_perms (void *addr) -{ -} -#endif - -/* for /proc/kcore */ -extern long vread_iter (struct iov_iter *iter,const char *addr,size_t count); - -/* -* Internals. Don't use.. -*/ -extern __init void vm_area_add_early (struct vm_struct *vm); -extern __init void vm_area_register_early (struct vm_struct *vm,size_t align); +static inline unsigned long vmalloc_nr_pages (void ) {return 0 ; } +static inline void set_vm_flush_reset_perms (void *addr) {} +#endif /* CONFIG_MMU */ -#ifdef CONFIG_SMP -# ifdef CONFIG_MMU +#if defined(CONFIG_MMU ) && defined(CONFIG_SMP )struct vm_struct **pcpu_get_vm_areas (const unsigned long *offsets,const size_t *sizes,int nr_vms,size_t align);
@@ -311,22 +312,9 @@ pcpu_get_vm_areas(const unsigned long *offsets, return NULL; } -static inline void -pcpu_free_vm_areas (struct vm_struct **vms,int nr_vms) -{ -} -# endif -#endif - -#ifdef CONFIG_MMU -#define VMALLOC_TOTAL (VMALLOC_END - VMALLOC_START) -#else -#define VMALLOC_TOTAL 0UL +static inline void pcpu_free_vm_areas (struct vm_struct **vms,int nr_vms) {}#endif -int register_vmap_purge_notifier (struct notifier_block *nb); -int unregister_vmap_purge_notifier (struct notifier_block *nb); -#if defined(CONFIG_MMU ) && defined(CONFIG_PRINTK )bool vmalloc_dump_obj (void *object);#else
Signature · 1 line
2.43.0
Mike Rapoport
maintainerPATCH v7 2/8 mm: vmalloc: don't account for number of nodes for HUGE_VMAP allocations
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
vmalloc allocations with VM_ALLOW_HUGE_VMAP that do not explicitly specify node ID will use huge pages only if size_per_node is larger than a huge page.
Still the actual allocated memory is not distributed between nodes and there is no advantage in such approach.
On the contrary, BPF allocates SZ_2M * num_possible_nodes() for each new bpf_prog_pack, while it could do with a single huge page per pack.
Don't account for number of nodes for VM_ALLOW_HUGE_VMAP with NUMA_NO_NODE and use huge pages whenever the requested allocation size is larger than a huge page.
--- mm/vmalloc.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
mm/vmalloc.c +2 -7
@@ -3763,8 +3763,6 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, }if (vmap_allow_huge && (vm_flags &VM_ALLOW_HUGE_VMAP )) { -unsigned long size_per_node; -/* * Try huge pages. Only try for PAGE_KERNEL allocations, * others like modules don't yet expect huge pages in
@@ -3772,13 +3770,10 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,* supporting them. */ - size_per_node = size; -if (node ==NUMA_NO_NODE ) - size_per_node /=num_online_nodes (); -if (arch_vmap_pmd_supported (prot) && size_per_node >=PMD_SIZE ) +if (arch_vmap_pmd_supported (prot) && size >=PMD_SIZE ) shift =PMD_SHIFT ;else - shift =arch_vmap_pte_supported_shift (size_per_node); + shift =arch_vmap_pte_supported_shift (size); align =max (real_align,1UL << shift); size =ALIGN (real_size,1UL << shift);
Signature · 1 line
2.43.0
Mike Rapoport
maintainerPATCH v7 3/8 asm-generic: introduce text-patching.h
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Several architectures support text patching, but they name the header files that declare patching functions differently.
Make all such headers consistently named text-patching.h and add an empty header in asm-generic for architectures that do not support text patching.
---
arch/alpha/include/asm/Kbuild | 1 +
arch/arc/include/asm/Kbuild | 1 +
arch/arm/include/asm/{patch.h => text-patching.h} | 0
arch/arm/kernel/ftrace.c | 2 +-
arch/arm/kernel/jump_label.c | 2 +-
arch/arm/kernel/kgdb.c | 2 +-
arch/arm/kernel/patch.c | 2 +-
arch/arm/probes/kprobes/core.c | 2 +-
arch/arm/probes/kprobes/opt-arm.c | 2 +-
.../include/asm/{patching.h => text-patching.h} | 0
arch/arm64/kernel/ftrace.c | 2 +-
arch/arm64/kernel/jump_label.c | 2 +-
arch/arm64/kernel/kgdb.c | 2 +-
arch/arm64/kernel/patching.c | 2 +-
arch/arm64/kernel/probes/kprobes.c | 2 +-
arch/arm64/kernel/traps.c | 2 +-
arch/arm64/net/bpf_jit_comp.c | 2 +-
arch/csky/include/asm/Kbuild | 1 +
arch/hexagon/include/asm/Kbuild | 1 +
arch/loongarch/include/asm/Kbuild | 1 +
arch/m68k/include/asm/Kbuild | 1 +
arch/microblaze/include/asm/Kbuild | 1 +
arch/mips/include/asm/Kbuild | 1 +
arch/nios2/include/asm/Kbuild | 1 +
arch/openrisc/include/asm/Kbuild | 1 +
.../include/asm/{patch.h => text-patching.h} | 0
arch/parisc/kernel/ftrace.c | 2 +-
arch/parisc/kernel/jump_label.c | 2 +-
arch/parisc/kernel/kgdb.c | 2 +-
arch/parisc/kernel/kprobes.c | 2 +-
arch/parisc/kernel/patch.c | 2 +-
arch/powerpc/include/asm/kprobes.h | 2 +-
.../asm/{code-patching.h => text-patching.h} | 0
arch/powerpc/kernel/crash_dump.c | 2 +-
arch/powerpc/kernel/epapr_paravirt.c | 2 +-
arch/powerpc/kernel/jump_label.c | 2 +-
arch/powerpc/kernel/kgdb.c | 2 +-
arch/powerpc/kernel/kprobes.c | 2 +-
arch/powerpc/kernel/module_32.c | 2 +-
arch/powerpc/kernel/module_64.c | 2 +-
arch/powerpc/kernel/optprobes.c | 2 +-
arch/powerpc/kernel/process.c | 2 +-
arch/powerpc/kernel/security.c | 2 +-
arch/powerpc/kernel/setup_32.c | 2 +-
arch/powerpc/kernel/setup_64.c | 2 +-
arch/powerpc/kernel/static_call.c | 2 +-
arch/powerpc/kernel/trace/ftrace.c | 2 +-
arch/powerpc/kernel/trace/ftrace_64_pg.c | 2 +-
arch/powerpc/lib/code-patching.c | 2 +-
arch/powerpc/lib/feature-fixups.c | 2 +-
arch/powerpc/lib/test-code-patching.c | 2 +-
arch/powerpc/lib/test_emulate_step.c | 2 +-
arch/powerpc/mm/book3s32/mmu.c | 2 +-
arch/powerpc/mm/book3s64/hash_utils.c | 2 +-
arch/powerpc/mm/book3s64/slb.c | 2 +-
arch/powerpc/mm/kasan/init_32.c | 2 +-
arch/powerpc/mm/mem.c | 2 +-
arch/powerpc/mm/nohash/44x.c | 2 +-
arch/powerpc/mm/nohash/book3e_pgtable.c | 2 +-
arch/powerpc/mm/nohash/tlb.c | 2 +-
arch/powerpc/mm/nohash/tlb_64e.c | 2 +-
arch/powerpc/net/bpf_jit_comp.c | 2 +-
arch/powerpc/perf/8xx-pmu.c | 2 +-
arch/powerpc/perf/core-book3s.c | 2 +-
arch/powerpc/platforms/85xx/smp.c | 2 +-
arch/powerpc/platforms/86xx/mpc86xx_smp.c | 2 +-
arch/powerpc/platforms/cell/smp.c | 2 +-
arch/powerpc/platforms/powermac/smp.c | 2 +-
arch/powerpc/platforms/powernv/idle.c | 2 +-
arch/powerpc/platforms/powernv/smp.c | 2 +-
arch/powerpc/platforms/pseries/smp.c | 2 +-
arch/powerpc/xmon/xmon.c | 2 +-
arch/riscv/errata/andes/errata.c | 2 +-
arch/riscv/errata/sifive/errata.c | 2 +-
arch/riscv/errata/thead/errata.c | 2 +-
.../include/asm/{patch.h => text-patching.h} | 0
arch/riscv/include/asm/uprobes.h | 2 +-
arch/riscv/kernel/alternative.c | 2 +-
arch/riscv/kernel/cpufeature.c | 3 ++-
arch/riscv/kernel/ftrace.c | 2 +-
arch/riscv/kernel/jump_label.c | 2 +-
arch/riscv/kernel/patch.c | 2 +-
arch/riscv/kernel/probes/kprobes.c | 2 +-
arch/riscv/net/bpf_jit_comp64.c | 2 +-
arch/riscv/net/bpf_jit_core.c | 2 +-
arch/sh/include/asm/Kbuild | 1 +
arch/sparc/include/asm/Kbuild | 1 +
arch/um/kernel/um_arch.c | 5 +++++
arch/x86/include/asm/text-patching.h | 1 +
arch/xtensa/include/asm/Kbuild | 1 +
include/asm-generic/text-patching.h | 5 +++++
include/linux/text-patching.h | 15 +++++++++++++++
92 files changed, 110 insertions(+), 70 deletions(-)
rename arch/arm/include/asm/{patch.h => text-patching.h} (100%)
rename arch/arm64/include/asm/{patching.h => text-patching.h} (100%)
rename arch/parisc/include/asm/{patch.h => text-patching.h} (100%)
rename arch/powerpc/include/asm/{code-patching.h => text-patching.h} (100%)
rename arch/riscv/include/asm/{patch.h => text-patching.h} (100%)
create mode 100644 include/asm-generic/text-patching.h
create mode 100644 include/linux/text-patching.harch/alpha/include/asm/Kbuild +1 -0
@@ -5,3 +5,4 @@ generic-y += agp.h generic-y += asm-offsets.h generic-y += kvm_para.h generic-y += mcs_spinlock.h +generic-y += text-patching.h
arch/arc/include/asm/Kbuild +1 -0
@@ -6,3 +6,4 @@ generic-y += kvm_para.h generic-y += mcs_spinlock.h generic-y += parport.h generic-y += user.h +generic-y += text-patching.h
arch/arm/include/asm/text-patching.h renamed from arch/arm/include/asm/patch.h +0 -0
arch/arm/kernel/ftrace.c +1 -1
@@ -23,7 +23,7 @@#include <asm/insn.h> #include <asm/set_memory.h> #include <asm/stacktrace.h> -#include <asm/patch.h> +#include <asm/text-patching.h> /* * The compiler emitted profiling hook consists of
arch/arm/kernel/jump_label.c +1 -1
@@ -1,7 +1,7 @@// SPDX-License-Identifier: GPL-2.0 #include <linux/kernel.h> #include <linux/jump_label.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #include <asm/insn.h> static void __arch_jump_label_transform(struct jump_entry *entry,
arch/arm/kernel/kgdb.c +1 -1
@@ -15,7 +15,7 @@#include <linux/kgdb.h> #include <linux/uaccess.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #include <asm/traps.h> struct dbg_reg_def_t dbg_reg_def[DBG_MAX_REG_NUM] =
arch/arm/kernel/patch.c +1 -1
@@ -9,7 +9,7 @@#include <asm/fixmap.h> #include <asm/smp_plat.h> #include <asm/opcodes.h> -#include <asm/patch.h> +#include <asm/text-patching.h> struct patch {void *addr;
arch/arm/probes/kprobes/core.c +1 -1
@@ -25,7 +25,7 @@#include <asm/cacheflush.h> #include <linux/percpu.h> #include <linux/bug.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #include <asm/sections.h> #include "../decode-arm.h"
arch/arm/probes/kprobes/opt-arm.c +1 -1
@@ -14,7 +14,7 @@/* for arm_gen_branch */ #include <asm/insn.h> /* for patch_text */ -#include <asm/patch.h> +#include <asm/text-patching.h> #include "core.h"
arch/arm64/include/asm/text-patching.h renamed from arch/arm64/include/asm/patching.h +0 -0
arch/arm64/kernel/ftrace.c +1 -1
@@ -15,7 +15,7 @@#include <asm/debug-monitors.h> #include <asm/ftrace.h> #include <asm/insn.h> -#include <asm/patching.h> +#include <asm/text-patching.h> #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS struct fregs_offset {
arch/arm64/kernel/jump_label.c +1 -1
@@ -9,7 +9,7 @@#include <linux/jump_label.h> #include <linux/smp.h> #include <asm/insn.h> -#include <asm/patching.h> +#include <asm/text-patching.h> bool arch_jump_label_transform_queue (struct jump_entry *entry,enum jump_label_type type)
arch/arm64/kernel/kgdb.c +1 -1
@@ -17,7 +17,7 @@#include <asm/debug-monitors.h> #include <asm/insn.h> -#include <asm/patching.h> +#include <asm/text-patching.h> #include <asm/traps.h> struct dbg_reg_def_t dbg_reg_def[DBG_MAX_REG_NUM ] = {
arch/arm64/kernel/patching.c +1 -1
@@ -10,7 +10,7 @@#include <asm/fixmap.h> #include <asm/insn.h> #include <asm/kprobes.h> -#include <asm/patching.h> +#include <asm/text-patching.h> #include <asm/sections.h> static DEFINE_RAW_SPINLOCK (patch_lock);
arch/arm64/kernel/probes/kprobes.c +1 -1
@@ -27,7 +27,7 @@#include <asm/debug-monitors.h> #include <asm/insn.h> #include <asm/irq.h> -#include <asm/patching.h> +#include <asm/text-patching.h> #include <asm/ptrace.h> #include <asm/sections.h> #include <asm/system_misc.h>
arch/arm64/kernel/traps.c +1 -1
@@ -41,7 +41,7 @@#include <asm/extable.h> #include <asm/insn.h> #include <asm/kprobes.h> -#include <asm/patching.h> +#include <asm/text-patching.h> #include <asm/traps.h> #include <asm/smp.h> #include <asm/stack_pointer.h>
arch/arm64/net/bpf_jit_comp.c +1 -1
@@ -19,7 +19,7 @@#include <asm/cacheflush.h> #include <asm/debug-monitors.h> #include <asm/insn.h> -#include <asm/patching.h> +#include <asm/text-patching.h> #include <asm/set_memory.h> #include "bpf_jit.h"
arch/csky/include/asm/Kbuild +1 -0
@@ -11,3 +11,4 @@ generic-y += qspinlock.h generic-y += parport.h generic-y += user.h generic-y += vmlinux.lds.h +generic-y += text-patching.h
arch/hexagon/include/asm/Kbuild +1 -0
@@ -5,3 +5,4 @@ generic-y += extable.h generic-y += iomap.h generic-y += kvm_para.h generic-y += mcs_spinlock.h +generic-y += text-patching.h
arch/loongarch/include/asm/Kbuild +1 -0
@@ -11,3 +11,4 @@ generic-y += ioctl.h generic-y += mmzone.h generic-y += statfs.h generic-y += param.h +generic-y += text-patching.h
arch/m68k/include/asm/Kbuild +1 -0
@@ -4,3 +4,4 @@ generic-y += extable.h generic-y += kvm_para.h generic-y += mcs_spinlock.h generic-y += spinlock.h +generic-y += text-patching.h
arch/microblaze/include/asm/Kbuild +1 -0
@@ -8,3 +8,4 @@ generic-y += parport.h generic-y += syscalls.h generic-y += tlb.h generic-y += user.h +generic-y += text-patching.h
arch/mips/include/asm/Kbuild +1 -0
@@ -13,3 +13,4 @@ generic-y += parport.h generic-y += qrwlock.h generic-y += qspinlock.h generic-y += user.h +generic-y += text-patching.h
arch/nios2/include/asm/Kbuild +1 -0
@@ -7,3 +7,4 @@ generic-y += kvm_para.h generic-y += mcs_spinlock.h generic-y += spinlock.h generic-y += user.h +generic-y += text-patching.h
arch/openrisc/include/asm/Kbuild +1 -0
@@ -9,3 +9,4 @@ generic-y += spinlock.h generic-y += qrwlock_types.h generic-y += qrwlock.h generic-y += user.h +generic-y += text-patching.h
arch/parisc/include/asm/text-patching.h renamed from arch/parisc/include/asm/patch.h +0 -0
arch/parisc/kernel/ftrace.c +1 -1
@@ -20,7 +20,7 @@#include <asm/assembly.h> #include <asm/sections.h> #include <asm/ftrace.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #define __hot __section(".text.hot")
arch/parisc/kernel/jump_label.c +1 -1
@@ -8,7 +8,7 @@#include <linux/jump_label.h> #include <linux/bug.h> #include <asm/alternative.h> -#include <asm/patch.h> +#include <asm/text-patching.h> static inline int reassemble_17 (int as17) {
arch/parisc/kernel/kgdb.c +1 -1
@@ -16,7 +16,7 @@#include <asm/ptrace.h> #include <asm/traps.h> #include <asm/processor.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #include <asm/cacheflush.h> const struct kgdb_arch arch_kgdb_ops = {
arch/parisc/kernel/kprobes.c +1 -1
@@ -12,7 +12,7 @@#include <linux/kprobes.h> #include <linux/slab.h> #include <asm/cacheflush.h> -#include <asm/patch.h> +#include <asm/text-patching.h> DEFINE_PER_CPU(struct kprobe *, current_kprobe) = NULL; DEFINE_PER_CPU(struct kprobe_ctlblk, kprobe_ctlblk);
arch/parisc/kernel/patch.c +1 -1
@@ -13,7 +13,7 @@#include <asm/cacheflush.h> #include <asm/fixmap.h> -#include <asm/patch.h> +#include <asm/text-patching.h> struct patch {void *addr;
arch/powerpc/include/asm/kprobes.h +1 -1
@@ -21,7 +21,7 @@#include <linux/percpu.h> #include <linux/module.h> #include <asm/probes.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #ifdef CONFIG_KPROBES #define __ARCH_WANT_KPROBES_INSN_SLOT
arch/powerpc/include/asm/text-patching.h renamed from arch/powerpc/include/asm/code-patching.h +0 -0
arch/powerpc/kernel/crash_dump.c +1 -1
@@ -13,7 +13,7 @@#include <linux/io.h> #include <linux/memblock.h> #include <linux/of.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/kdump.h> #include <asm/firmware.h> #include <linux/uio.h>
arch/powerpc/kernel/epapr_paravirt.c +1 -1
@@ -9,7 +9,7 @@#include <linux/of_fdt.h> #include <asm/epapr_hcalls.h> #include <asm/cacheflush.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/machdep.h> #include <asm/inst.h>
arch/powerpc/kernel/jump_label.c +1 -1
@@ -5,7 +5,7 @@#include <linux/kernel.h> #include <linux/jump_label.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/inst.h> void arch_jump_label_transform(struct jump_entry *entry,
arch/powerpc/kernel/kgdb.c +1 -1
@@ -21,7 +21,7 @@#include <asm/processor.h> #include <asm/machdep.h> #include <asm/debug.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <linux/slab.h> #include <asm/inst.h>
arch/powerpc/kernel/kprobes.c +1 -1
@@ -21,7 +21,7 @@#include <linux/slab.h> #include <linux/set_memory.h> #include <linux/execmem.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/cacheflush.h> #include <asm/sstep.h> #include <asm/sections.h>
arch/powerpc/kernel/module_32.c +1 -1
@@ -18,7 +18,7 @@#include <linux/bug.h> #include <linux/sort.h> #include <asm/setup.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> /* Count how many different relocations (different symbol, different addend) */
arch/powerpc/kernel/module_64.c +1 -1
@@ -17,7 +17,7 @@#include <linux/kernel.h> #include <asm/module.h> #include <asm/firmware.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <linux/sort.h> #include <asm/setup.h> #include <asm/sections.h>
arch/powerpc/kernel/optprobes.c +1 -1
@@ -13,7 +13,7 @@#include <asm/kprobes.h> #include <asm/ptrace.h> #include <asm/cacheflush.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/sstep.h> #include <asm/ppc-opcode.h> #include <asm/inst.h>
arch/powerpc/kernel/process.c +1 -1
@@ -54,7 +54,7 @@#include <asm/firmware.h> #include <asm/hw_irq.h> #endif -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/exec.h> #include <asm/livepatch.h> #include <asm/cpu_has_feature.h>
arch/powerpc/kernel/security.c +1 -1
@@ -14,7 +14,7 @@#include <linux/debugfs.h> #include <asm/asm-prototypes.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/security_features.h> #include <asm/sections.h> #include <asm/setup.h>
arch/powerpc/kernel/setup_32.c +1 -1
@@ -40,7 +40,7 @@#include <asm/time.h> #include <asm/serial.h> #include <asm/udbg.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/cpu_has_feature.h> #include <asm/asm-prototypes.h> #include <asm/kdump.h>
arch/powerpc/kernel/setup_64.c +1 -1
@@ -60,7 +60,7 @@#include <asm/xmon.h> #include <asm/udbg.h> #include <asm/kexec.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/ftrace.h> #include <asm/opal.h> #include <asm/cputhreads.h>
arch/powerpc/kernel/static_call.c +1 -1
@@ -2,7 +2,7 @@#include <linux/memory.h> #include <linux/static_call.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> void arch_static_call_transform (void *site,void *tramp,void *func,bool tail) {
arch/powerpc/kernel/trace/ftrace.c +1 -1
@@ -23,7 +23,7 @@#include <linux/list.h> #include <asm/cacheflush.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/ftrace.h> #include <asm/syscall.h> #include <asm/inst.h>
arch/powerpc/kernel/trace/ftrace_64_pg.c +1 -1
@@ -23,7 +23,7 @@#include <linux/list.h> #include <asm/cacheflush.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/ftrace.h> #include <asm/syscall.h> #include <asm/inst.h>
arch/powerpc/lib/code-patching.c +1 -1
@@ -17,7 +17,7 @@#include <asm/tlb.h> #include <asm/tlbflush.h> #include <asm/page.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/inst.h> static int __patch_mem (void *exec_addr,unsigned long val,void *patch_addr,bool is_dword)
arch/powerpc/lib/feature-fixups.c +1 -1
@@ -16,7 +16,7 @@#include <linux/sched/mm.h> #include <linux/stop_machine.h> #include <asm/cputable.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/interrupt.h> #include <asm/page.h> #include <asm/sections.h>
arch/powerpc/lib/test-code-patching.c +1 -1
@@ -6,7 +6,7 @@#include <linux/vmalloc.h> #include <linux/init.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> static int __init instr_is_branch_to_addr (const u32 *instr,unsigned long addr) {
arch/powerpc/lib/test_emulate_step.c +1 -1
@@ -11,7 +11,7 @@#include <asm/cpu_has_feature.h> #include <asm/sstep.h> #include <asm/ppc-opcode.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/inst.h> #define MAX_SUBTESTS 16
arch/powerpc/mm/book3s32/mmu.c +1 -1
@@ -25,7 +25,7 @@#include <asm/mmu.h> #include <asm/machdep.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/sections.h> #include <mm/mmu_decl.h>
arch/powerpc/mm/book3s64/hash_utils.c +1 -1
@@ -57,7 +57,7 @@#include <asm/sections.h> #include <asm/copro.h> #include <asm/udbg.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/fadump.h> #include <asm/firmware.h> #include <asm/tm.h>
arch/powerpc/mm/book3s64/slb.c +1 -1
@@ -24,7 +24,7 @@#include <linux/pgtable.h> #include <asm/udbg.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include "internal.h"
arch/powerpc/mm/kasan/init_32.c +1 -1
@@ -7,7 +7,7 @@#include <linux/memblock.h> #include <linux/sched/task.h> #include <asm/pgalloc.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <mm/mmu_decl.h> static pgprot_t __init kasan_prot_ro (void )
arch/powerpc/mm/mem.c +1 -1
@@ -26,7 +26,7 @@#include <asm/svm.h> #include <asm/mmzone.h> #include <asm/ftrace.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/setup.h> #include <asm/fixmap.h>
arch/powerpc/mm/nohash/44x.c +1 -1
@@ -24,7 +24,7 @@#include <asm/mmu.h> #include <asm/page.h> #include <asm/cacheflush.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/smp.h> #include <mm/mmu_decl.h>
arch/powerpc/mm/nohash/book3e_pgtable.c +1 -1
@@ -10,7 +10,7 @@#include <asm/pgalloc.h> #include <asm/tlb.h> #include <asm/dma.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <mm/mmu_decl.h>
arch/powerpc/mm/nohash/tlb.c +1 -1
@@ -37,7 +37,7 @@#include <asm/pgalloc.h> #include <asm/tlbflush.h> #include <asm/tlb.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/cputhreads.h> #include <asm/hugetlb.h> #include <asm/paca.h>
arch/powerpc/mm/nohash/tlb_64e.c +1 -1
@@ -24,7 +24,7 @@#include <asm/pgalloc.h> #include <asm/tlbflush.h> #include <asm/tlb.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/cputhreads.h> #include <mm/mmu_decl.h>
arch/powerpc/net/bpf_jit_comp.c +1 -1
@@ -18,7 +18,7 @@#include <linux/bpf.h> #include <asm/kprobes.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include "bpf_jit.h"
arch/powerpc/perf/8xx-pmu.c +1 -1
@@ -14,7 +14,7 @@#include <asm/machdep.h> #include <asm/firmware.h> #include <asm/ptrace.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/inst.h> #define PERF_8xx_ID_CPU_CYCLES 1
arch/powerpc/perf/core-book3s.c +1 -1
@@ -16,7 +16,7 @@#include <asm/machdep.h> #include <asm/firmware.h> #include <asm/ptrace.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/hw_irq.h> #include <asm/interrupt.h>
arch/powerpc/platforms/85xx/smp.c +1 -1
@@ -23,7 +23,7 @@#include <asm/mpic.h> #include <asm/cacheflush.h> #include <asm/dbell.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/cputhreads.h> #include <asm/fsl_pm.h>
arch/powerpc/platforms/86xx/mpc86xx_smp.c +1 -1
@@ -12,7 +12,7 @@#include <linux/delay.h> #include <linux/pgtable.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/page.h> #include <asm/pci-bridge.h> #include <asm/mpic.h>
arch/powerpc/platforms/cell/smp.c +1 -1
@@ -35,7 +35,7 @@#include <asm/firmware.h> #include <asm/rtas.h> #include <asm/cputhreads.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include "interrupt.h" #include <asm/udbg.h>
arch/powerpc/platforms/powermac/smp.c +1 -1
@@ -35,7 +35,7 @@#include <asm/ptrace.h> #include <linux/atomic.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/irq.h> #include <asm/page.h> #include <asm/sections.h>
arch/powerpc/platforms/powernv/idle.c +1 -1
@@ -18,7 +18,7 @@#include <asm/opal.h> #include <asm/cputhreads.h> #include <asm/cpuidle.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/smp.h> #include <asm/runlatch.h> #include <asm/dbell.h>
arch/powerpc/platforms/powernv/smp.c +1 -1
@@ -28,7 +28,7 @@#include <asm/xive.h> #include <asm/opal.h> #include <asm/runlatch.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/dbell.h> #include <asm/kvm_ppc.h> #include <asm/ppc-opcode.h>
arch/powerpc/platforms/pseries/smp.c +1 -1
@@ -39,7 +39,7 @@#include <asm/xive.h> #include <asm/dbell.h> #include <asm/plpar_wrappers.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/svm.h> #include <asm/kvm_guest.h>
arch/powerpc/xmon/xmon.c +1 -1
@@ -50,7 +50,7 @@#include <asm/xive.h> #include <asm/opal.h> #include <asm/firmware.h> -#include <asm/code-patching.h> +#include <asm/text-patching.h> #include <asm/sections.h> #include <asm/inst.h> #include <asm/interrupt.h>
arch/riscv/errata/andes/errata.c +1 -1
@@ -13,7 +13,7 @@#include <asm/alternative.h> #include <asm/cacheflush.h> #include <asm/errata_list.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #include <asm/processor.h> #include <asm/sbi.h> #include <asm/vendorid_list.h>
arch/riscv/errata/sifive/errata.c +1 -1
@@ -8,7 +8,7 @@#include <linux/module.h> #include <linux/string.h> #include <linux/bug.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #include <asm/alternative.h> #include <asm/vendorid_list.h> #include <asm/errata_list.h>
arch/riscv/errata/thead/errata.c +1 -1
@@ -16,7 +16,7 @@#include <asm/errata_list.h> #include <asm/hwprobe.h> #include <asm/io.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #include <asm/vendorid_list.h> #include <asm/vendor_extensions.h>
arch/riscv/include/asm/text-patching.h renamed from arch/riscv/include/asm/patch.h +0 -0
arch/riscv/include/asm/uprobes.h +1 -1
@@ -4,7 +4,7 @@#define _ASM_RISCV_UPROBES_H#include <asm/probes.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #include <asm/bug.h> #define MAX_UINSN_BYTES 8
arch/riscv/kernel/alternative.c +1 -1
@@ -18,7 +18,7 @@#include <asm/sbi.h> #include <asm/csr.h> #include <asm/insn.h> -#include <asm/patch.h> +#include <asm/text-patching.h> struct cpu_manufacturer_info_t {unsigned long vendor_id;
arch/riscv/kernel/cpufeature.c +2 -1
@@ -20,7 +20,8 @@#include <asm/cacheflush.h> #include <asm/cpufeature.h> #include <asm/hwcap.h> -#include <asm/patch.h> +#include <asm/text-patching.h> +#include <asm/hwprobe.h> #include <asm/processor.h> #include <asm/sbi.h> #include <asm/vector.h>
arch/riscv/kernel/ftrace.c +1 -1
@@ -10,7 +10,7 @@#include <linux/memory.h> #include <linux/stop_machine.h> #include <asm/cacheflush.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #ifdef CONFIG_DYNAMIC_FTRACE void ftrace_arch_code_modify_prepare(void) __acquires(&text_mutex)
arch/riscv/kernel/jump_label.c +1 -1
@@ -10,7 +10,7 @@#include <linux/mutex.h> #include <asm/bug.h> #include <asm/cacheflush.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #define RISCV_INSN_NOP 0x00000013U#define RISCV_INSN_JAL 0x0000006fU
arch/riscv/kernel/patch.c +1 -1
@@ -13,7 +13,7 @@#include <asm/cacheflush.h> #include <asm/fixmap.h> #include <asm/ftrace.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #include <asm/sections.h> struct patch_insn {
arch/riscv/kernel/probes/kprobes.c +1 -1
@@ -12,7 +12,7 @@#include <asm/sections.h> #include <asm/cacheflush.h> #include <asm/bug.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #include "decode-insn.h"
arch/riscv/net/bpf_jit_comp64.c +1 -1
@@ -10,7 +10,7 @@#include <linux/filter.h> #include <linux/memory.h> #include <linux/stop_machine.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #include <asm/cfi.h> #include <asm/percpu.h> #include "bpf_jit.h"
arch/riscv/net/bpf_jit_core.c +1 -1
@@ -9,7 +9,7 @@#include <linux/bpf.h> #include <linux/filter.h> #include <linux/memory.h> -#include <asm/patch.h> +#include <asm/text-patching.h> #include <asm/cfi.h> #include "bpf_jit.h"
arch/sh/include/asm/Kbuild +1 -0
@@ -3,3 +3,4 @@ generated-y += syscall_table.h generic-y += kvm_para.h generic-y += mcs_spinlock.h generic-y += parport.h +generic-y += text-patching.h
arch/sparc/include/asm/Kbuild +1 -0
@@ -4,3 +4,4 @@ generated-y += syscall_table_64.h generic-y += agp.h generic-y += kvm_para.h generic-y += mcs_spinlock.h +generic-y += text-patching.h
arch/um/kernel/um_arch.c +5 -0
@@ -468,6 +468,11 @@ void *text_poke(void *addr, const void *opcode, size_t len)return memcpy (addr, opcode, len); } +void *text_poke_copy (void *addr,const void *opcode,size_t len) +{ +return text_poke (addr, opcode, len); +} +void text_poke_sync (void ) { }
arch/x86/include/asm/text-patching.h +1 -0
@@ -35,6 +35,7 @@ extern void *text_poke(void *addr, const void *opcode, size_t len);extern void text_poke_sync (void );extern void *text_poke_kgdb (void *addr,const void *opcode,size_t len);extern void *text_poke_copy (void *addr,const void *opcode,size_t len); +#define text_poke_copy text_poke_copyextern void *text_poke_copy_locked (void *addr,const void *opcode,size_t len,bool core_ok);extern void *text_poke_set (void *addr,int c,size_t len);extern int poke_int3_handler (struct pt_regs *regs);
arch/xtensa/include/asm/Kbuild +1 -0
@@ -8,3 +8,4 @@ generic-y += parport.h generic-y += qrwlock.h generic-y += qspinlock.h generic-y += user.h +generic-y += text-patching.h
include/asm-generic/text-patching.h new file +5 -0
@@ -0,0 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_GENERIC_TEXT_PATCHING_H +#define _ASM_GENERIC_TEXT_PATCHING_H + +#endif /* _ASM_GENERIC_TEXT_PATCHING_H */
include/linux/text-patching.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_TEXT_PATCHING_H +#define _LINUX_TEXT_PATCHING_H + +#include <asm/text-patching.h> + +#ifndef text_poke_copy +static inline void *text_poke_copy (void *dst,const void *src,size_t len) +{ +return memcpy (dst, src, len); +} +#define text_poke_copy text_poke_copy +#endif + +#endif /* _LINUX_TEXT_PATCHING_H */
Signature · 1 line
2.43.0
Mike Rapoport
maintainerPATCH v7 4/8 module: prepare to handle ROX allocations for text
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
In order to support ROX allocations for module text, it is necessary to handle modifications to the code, such as relocations and alternatives patching, without write access to that memory.
One option is to use text patching, but this would make module loading extremely slow and will expose executable code that is not finally formed.
A better way is to have memory allocated with ROX permissions contain invalid instructions and keep a writable, but not executable copy of the module text. The relocations and alternative patches would be done on the writable copy using the addresses of the ROX memory.
Once the module is completely ready, the updated text will be copied to ROX memory using text patching in one go and the writable copy will be freed.
Add support for that to module initialization code and provide necessary interfaces in execmem.
--- include/linux/execmem.h | 23 +++++++++++ include/linux/module.h | 16 ++++++++ include/linux/moduleloader.h | 4 ++ kernel/module/debug_kmemleak.c | 3 +- kernel/module/main.c | 74 ++++++++++++++++++++++++++++++---- kernel/module/strict_rwx.c | 3 ++ mm/execmem.c | 11 +++++ 7 files changed, 126 insertions(+), 8 deletions(-)
include/linux/execmem.h +23 -0
@@ -46,9 +46,11 @@ enum execmem_type {/** * enum execmem_range_flags - options for executable memory allocations * @EXECMEM_KASAN_SHADOW: allocate kasan shadow +* @EXECMEM_ROX_CACHE: allocations should use ROX cache of huge pages */ enum execmem_range_flags {EXECMEM_KASAN_SHADOW = (1 <<0 ), +EXECMEM_ROX_CACHE = (1 <<1 ), };/**
@@ -123,6 +125,27 @@ void *execmem_alloc(enum execmem_type type, size_t size);*/ void execmem_free (void *ptr); +/** +* execmem_update_copy - copy an update to executable memory +* @dst: destination address to update +* @src: source address containing the data +* @size: how many bytes of memory shold be copied +* +* Copy @size bytes from @src to @dst using text poking if the memory at +* @dst is read-only. +* +* Return: a pointer to @dst or NULL on error +*/ +void *execmem_update_copy (void *dst,const void *src,size_t size); + +/** +* execmem_is_rox - check if execmem is read-only +* @type - the execmem type to check +* +* Return: %true if the @type is read-only, %false if it's writable +*/ +bool execmem_is_rox (enum execmem_type type); +#if defined(CONFIG_EXECMEM ) && !defined(CONFIG_ARCH_WANTS_EXECMEM_LATE )void execmem_init (void );#else
include/linux/module.h +16 -0
@@ -367,6 +367,8 @@ enum mod_mem_type {struct module_memory {void *base; +void *rw_copy; +bool is_rox;unsigned int size;#ifdef CONFIG_MODULES_TREE_LOOKUP
@@ -767,6 +769,15 @@ static inline bool is_livepatch_module(struct module *mod)void set_module_sig_enforced (void ); +void *__module_writable_address (struct module *mod,void *loc); + +static inline void *module_writable_address (struct module *mod,void *loc) +{ +if (!IS_ENABLED (CONFIG_ARCH_HAS_EXECMEM_ROX ) || !mod) +return loc; +return __module_writable_address (mod, loc); +} +#else /* !CONFIG_MODULES... */ static inline struct module *__module_address (unsigned long addr)
@@ -874,6 +885,11 @@ static inline bool module_is_coming(struct module *mod) {return false; } + +static inline void *module_writable_address (struct module *mod,void *loc) +{ +return loc; +}#endif /* CONFIG_MODULES */ #ifdef CONFIG_SYSFS
include/linux/moduleloader.h +4 -0
@@ -108,6 +108,10 @@ int module_finalize(const Elf_Ehdr *hdr,const Elf_Shdr *sechdrs, struct module *mod); +int module_post_finalize (const Elf_Ehdr *hdr, +const Elf_Shdr *sechdrs, +struct module *mod); +#ifdef CONFIG_MODULES void flush_module_init_free_work (void );#else
kernel/module/debug_kmemleak.c +2 -1
@@ -14,7 +14,8 @@ void kmemleak_load_module(const struct module *mod, {/* only scan writable, non-executable sections */ for_each_mod_mem_type (type) { -if (type !=MOD_DATA && type !=MOD_INIT_DATA ) +if (type !=MOD_DATA && type !=MOD_INIT_DATA && + !mod->mem[type].is_rox)kmemleak_no_scan (mod->mem[type].base); } }
kernel/module/main.c +67 -7
134 lines not on this page · raw message
kernel/module/strict_rwx.c +3 -0
@@ -34,6 +34,9 @@ int module_enable_text_rox(const struct module *mod) for_class_mod_mem_type(type, text) {int ret; +if (mod->mem[type].is_rox) +continue ; +if (IS_ENABLED (CONFIG_STRICT_MODULE_RWX )) ret =module_set_memory (mod, type, set_memory_rox);else
mm/execmem.c +11 -0
@@ -10,6 +10,7 @@#include <linux/vmalloc.h> #include <linux/execmem.h> #include <linux/moduleloader.h> +#include <linux/text-patching.h> static struct execmem_info *execmem_info__ro_after_init ;static struct execmem_info default_execmem_info__ro_after_init ;
@@ -69,6 +70,16 @@ void execmem_free(void *ptr)vfree (ptr); } +void *execmem_update_copy (void *dst,const void *src,size_t size) +{ +return text_poke_copy (dst, src, size); +} + +bool execmem_is_rox (enum execmem_type type) +{ +return !!(execmem_info->ranges[type].flags &EXECMEM_ROX_CACHE ); +} +static bool execmem_validate (struct execmem_info *info) {struct execmem_range *r = &info->ranges[EXECMEM_DEFAULT ];
Signature · 1 line
2.43.0
Mike Rapoport
maintainerPATCH v7 5/8 arch: introduce set_direct_map_valid_noflush()
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Add an API that will allow updates of the direct/linear map for a set of physically contiguous pages.
It will be used in the following patches.
--- arch/arm64/include/asm/set_memory.h | 1 + arch/arm64/mm/pageattr.c | 10 ++++++++++ arch/loongarch/include/asm/set_memory.h | 1 + arch/loongarch/mm/pageattr.c | 19 +++++++++++++++++++ arch/riscv/include/asm/set_memory.h | 1 + arch/riscv/mm/pageattr.c | 15 +++++++++++++++ arch/s390/include/asm/set_memory.h | 1 + arch/s390/mm/pageattr.c | 11 +++++++++++ arch/x86/include/asm/set_memory.h | 1 + arch/x86/mm/pat/set_memory.c | 8 ++++++++ include/linux/set_memory.h | 6 ++++++ 11 files changed, 74 insertions(+)
arch/arm64/include/asm/set_memory.h +1 -0
@@ -13,6 +13,7 @@ int set_memory_valid(unsigned long addr, int numpages, int enable);int set_direct_map_invalid_noflush (struct page *page);int set_direct_map_default_noflush (struct page *page); +int set_direct_map_valid_noflush (struct page *page,unsigned nr,bool valid);bool kernel_page_present (struct page *page);#endif /* _ASM_ARM64_SET_MEMORY_H */
arch/arm64/mm/pageattr.c +10 -0
@@ -192,6 +192,16 @@ int set_direct_map_default_noflush(struct page *page) PAGE_SIZE, change_page_range, &data); } +int set_direct_map_valid_noflush (struct page *page,unsigned nr,bool valid) +{ +unsigned long addr = (unsigned long )page_address (page); + +if (!can_set_direct_map ()) +return 0 ; + +return set_memory_valid (addr, nr, valid); +} +#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages (struct page *page,int numpages,int enable) {
arch/loongarch/include/asm/set_memory.h +1 -0
@@ -17,5 +17,6 @@ int set_memory_rw(unsigned long addr, int numpages);bool kernel_page_present (struct page *page);int set_direct_map_default_noflush (struct page *page);int set_direct_map_invalid_noflush (struct page *page); +int set_direct_map_valid_noflush (struct page *page,unsigned nr,bool valid);#endif /* _ASM_LOONGARCH_SET_MEMORY_H */
arch/loongarch/mm/pageattr.c +19 -0
@@ -216,3 +216,22 @@ int set_direct_map_invalid_noflush(struct page *page)return __set_memory (addr,1 ,__pgprot (0 ),__pgprot (_PAGE_PRESENT | _PAGE_VALID)); } + +int set_direct_map_valid_noflush (struct page *page,unsigned nr,bool valid) +{ +unsigned long addr = (unsigned long )page_address (page); +pgprot_t set, clear; + +if (addr < vm_map_base) +return 0 ; + +if (valid) { + set =PAGE_KERNEL ; + clear =__pgprot (0 ); + }else { + set =__pgprot (0 ); + clear =__pgprot (_PAGE_PRESENT | _PAGE_VALID); + } + +return __set_memory (addr,1 , set, clear); +}
arch/riscv/include/asm/set_memory.h +1 -0
@@ -42,6 +42,7 @@ static inline int set_kernel_memory(char *startp, char *endp,int set_direct_map_invalid_noflush (struct page *page);int set_direct_map_default_noflush (struct page *page); +int set_direct_map_valid_noflush (struct page *page,unsigned nr,bool valid);bool kernel_page_present (struct page *page);#endif /* __ASSEMBLY__ */
arch/riscv/mm/pageattr.c +15 -0
@@ -386,6 +386,21 @@ int set_direct_map_default_noflush(struct page *page) PAGE_KERNEL, __pgprot(_PAGE_EXEC)); } +int set_direct_map_valid_noflush (struct page *page,unsigned nr,bool valid) +{ +pgprot_t set, clear; + +if (valid) { + set =PAGE_KERNEL ; + clear =__pgprot (_PAGE_EXEC); + }else { + set =__pgprot (0 ); + clear =__pgprot (_PAGE_PRESENT); + } + +return __set_memory ((unsigned long )page_address (page), nr, set, clear); +} +#ifdef CONFIG_DEBUG_PAGEALLOC static int debug_pagealloc_set_page (pte_t *pte,unsigned long addr,void *data) {
arch/s390/include/asm/set_memory.h +1 -0
@@ -62,5 +62,6 @@ __SET_MEMORY_FUNC(set_memory_4k, SET_MEMORY_4K)int set_direct_map_invalid_noflush (struct page *page);int set_direct_map_default_noflush (struct page *page); +int set_direct_map_valid_noflush (struct page *page,unsigned nr,bool valid);#endif
arch/s390/mm/pageattr.c +11 -0
@@ -406,6 +406,17 @@ int set_direct_map_default_noflush(struct page *page)return __set_memory ((unsigned long )page_to_virt (page),1 ,SET_MEMORY_DEF ); } +int set_direct_map_valid_noflush (struct page *page,unsigned nr,bool valid) +{ +unsigned long flags; + +if (valid) + flags =SET_MEMORY_DEF ; +else + flags =SET_MEMORY_INV ; + +return __set_memory ((unsigned long )page_to_virt (page), nr, flags); +}#if defined(CONFIG_DEBUG_PAGEALLOC ) || defined(CONFIG_KFENCE )static void ipte_range (pte_t *pte,unsigned long address,int nr)
arch/x86/include/asm/set_memory.h +1 -0
@@ -89,6 +89,7 @@ int set_pages_rw(struct page *page, int numpages);int set_direct_map_invalid_noflush (struct page *page);int set_direct_map_default_noflush (struct page *page); +int set_direct_map_valid_noflush (struct page *page,unsigned nr,bool valid);bool kernel_page_present (struct page *page);extern int kernel_set_to_readonly;
arch/x86/mm/pat/set_memory.c +8 -0
@@ -2444,6 +2444,14 @@ int set_direct_map_default_noflush(struct page *page)return __set_pages_p (page,1 ); } +int set_direct_map_valid_noflush (struct page *page,unsigned nr,bool valid) +{ +if (valid) +return __set_pages_p (page, nr); + +return __set_pages_np (page, nr); +} +#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages (struct page *page,int numpages,int enable) {
include/linux/set_memory.h +6 -0
@@ -34,6 +34,12 @@ static inline int set_direct_map_default_noflush(struct page *page)return 0 ; } +static inline int set_direct_map_valid_noflush (struct page *page, +unsigned nr,bool valid) +{ +return 0 ; +} +static inline bool kernel_page_present (struct page *page) {return true;
Signature · 1 line
2.43.0
Mike Rapoport
maintainerPATCH v7 6/8 x86/module: prepare module loading for ROX allocations of text
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
When module text memory will be allocated with ROX permissions, the memory at the actual address where the module will live will contain invalid instructions and there will be a writable copy that contains the actual module code.
Update relocations and alternatives patching to deal with it.
--- arch/um/kernel/um_arch.c | 11 +- arch/x86/entry/vdso/vma.c | 3 +- arch/x86/include/asm/alternative.h | 14 +-- arch/x86/kernel/alternative.c | 181 +++++++++++++++++------------ arch/x86/kernel/ftrace.c | 30 ++--- arch/x86/kernel/module.c | 45 ++++--- 6 files changed, 167 insertions(+), 117 deletions(-)
arch/um/kernel/um_arch.c +6 -5
@@ -435,24 +435,25 @@ void __init arch_cpu_finalize_init(void)os_check_bugs (); } -void apply_seal_endbr (s32 *start,s32 *end) +void apply_seal_endbr (s32 *start,s32 *end,struct module *mod) { } -void apply_retpolines (s32 *start,s32 *end) +void apply_retpolines (s32 *start,s32 *end,struct module *mod) { } -void apply_returns (s32 *start,s32 *end) +void apply_returns (s32 *start,s32 *end,struct module *mod) { }void apply_fineibt (s32 *start_retpoline,s32 *end_retpoline, -s32 *start_cfi,s32 *end_cfi) +s32 *start_cfi,s32 *end_cfi,struct module *mod) { } -void apply_alternatives (struct alt_instr *start,struct alt_instr *end) +void apply_alternatives (struct alt_instr *start,struct alt_instr *end, +struct module *mod) { }
arch/x86/entry/vdso/vma.c +2 -1
@@ -54,7 +54,8 @@ int __init init_vdso_image(const struct vdso_image *image)apply_alternatives ((struct alt_instr *)(image->data + image->alt), (struct alt_instr *)(image->data + image->alt + - image->alt_len)); + image->alt_len), +NULL );return 0 ; }
arch/x86/include/asm/alternative.h +7 -7
@@ -96,16 +96,16 @@ extern struct alt_instr __alt_instructions[], __alt_instructions_end[];* instructions were patched in already: */ extern int alternatives_patched; +struct module ;extern void alternative_instructions (void ); -extern void apply_alternatives (struct alt_instr *start,struct alt_instr *end); -extern void apply_retpolines (s32 *start,s32 *end); -extern void apply_returns (s32 *start,s32 *end); -extern void apply_seal_endbr (s32 *start,s32 *end); +extern void apply_alternatives (struct alt_instr *start,struct alt_instr *end, +struct module *mod); +extern void apply_retpolines (s32 *start,s32 *end,struct module *mod); +extern void apply_returns (s32 *start,s32 *end,struct module *mod); +extern void apply_seal_endbr (s32 *start,s32 *end,struct module *mod);extern void apply_fineibt (s32 *start_retpoline,s32 *end_retpoine, -s32 *start_cfi,s32 *end_cfi); - -struct module ; +s32 *start_cfi,s32 *end_cfi,struct module *mod);struct callthunk_sites {s32 *call_start, *call_end;
arch/x86/kernel/alternative.c +105 -76
535 lines not on this page · raw message
arch/x86/kernel/ftrace.c +16 -14
@@ -118,10 +118,13 @@ ftrace_modify_code_direct(unsigned long ip, const char *old_code,return ret;/* replace the text with the new text */ -if (ftrace_poke_late) +if (ftrace_poke_late) {text_poke_queue ((void *)ip, new_code,MCOUNT_INSN_SIZE ,NULL ); -else -text_poke_early ((void *)ip, new_code,MCOUNT_INSN_SIZE ); + }else { +mutex_lock (&text_mutex); +text_poke ((void *)ip, new_code,MCOUNT_INSN_SIZE ); +mutex_unlock (&text_mutex); + }return 0 ; }
@@ -318,7 +321,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)unsigned const char op_ref[] = {0x48 ,0x8b ,0x15 };unsigned const char retq[] = {RET_INSN_OPCODE ,INT3_INSN_OPCODE };union ftrace_op_code_union op_ptr; -int ret; +void *ret;if (ops->flags &FTRACE_OPS_FL_SAVE_REGS ) { start_offset = (unsigned long )ftrace_regs_caller;
@@ -349,15 +352,15 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) npages =DIV_ROUND_UP (*tramp_size,PAGE_SIZE );/* Copy ftrace_caller onto the trampoline memory */ - ret =copy_from_kernel_nofault (trampoline, (void *)start_offset, size); -if (WARN_ON (ret <0 )) + ret =text_poke_copy (trampoline, (void *)start_offset, size); +if (WARN_ON (!ret)) goto fail; ip = trampoline + size;if (cpu_feature_enabled (X86_FEATURE_RETHUNK ))__text_gen_insn (ip,JMP32_INSN_OPCODE , ip, x86_return_thunk,JMP32_INSN_SIZE );else -memcpy (ip, retq,sizeof (retq)); +text_poke_copy (ip, retq,sizeof (retq));/* No need to test direct calls on created trampolines */ if (ops->flags &FTRACE_OPS_FL_SAVE_REGS ) {
@@ -365,8 +368,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) ip = trampoline + (jmp_offset - start_offset);if (WARN_ON (*(char *)ip !=0x75 )) goto fail; - ret =copy_from_kernel_nofault (ip, x86_nops[2 ],2 ); -if (ret <0 ) +if (!text_poke_copy (ip, x86_nops[2 ],2 )) goto fail; }
@@ -379,7 +381,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)*/ ptr = (unsigned long *)(trampoline + size +RET_SIZE ); - *ptr = (unsigned long )ops; +text_poke_copy (ptr, &ops,sizeof (unsigned long )); op_offset -= start_offset;memcpy (&op_ptr, trampoline + op_offset,OP_REF_SIZE );
@@ -395,7 +397,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) op_ptr.offset = offset;/* put in the new offset to the ftrace_ops */ -memcpy (trampoline + op_offset, &op_ptr,OP_REF_SIZE ); +text_poke_copy (trampoline + op_offset, &op_ptr,OP_REF_SIZE );/* put in the call to the function */ mutex_lock (&text_mutex);
@@ -405,9 +407,9 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)* the depth accounting before the call already. */ dest =ftrace_ops_get_func (ops); -memcpy (trampoline + call_offset, -text_gen_insn (CALL_INSN_OPCODE , trampoline + call_offset, dest), -CALL_INSN_SIZE ); +text_poke_copy_locked (trampoline + call_offset, +text_gen_insn (CALL_INSN_OPCODE , trampoline + call_offset, dest), +CALL_INSN_SIZE , false);mutex_unlock (&text_mutex);/* ALLOC_TRAMP flags lets us know we created it */
arch/x86/kernel/module.c +31 -14
102 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v7 6/8] x86/module: prepare module loading for ROX allocations of text
Hi Mike,
Quoted 11 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
When module text memory will be allocated with ROX permissions, the memory at the actual address where the module will live will contain invalid instructions and there will be a writable copy that contains the actual module code.
Update relocations and alternatives patching to deal with it.
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Tested-by kdevops <[email protected]>
Hopefully the last time you have to hear from me, as I am only experiencing issues with only one of my test machines at this point and it is my only machine that supports IBT, so it seems to point to something specific with the IBT part of the FineIBT support. I notice either a boot hang or an almost immediate reboot (triple fault?). I guess this is how I missed reporting this earlier, as my machine was falling back to the default distribution kernel after the restart and I did not notice I was not actually testing a -next kernel.
Checking out the version of this change that is in next-20241104, commit 7ca6ed09db62 ("x86/module: prepare module loading for ROX allocations of text"), it boots with either 'cfi=off' or 'cfi=kcfi' but it exhibits the issues noted above with 'cfi=fineibt'. At the immediate parent, commit b575d981092f ("arch: introduce set_direct_map_valid_noflush()"), all three combinations boot fine.
$ uname -r; tr ' ' '\n' </proc/cmdline | grep cfi=
6.12.0-rc5-debug-00214-g7ca6ed09db62
cfi=kcfi
6.12.0-rc5-debug-00214-g7ca6ed09db62
cfi=off
6.12.0-rc5-debug-00213-gb575d981092f
cfi=fineibt
6.12.0-rc5-debug-00213-gb575d981092f
cfi=kcfi
6.12.0-rc5-debug-00213-gb575d981092f
cfi=off
I do not think this machine has an accessible serial port and I do not think IBT virtualization is supported via either KVM or TCG in QEMU, so I am not sure how to get more information about what is going on here. I wanted to try reverting these changes on top of next-20241104 but there was a non-trivial conflict in mm/execmem.c due to some changes on top, so I just tested in the mm history.
If there is any other information I can provide or patches I can test, I am more than happy to do so.
Cheers,
Nathan
Mike Rapoport
maintainerRe: [PATCH v7 6/8] x86/module: prepare module loading for ROX allocations of text
Hi Nathan,
Quoted 55 lines · Nathan Chancellor
Hi Mike,
Quoted 11 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
When module text memory will be allocated with ROX permissions, the memory at the actual address where the module will live will contain invalid instructions and there will be a writable copy that contains the actual module code.
Update relocations and alternatives patching to deal with it.
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Tested-by kdevops <[email protected]>Hopefully the last time you have to hear from me, as I am only experiencing issues with only one of my test machines at this point and it is my only machine that supports IBT, so it seems to point to something specific with the IBT part of the FineIBT support. I notice either a boot hang or an almost immediate reboot (triple fault?). I guess this is how I missed reporting this earlier, as my machine was falling back to the default distribution kernel after the restart and I did not notice I was not actually testing a -next kernel.
Checking out the version of this change that is in next-20241104, commit
7ca6ed09db62("x86/module: prepare module loading for ROX allocations of text"), it boots with either 'cfi=off' or 'cfi=kcfi' but it exhibits the issues noted above with 'cfi=fineibt'. At the immediate parent, commitb575d981092f("arch: introduceset_direct_map_valid_noflush()"), all three combinations boot fine.$ uname -r; tr ' ' '\n' </proc/cmdline | grep cfi=6.12.0-rc5-debug-00214-g7ca6ed09db62
cfi=kcfi6.12.0-rc5-debug-00214-g7ca6ed09db62
cfi=off6.12.0-rc5-debug-00213-gb575d981092f
cfi=fineibt6.12.0-rc5-debug-00213-gb575d981092f
cfi=kcfi6.12.0-rc5-debug-00213-gb575d981092f
cfi=offI do not think this machine has an accessible serial port and I do not think IBT virtualization is supported via either KVM or TCG in QEMU, so I am not sure how to get more information about what is going on here. I wanted to try reverting these changes on top of next-20241104 but there was a non-trivial conflict in
mm/execmem.cdue to some changes on top, so I just tested in the mm history.
Quoted 2 lines
If there is any other information I can provide or patches I can test, I am more than happy to do so.
Yes, please :)
There's a silly mistake in cfi_rewrite_endbr() in that commit, the patch below should fix it. Can you please test?
arch/x86/kernel/alternative.c +1 -1
@@ -1241,7 +1241,7 @@ static void cfi_rewrite_endbr(s32 *start, s32 *end, struct module *mod)void *addr = (void *)s + *s;void *wr_addr =module_writable_address (mod, addr); -poison_endbr (addr+16 , wr_addr, false); +poison_endbr (addr +16 , wr_addr +16 , false); } }
Quoted 2 lines
Cheers,
Nathan
Signature · 2 lines
Sincerely yours, Mike.
Nathan Chancellor
Tested-byRe: [PATCH v7 6/8] x86/module: prepare module loading for ROX allocations of text
Quoted 2 lines · Mike Rapoport
There's a silly mistake in
cfi_rewrite_endbr()in that commit, the patch below should fix it. Can you please test?
Yup, that was it! All my machines boot with this diff applied on top of next-20241105, so with that fixed, I think we are all good here.
arch/x86/kernel/alternative.c static void cfi_rewrite_endbr(s32 *start, s32 *end, struct module *mod) · line 1246
Quoted 12 lines
arch/x86/kernel/alternative.c +1
-1@@ -1241,7 +1241,7 @@ static void cfi_rewrite_endbr(s32 *start, s32 *end, struct module *mod)void *addr = (void *)s + *s;void *wr_addr =module_writable_address (mod, addr); -poison_endbr (addr+16 , wr_addr, false); +poison_endbr (addr +16 , wr_addr +16 , false); } }
Cheers,
Nathan
Mike Rapoport
maintainerPATCH v7 7/8 execmem: add support for cache of large ROX pages
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Using large pages to map text areas reduces iTLB pressure and improves performance.
Extend execmem_alloc() with an ability to use huge pages with ROX permissions as a cache for smaller allocations.
To populate the cache, a writable large page is allocated from vmalloc with VM_ALLOW_HUGE_VMAP, filled with invalid instructions and then remapped as ROX.
The direct map alias of that large page is exculded from the direct map.
Portions of that large page are handed out to execmem_alloc() callers without any changes to the permissions.
When the memory is freed with execmem_free() it is invalidated again so that it won't contain stale instructions.
An architecture has to implement execmem_fill_trapping_insns() callback and select ARCH_HAS_EXECMEM_ROX configuration option to be able to use the ROX cache.
The cache is enabled on per-range basis when an architecture sets EXECMEM_ROX_CACHE flag in definition of an execmem_range.
--- arch/Kconfig | 8 + include/linux/execmem.h | 14 ++ mm/execmem.c | 325 +++++++++++++++++++++++++++++++++++++++- mm/internal.h | 1 + mm/vmalloc.c | 5 + 5 files changed, 345 insertions(+), 8 deletions(-)
arch/Kconfig +8 -0
@@ -1010,6 +1010,14 @@ config ARCH_WANTS_EXECMEM_LATE enough entropy for module space randomization, for instance arm64. +config ARCH_HAS_EXECMEM_ROX + bool + depends on MMU && !HIGHMEM + help + For architectures that support allocations of executable memory + with read-only execute permissions. Architecture must implement + execmem_fill_trapping_insns() callback to enable this. + config HAVE_IRQ_EXIT_ON_IRQ_STACK bool help
include/linux/execmem.h +14 -0
@@ -53,6 +53,20 @@ enum execmem_range_flags { EXECMEM_ROX_CACHE = (1 << 1), }; +#ifdef CONFIG_ARCH_HAS_EXECMEM_ROX +/** +* execmem_fill_trapping_insns - set memory to contain instructions that +* will trap +* @ptr: pointer to memory to fill +* @size: size of the range to fill +* @writable: is the memory poited by @ptr is writable or ROX +* +* A hook for architecures to fill execmem ranges with invalid instructions. +* Architectures that use EXECMEM_ROX_CACHE must implement this. +*/ +void execmem_fill_trapping_insns (void *ptr,size_t size,bool writable); +#endif +/** * struct execmem_range - definition of an address space suitable for code and * related data allocations
mm/execmem.c +317 -8
380 lines not on this page · raw message
mm/internal.h +1 -0
@@ -1189,6 +1189,7 @@ size_t splice_folio_into_pipe(struct pipe_inode_info *pipe,void __init vmalloc_init (void );int __must_check vmap_pages_range_noflush (unsigned long addr,unsigned long end,pgprot_t prot,struct page **pages,unsigned int page_shift); +unsigned int get_vm_area_page_order (struct vm_struct *vm);#else static inline void vmalloc_init (void ) {
mm/vmalloc.c +5 -0
@@ -3007,6 +3007,11 @@ static inline unsigned int vm_area_page_order(struct vm_struct *vm)#endif } +unsigned int get_vm_area_page_order (struct vm_struct *vm) +{ +return vm_area_page_order (vm); +} +static inline void set_vm_area_page_order (struct vm_struct *vm,unsigned int order) {#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC
Signature · 1 line
2.43.0
Re: [PATCH v7 7/8] execmem: add support for cache of large ROX pages
Hi Mike,
Drive by review comments below...
Quoted 31 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Using large pages to map text areas reduces iTLB pressure and improves performance.
Extend
execmem_alloc()with an ability to use huge pages with ROX permissions as a cache for smaller allocations.To populate the cache, a writable large page is allocated from vmalloc with
VM_ALLOW_HUGE_VMAP, filled with invalid instructions and then remapped as ROX.The direct map alias of that large page is exculded from the direct map.
Portions of that large page are handed out to
execmem_alloc()callers without any changes to the permissions.When the memory is freed with
execmem_free()it is invalidated again so that it won't contain stale instructions.An architecture has to implement
execmem_fill_trapping_insns()callback and selectARCH_HAS_EXECMEM_ROXconfiguration option to be able to use the ROX cache.The cache is enabled on per-range basis when an architecture sets
EXECMEM_ROX_CACHEflag in definition of anexecmem_range.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Luis Chamberlain <[email protected]>Tested-by kdevops <[email protected]>---
[...]
mm/execmem.c static void *__execmem_alloc(struct execmem_range *range, size_t size) · line 251
Quoted 12 lines
mm/execmem.c +12
-0+ +static int execmem_cache_populate (struct execmem_range *range,size_t size) +{ +unsigned long vm_flags =VM_ALLOW_HUGE_VMAP ; +unsigned long start, end; +struct vm_struct *vm; +size_t alloc_size; +int err = -ENOMEM ; +void *p; + + alloc_size =round_up (size,PMD_SIZE ); + p =execmem_vmalloc (range, alloc_size,PAGE_KERNEL , vm_flags);
Shouldn't this be passing PAGE_KERNEL_ROX? Otherwise I don't see how the allocated memory is ROX? I don't see any call below where you change the permission.
Given the range has the pgprot in it, you could just drop passing the pgprot explicitly here and have execmem_vmalloc() use range->pgprot directly?
Thanks,
Ryan
Quoted 34 lines
mm/execmem.c +34
-0+if (!p) +return err; + + vm =find_vm_area (p); +if (!vm) + goto err_free_mem; + +/* fill memory with instructions that will trap */ +execmem_fill_trapping_insns (p, alloc_size,/* writable = */ true); + + start = (unsigned long )p; + end = start + alloc_size; + +vunmap_range (start, end); + + err =execmem_set_direct_map_valid (vm, false); +if (err) + goto err_free_mem; + + err =vmap_pages_range_noflush (start, end, range->pgprot, vm->pages, +PMD_SHIFT ); +if (err) + goto err_free_mem; + + err =execmem_cache_add (p, alloc_size); +if (err) + goto err_free_mem; + +return 0 ; + +err_free_mem: +vfree (p); +return err; +}
[...]
Mike Rapoport
maintainerRe: [PATCH v7 7/8] execmem: add support for cache of large ROX pages
Hi Ryan,
Quoted 40 lines · Ryan Roberts
Hi Mike,
Drive by review comments below...
Quoted 31 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Using large pages to map text areas reduces iTLB pressure and improves performance.
Extend
execmem_alloc()with an ability to use huge pages with ROX permissions as a cache for smaller allocations.To populate the cache, a writable large page is allocated from vmalloc with
VM_ALLOW_HUGE_VMAP, filled with invalid instructions and then remapped as ROX.The direct map alias of that large page is exculded from the direct map.
Portions of that large page are handed out to
execmem_alloc()callers without any changes to the permissions.When the memory is freed with
execmem_free()it is invalidated again so that it won't contain stale instructions.An architecture has to implement
execmem_fill_trapping_insns()callback and selectARCH_HAS_EXECMEM_ROXconfiguration option to be able to use the ROX cache.The cache is enabled on per-range basis when an architecture sets
EXECMEM_ROX_CACHEflag in definition of anexecmem_range.Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Luis Chamberlain <[email protected]>Tested-by kdevops <[email protected]>---
[...]
Quoted 15 lines
mm/execmem.c static void *__execmem_alloc(struct execmem_range *range, size_t size) · line 251
Quoted 12 lines
mm/execmem.c +12
-0+ +static int execmem_cache_populate (struct execmem_range *range,size_t size) +{ +unsigned long vm_flags =VM_ALLOW_HUGE_VMAP ; +unsigned long start, end; +struct vm_struct *vm; +size_t alloc_size; +int err = -ENOMEM ; +void *p; + + alloc_size =round_up (size,PMD_SIZE ); + p =execmem_vmalloc (range, alloc_size,PAGE_KERNEL , vm_flags);Shouldn't this be passing
PAGE_KERNEL_ROX? Otherwise I don't see how the allocated memory is ROX? I don't see any call below where you change the permission.
The memory is allocated RW, filled with invalid instructions, unammped in vmalloc space, removed from the direct map and then mapped as ROX in vmalloc address space.
Quoted 2 lines
Given the range has the pgprot in it, you could just drop passing the pgprot explicitly here and have
execmem_vmalloc()userange->pgprotdirectly?
Here range->prprot and the prot passed to vmalloc are different.
Quoted 40 lines
Thanks,
RyanQuoted 34 lines
mm/execmem.c +34
-0+if (!p) +return err; + + vm =find_vm_area (p); +if (!vm) + goto err_free_mem; + +/* fill memory with instructions that will trap */ +execmem_fill_trapping_insns (p, alloc_size,/* writable = */ true); + + start = (unsigned long )p; + end = start + alloc_size; + +vunmap_range (start, end); + + err =execmem_set_direct_map_valid (vm, false); +if (err) + goto err_free_mem; + + err =vmap_pages_range_noflush (start, end, range->pgprot, vm->pages, +PMD_SHIFT ); +if (err) + goto err_free_mem; + + err =execmem_cache_add (p, alloc_size); +if (err) + goto err_free_mem; + +return 0 ; + +err_free_mem: +vfree (p); +return err; +}[...]
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerPATCH v7 8/8 x86/module: enable ROX caches for module text on 64 bit
In reply to Mike RapoportFrom: "Mike Rapoport (Microsoft)" <[email protected]>
Enable execmem's cache of PMD_SIZE'ed pages mapped as ROX for module text allocations on 64 bit.
--- arch/x86/Kconfig | 1 + arch/x86/mm/init.c | 37 ++++++++++++++++++++++++++++++++++++- 2 files changed, 37 insertions(+), 1 deletion(-)
arch/x86/Kconfig +1 -0
@@ -83,6 +83,7 @@ config X86 select ARCH_HAS_DMA_OPS if GART_IOMMU || XEN select ARCH_HAS_EARLY_DEBUG if KGDB select ARCH_HAS_ELF_RANDOMIZE + select ARCH_HAS_EXECMEM_ROX if X86_64 select ARCH_HAS_FAST_MULTIPLIER select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL
arch/x86/mm/init.c +36 -1
@@ -1053,18 +1053,53 @@ unsigned long arch_max_swapfile_size(void)#ifdef CONFIG_EXECMEM static struct execmem_info execmem_info__ro_after_init ; +#ifdef CONFIG_ARCH_HAS_EXECMEM_ROX +void execmem_fill_trapping_insns (void *ptr,size_t size,bool writeable) +{ +/* fill memory with INT3 instructions */ +if (writeable) +memset (ptr,INT3_INSN_OPCODE , size); +else +text_poke_set (ptr,INT3_INSN_OPCODE , size); +} +#endif +struct execmem_info __init *execmem_arch_setup (void ) {unsigned long start, offset =0 ; +enum execmem_range_flags flags; +pgprot_t pgprot;if (kaslr_enabled ()) offset =get_random_u32_inclusive (1 ,1024 ) *PAGE_SIZE ; start =MODULES_VADDR + offset; +if (IS_ENABLED (CONFIG_ARCH_HAS_EXECMEM_ROX )) { + pgprot =PAGE_KERNEL_ROX ; + flags =EXECMEM_KASAN_SHADOW |EXECMEM_ROX_CACHE ; + }else { + pgprot =PAGE_KERNEL ; + flags =EXECMEM_KASAN_SHADOW ; + } + execmem_info = (struct execmem_info ){ .ranges = { - [EXECMEM_DEFAULT ] = { + [EXECMEM_MODULE_TEXT ] = { + .flags = flags, + .start = start, + .end =MODULES_END , + .pgprot = pgprot, + .alignment =MODULE_ALIGN , + }, + [EXECMEM_KPROBES ...EXECMEM_BPF ] = { + .flags =EXECMEM_KASAN_SHADOW , + .start = start, + .end =MODULES_END , + .pgprot =PAGE_KERNEL , + .alignment =MODULE_ALIGN , + }, + [EXECMEM_MODULE_DATA ] = { .flags =EXECMEM_KASAN_SHADOW , .start = start, .end =MODULES_END ,
Signature · 1 line
2.43.0
[REGRESSION] Re: [PATCH v7 8/8] x86/module: enable ROX caches for module text on 64 bit
Quoted 4 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Enable execmem's cache of PMD_SIZE'ed pages mapped as ROX for module text allocations on 64 bit.
Hi,
This breaks resume from hibernation on my Alderlake laptop.
Fortunately this still reverts cleanly.
pstore managed to catch the following oops (seems to blow up on bringing up the second ht of the second pcore for whatever reason):
<6>[ 33.993727] PM: hibernation: hibernation entry <6>[ 34.154410] Filesystems sync: 0.006 seconds <6>[ 34.154418] Freezing user space processes <6>[ 34.156019] Freezing user space processes completed (elapsed 0.001 seconds) <6>[ 34.156026] OOM killer disabled. <6>[ 34.160470] PM: hibernation: Preallocating image memory <6>[ 34.330327] PM: hibernation: Allocated 387861 pages for snapshot <6>[ 34.330328] PM: hibernation: Allocated 1551444 kbytes in 0.16 seconds (9696.52 MB/s) <6>[ 34.330330] Freezing remaining freezable tasks <6>[ 34.331499] Freezing remaining freezable tasks completed (elapsed 0.001 seconds) <6>[ 34.351602] printk: Suspending console(s) (use no_console_suspend to debug) <6>[ 34.836184] ACPI: EC: interrupt blocked <6>[ 34.858903] ACPI: PM: Preparing to enter system sleep state S4 <6>[ 34.868434] ACPI: EC: event blocked <6>[ 34.868434] ACPI: EC: EC stopped <6>[ 34.868435] ACPI: PM: Saving platform NVS memory <6>[ 34.872538] Disabling non-boot CPUs ... <6>[ 34.874012] smpboot: CPU 15 is now offline <6>[ 34.875511] smpboot: CPU 14 is now offline <6>[ 34.877209] smpboot: CPU 13 is now offline <6>[ 34.879173] smpboot: CPU 12 is now offline <6>[ 34.881117] smpboot: CPU 11 is now offline <6>[ 34.882952] smpboot: CPU 10 is now offline <6>[ 34.885175] smpboot: CPU 9 is now offline <6>[ 34.886844] smpboot: CPU 8 is now offline <6>[ 34.889056] smpboot: CPU 7 is now offline <6>[ 34.891866] smpboot: CPU 6 is now offline <6>[ 34.893685] smpboot: CPU 5 is now offline <6>[ 34.896856] smpboot: CPU 4 is now offline <6>[ 34.900015] smpboot: CPU 3 is now offline <6>[ 34.902355] smpboot: CPU 2 is now offline <6>[ 34.906162] smpboot: CPU 1 is now offline <6>[ 34.914835] PM: hibernation: Creating image: <6>[ 35.074645] PM: hibernation: Need to copy 348072 pages <6>[ 34.915076] ACPI: PM: Restoring platform NVS memory <6>[ 34.916499] ACPI: EC: EC started <6>[ 34.918228] Enabling non-boot CPUs ... <6>[ 34.918248] smpboot: Booting Node 0 Processor 1 APIC 0x1 <6>[ 34.920121] CPU1 is up <6>[ 34.920132] smpboot: Booting Node 0 Processor 2 APIC 0x8 <6>[ 34.923737] CPU2 is up <6>[ 34.923746] smpboot: Booting Node 0 Processor 3 APIC 0x9 <1>[ 34.927019] BUG: kernel NULL pointer dereference, address: 0000000000000008 <1>[ 34.927023] #PF: supervisor write access in kernel mode <1>[ 34.927024] #PF: error_code(0x0002) - not-present page <6>[ 34.927026] PGD 0 P4D 0 <4>[ 34.927028] Oops: Oops: 0002 [#1] PREEMPT SMP NOPTI <4>[ 34.927030] CPU: 3 UID: 0 PID: 93 Comm: cpuhp/3 Not tainted 6.13.0-rc5+ #406 <4>[ 34.927033] Hardware name: LENOVO 21AJS29M0Q/21AJS29M0Q, BIOS N3MET18W (1.17 ) 10/24/2023 <4>[ 34.927033] RIP: 0010:__rmqueue_pcplist+0x1c9/0x6f0 <4>[ 34.927039] Code: 00 00 48 39 f0 0f 84 7c 01 00 00 49 89 c6 49 83 ee 08 0f 84 6f 01 00 00 48 8b 78 08 41 b8 01 00 00 00 89 ce 4c 8b 08 41 d3 e0 <49> 89 79 08 4c 89 0f 48 bf 00 01 00 00 00 00 ad de 48 89 38 48 83 <4>[ 34.927041] RSP: 0000:ffffc90000457840 EFLAGS: 00010082 <4>[ 34.927042] RAX: ffffea000427af48 RBX: ffff88844e2f0720 RCX: 0000000000000000 <4>[ 34.927043] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff827f0f00 <4>[ 34.927043] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000 <4>[ 34.927044] R10: ffff88844e2f0700 R11: 0000000000000001 R12: 0000000000000000 <4>[ 34.927044] R13: ffffffff827f0e40 R14: ffffea000427af40 R15: ffffffffffffffff <4>[ 34.927045] FS: 0000000000000000(0000) GS:ffff88844e2c0000(0000) knlGS:0000000000000000 <4>[ 34.927046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 34.927046] CR2: 0000000000000008 CR3: 000000000402e001 CR4: 0000000000372ef0 <4>[ 34.927047] Call Trace: <4>[ 34.927048] <TASK> <4>[ 34.927049] ? __die+0x1a/0x60 <4>[ 34.927053] ? page_fault_oops+0x157/0x440 <4>[ 34.927055] ? exc_page_fault+0x444/0x710 <4>[ 34.927058] ? asm_exc_page_fault+0x22/0x30 <4>[ 34.927060] ? __rmqueue_pcplist+0x1c9/0x6f0 <4>[ 34.927061] get_page_from_freelist+0x20f/0xf00 <4>[ 34.927063] __alloc_pages_noprof+0x111/0x240 <4>[ 34.927065] allocate_slab+0x24f/0x3c0 <4>[ 34.927066] ? get_page_from_freelist+0x21a/0xf00 <4>[ 34.927067] ___slab_alloc+0x42a/0xb80 <4>[ 34.927069] ? __kernfs_new_node.isra.0+0x59/0x1e0 <4>[ 34.927071] ? get_nohz_timer_target+0x29/0x140 <4>[ 34.927073] ? timerqueue_add+0x62/0xb0 <4>[ 34.927074] kmem_cache_alloc_noprof+0x17a/0x220 <4>[ 34.927076] __kernfs_new_node.isra.0+0x59/0x1e0 <4>[ 34.927077] ? dl_server_stop+0x26/0x30 <4>[ 34.927079] ? dequeue_entities+0x33b/0x5f0 <4>[ 34.927080] kernfs_new_node+0x3b/0x60 <4>[ 34.927081] kernfs_create_dir_ns+0x22/0x80 <4>[ 34.927083] sysfs_create_dir_ns+0x6b/0xd0 <4>[ 34.927085] kobject_add_internal+0x9e/0x270 <4>[ 34.927087] kobject_add+0x79/0xe0 <4>[ 34.927088] ? __kmalloc_cache_noprof+0x105/0x240 <4>[ 34.927090] device_add+0xb7/0x7d0 <4>[ 34.927092] ? complete_all+0x1b/0x80 <4>[ 34.927095] cpu_device_create+0xea/0x100 <4>[ 34.927097] ? __pfx_smpboot_thread_fn+0x10/0x10 <4>[ 34.927099] ? __set_cpus_allowed_ptr+0x47/0x90 <4>[ 34.927100] ? detect_cache_attributes+0xd2/0x330 <4>[ 34.927102] ? __pfx_cacheinfo_cpu_online+0x10/0x10 <4>[ 34.927104] cacheinfo_cpu_online+0x94/0x270 <4>[ 34.927105] ? __pfx_cacheinfo_cpu_online+0x10/0x10 <4>[ 34.927107] ? __pfx_smpboot_thread_fn+0x10/0x10 <4>[ 34.927108] cpuhp_invoke_callback+0x10e/0x4b0 <4>[ 34.927110] ? __pfx_smpboot_thread_fn+0x10/0x10 <4>[ 34.927111] cpuhp_thread_fun+0xcf/0x150 <4>[ 34.927112] smpboot_thread_fn+0xd1/0x1c0 <4>[ 34.927113] kthread+0xc6/0x100 <4>[ 34.927115] ? __pfx_kthread+0x10/0x10 <4>[ 34.927115] ret_from_fork+0x28/0x40 <4>[ 34.927117] ? __pfx_kthread+0x10/0x10 <4>[ 34.927118] ret_from_fork_asm+0x1a/0x30 <4>[ 34.927120] </TASK> <4>[ 34.927120] Modules linked in: fuse ctr ccm sch_fq_codel cmac xt_multiport xt_tcpudp xt_state iptable_filter xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv4 ip_tables x_tables bnep btusb uvcvideo btrtl btintel videobuf2_vmalloc btbcm videobuf2_memops btmtk uvc videobuf2_v4l2 bluetooth videodev videobuf2_common ecdh_generic mc ecc binfmt_misc snd_hda_codec_hdmi mousedev snd_hda_codec_realtek snd_hda_codec_generic snd_hda_scodec_component snd_ctl_led iwlmvm mac80211 libarc4 i915 snd_hda_intel snd_intel_dspcfg i2c_algo_bit snd_hda_codec drm_buddy xhci_pci intel_tcc_cooling mei_hdcp snd_hwdep ttm mei_pxp mei_wdt processor_thermal_device_pci snd_hda_core x86_pkg_temp_thermal processor_thermal_device xhci_hcd thinkpad_acpi intel_powerclamp 8250_pci processor_thermal_wt_hint drm_display_helper snd_pcm processor_thermal_rfim i2c_hid_acpi coretemp 8250 processor_thermal_rapl nvram i2c_hid iwlwifi kvm_intel drm_kms_helper usbcore intel_rapl_msr 8250_base iTCO_wdt intel_rapl_common platform_profile hid snd_timer <4>[ 34.927180] intel_gtt mei_me serial_mctrl_gpio led_class processor_thermal_wt_req cfg80211 kvm thunderbolt e1000e snd efi_pstore intel_lpss_pci serial_base usb_common agpgart processor_thermal_power_floor spi_intel_pci i2c_i801 intel_lpss mei soundcore processor_thermal_mbox drm rfkill pstore spi_intel mfd_core tpm_tis i2c_smbus psmouse tpm_tis_core zlib_deflate pcspkr evdev tpm think_lmi video intel_pmc_core firmware_attributes_class intel_hid sparse_keymap int3403_thermal intel_vsec pinctrl_tigerlake int3400_thermal wmi int340x_thermal_zone pmt_telemetry i2c_core backlight pinctrl_intel pmt_class acpi_thermal_rel efivarfs <4>[ 34.927209] CR2: 0000000000000008 <4>[ 34.927210] ---[ end trace 0000000000000000 ]---
Quoted 83 lines
Signed-off-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Luis Chamberlain <[email protected]>Tested-by kdevops <[email protected]>--- arch/x86/Kconfig | 1 + arch/x86/mm/init.c | 37 ++++++++++++++++++++++++++++++++++++- 2 files changed, 37 insertions(+), 1 deletion(-)arch/x86/Kconfig +1
-0@@ -83,6 +83,7 @@ config X86 select ARCH_HAS_DMA_OPS if GART_IOMMU || XEN select ARCH_HAS_EARLY_DEBUG if KGDB select ARCH_HAS_ELF_RANDOMIZE + select ARCH_HAS_EXECMEM_ROX if X86_64 select ARCH_HAS_FAST_MULTIPLIER select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALLarch/x86/mm/init.c +36
-1@@ -1053,18 +1053,53 @@ unsigned long arch_max_swapfile_size(void)#ifdef CONFIG_EXECMEM static struct execmem_info execmem_info__ro_after_init ; +#ifdef CONFIG_ARCH_HAS_EXECMEM_ROX +void execmem_fill_trapping_insns (void *ptr,size_t size,bool writeable) +{ +/* fill memory with INT3 instructions */ +if (writeable) +memset (ptr,INT3_INSN_OPCODE , size); +else +text_poke_set (ptr,INT3_INSN_OPCODE , size); +} +#endif +struct execmem_info __init *execmem_arch_setup (void ) {unsigned long start, offset =0 ; +enum execmem_range_flags flags; +pgprot_t pgprot;if (kaslr_enabled ()) offset =get_random_u32_inclusive (1 ,1024 ) *PAGE_SIZE ; start =MODULES_VADDR + offset; +if (IS_ENABLED (CONFIG_ARCH_HAS_EXECMEM_ROX )) { + pgprot =PAGE_KERNEL_ROX ; + flags =EXECMEM_KASAN_SHADOW |EXECMEM_ROX_CACHE ; + }else { + pgprot =PAGE_KERNEL ; + flags =EXECMEM_KASAN_SHADOW ; + } + execmem_info = (struct execmem_info ){ .ranges = { - [EXECMEM_DEFAULT ] = { + [EXECMEM_MODULE_TEXT ] = { + .flags = flags, + .start = start, + .end =MODULES_END , + .pgprot = pgprot, + .alignment =MODULE_ALIGN , + }, + [EXECMEM_KPROBES ...EXECMEM_BPF ] = { + .flags =EXECMEM_KASAN_SHADOW , + .start = start, + .end =MODULES_END , + .pgprot =PAGE_KERNEL , + .alignment =MODULE_ALIGN , + }, + [EXECMEM_MODULE_DATA ] = { .flags =EXECMEM_KASAN_SHADOW , .start = start, .end =MODULES_END ,Signature · 1 line
2.43.0
Signature · 2 lines
Ville Syrjälä Intel
Borislav Petkov
maintainerRe: [REGRESSION] Re: [PATCH v7 8/8] x86/module: enable ROX caches for module text on 64 bit
Quoted 11 lines · Ville Syrjälä
Quoted 4 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Enable execmem's cache of PMD_SIZE'ed pages mapped as ROX for module text allocations on 64 bit.
Hi,
This breaks resume from hibernation on my Alderlake laptop.
Fortunately this still reverts cleanly.
Does that hunk in the mail here fix it?
https://lore.kernel.org/all/[email protected]/
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePeter Zijlstra
maintainerRe: [REGRESSION] Re: [PATCH v7 8/8] x86/module: enable ROX caches for module text on 64 bit
Quoted 9 lines · Borislav Petkov
Quoted 8 lines · Ville Syrjälä
Quoted 4 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Enable execmem's cache of PMD_SIZE'ed pages mapped as ROX for module text allocations on 64 bit.
Hi,
Quoted 7 lines
Quoted 3 lines
This breaks resume from hibernation on my Alderlake laptop.
Fortunately this still reverts cleanly.
Does that hunk in the mail here fix it?
There's definiltely breakage with that module_writable_address() nonsense in alternative.c that will not be fixed by that patch.
The very simplest thing at this point is to remove:
select ARCH_HAS_EXECMEM_ROX if X86_64
and try again next cycle.
Peter Zijlstra
maintainerPATCH 1/1 x86: Disable EXECMEM_ROX support
Quoted 8 lines · Peter Zijlstra
There's definiltely breakage with that
module_writable_address()nonsense inalternative.cthat will not be fixed by that patch.The very simplest thing at this point is to remove:
select
ARCH_HAS_EXECMEM_ROXifX86_64and try again next cycle.
Boris asked I send it as a proper patch, so here goes. Perhaps next time let x86 merge x86 code :/
---
Subject: x86: Disable EXECMEM_ROX support
The whole module_writable_address() nonsense made a giant mess of alternative.c, not to mention it still contains bugs -- notable some of the CFI variants crash and burn.
Mike has been working on patches to clean all this up again, but given the current state of things, this stuff just isn't ready.
Disable for now, lets try again next cycle.
--- arch/x86/Kconfig | 1 - 1 file changed, 1 deletion(-)
arch/x86/Kconfig +0 -1
@@ -83,7 +83,6 @@ config X86 select ARCH_HAS_DMA_OPS if GART_IOMMU || XEN select ARCH_HAS_EARLY_DEBUG if KGDB select ARCH_HAS_ELF_RANDOMIZE - select ARCH_HAS_EXECMEM_ROX if X86_64 select ARCH_HAS_FAST_MULTIPLIER select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL
Borislav Petkov
maintainerRe: [PATCH] x86: Disable EXECMEM_ROX support
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Peter Zijlstra
There's definiltely breakage with that
module_writable_address()nonsense inalternative.cthat will not be fixed by that patch.
Quoted 8 lines
Quoted 5 lines
The very simplest thing at this point is to remove:
select
ARCH_HAS_EXECMEM_ROXifX86_64and try again next cycle.
Boris asked I send it as a proper patch, so here goes. Perhaps next time let x86 merge x86 code :/
I just love it how this went in without a single x86 maintainer Ack, it broke a bunch of things and then it is still there instead of getting reverted.
Let's not do this again please.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette[tip: x86/urgent] x86: Disable EXECMEM_ROX support
In reply to Peter ZijlstraThe following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: a9bbe341333109465605e8733bab0b573cddcc8c Gitweb: https://git.kernel.org/tip/a9bbe341333109465605e8733bab0b573cddcc8c Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 13 Jan 2025 12:29:34 +01:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Mon, 13 Jan 2025 12:42:51 +01:00
x86: Disable EXECMEM_ROX support
The whole module_writable_address() nonsense made a giant mess of alternative.c, not to mention it still contains bugs -- notable some of the CFI variants crash and burn.
Mike has been working on patches to clean all this up again, but given the current state of things, this stuff just isn't ready.
Disable for now, lets try again next cycle.
--- arch/x86/Kconfig | 1 - 1 file changed, 1 deletion(-)
arch/x86/Kconfig +0 -1
@@ -83,7 +83,6 @@ config X86 select ARCH_HAS_DMA_OPS if GART_IOMMU || XEN select ARCH_HAS_EARLY_DEBUG if KGDB select ARCH_HAS_ELF_RANDOMIZE - select ARCH_HAS_EXECMEM_ROX if X86_64 select ARCH_HAS_FAST_MULTIPLIER select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL
Re: [PATCH] x86: Disable EXECMEM_ROX support
In reply to Peter ZijlstraQuoted 37 lines · Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
There's definiltely breakage with that
module_writable_address()nonsense inalternative.cthat will not be fixed by that patch.The very simplest thing at this point is to remove:
select
ARCH_HAS_EXECMEM_ROXifX86_64and try again next cycle.
Boris asked I send it as a proper patch, so here goes. Perhaps next time let x86 merge x86 code :/
---
Subject: x86: DisableEXECMEM_ROXsupportThe whole
module_writable_address()nonsense made a giant mess ofalternative.c, not to mention it still contains bugs -- notable some of the CFI variants crash and burn.Mike has been working on patches to clean all this up again, but given the current state of things, this stuff just isn't ready.
Disable for now, lets try again next cycle.
Fixes 5185e7f9f3bd ("x86/module: enable ROX caches for module text on 64 bit")Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- arch/x86/Kconfig | 1 - 1 file changed, 1 deletion(-)arch/x86/Kconfig +0
-0
arch/x86/Kconfig config X86 · line 88
Quoted 7 lines
arch/x86/Kconfig +0
-1@@ -83,7 +83,6 @@ config X86 select ARCH_HAS_DMA_OPS if GART_IOMMU || XEN select ARCH_HAS_EARLY_DEBUG if KGDB select ARCH_HAS_ELF_RANDOMIZE - select ARCH_HAS_EXECMEM_ROX if X86_64 select ARCH_HAS_FAST_MULTIPLIER select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL
This one works for my hibernate woes.
In case you want it:
Signature · 2 lines
Ville Syrjälä Intel
Re: [REGRESSION] Re: [PATCH v7 8/8] x86/module: enable ROX caches for module text on 64 bit
In reply to Borislav PetkovQuoted 9 lines · Borislav Petkov
Quoted 8 lines · Ville Syrjälä
Quoted 4 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Enable execmem's cache of PMD_SIZE'ed pages mapped as ROX for module text allocations on 64 bit.
Hi,
Quoted 7 lines
Quoted 3 lines
This breaks resume from hibernation on my Alderlake laptop.
Fortunately this still reverts cleanly.
Does that hunk in the mail here fix it?
Still blows up with that one.
Signature · 2 lines
Ville Syrjälä Intel
Steven Rostedt
maintainerQuoted 6 lines · On Wed, 23 Oct 2024 19:27:03 +0300 Mike Rapoport <[email protected]> wrote:
From: "Mike Rapoport (Microsoft)" <[email protected]>
Hi,
This is an updated version of execmem ROX caches.
FYI, I booted a kernel before and after applying these patches with my change:
https://lore.kernel.org/[email protected]
Before these patches:
# cat /sys/kernel/tracing/dyn_ftrace_total_info 57695 pages:231 groups: 9 ftrace boot update time = 14733459 (ns) ftrace module total update time = 449016 (ns)
After:
# cat /sys/kernel/tracing/dyn_ftrace_total_info 57708 pages:231 groups: 9 ftrace boot update time = 47195374 (ns) ftrace module total update time = 592080 (ns)
Which caused boot time to slowdown by over 30ms. That may not seem like much, but we are very concerned about boot time and are fighting every ms we can get.
-- Steve
Mike Rapoport
maintainerQuoted 24 lines · Steven Rostedt
Quoted 6 lines · On Wed, 23 Oct 2024 19:27:03 +0300 Mike Rapoport <[email protected]> wrote:
From: "Mike Rapoport (Microsoft)" <[email protected]>
Hi,
This is an updated version of execmem ROX caches.
FYI, I booted a kernel before and after applying these patches with my change:
https://lore.kernel.org/[email protected]
Before these patches:
# cat /sys/kernel/tracing/dyn_ftrace_total_info 57695 pages:231 groups: 9 ftrace boot update time = 14733459 (ns) ftrace module total update time = 449016 (ns)After:
Quoted 8 lines
# cat /sys/kernel/tracing/dyn_ftrace_total_info 57708 pages:231 groups: 9 ftrace boot update time = 47195374 (ns) ftrace module total update time = 592080 (ns)Which caused boot time to slowdown by over 30ms. That may not seem like much, but we are very concerned about boot time and are fighting every ms we can get.
Hmm, looks like this change was lost in rebase :/
@Andrew, should I send it as a patch on top of mm-stable?
arch/x86/kernel/ftrace.c +10 -5
@@ -111,17 +111,22 @@ static int ftrace_verify_code(unsigned long ip, const char *old_code)*/ static int __ref ftrace_modify_code_direct (unsigned long ip,const char *old_code, -const char *new_code) +const char *new_code,struct module *mod) {int ret =ftrace_verify_code (ip, old_code);if (ret)return ret;/* replace the text with the new text */ -if (ftrace_poke_late) +if (ftrace_poke_late) {text_poke_queue ((void *)ip, new_code,MCOUNT_INSN_SIZE ,NULL ); -else + }else if (!mod) {text_poke_early ((void *)ip, new_code,MCOUNT_INSN_SIZE ); + }else { +mutex_lock (&text_mutex); +text_poke ((void *)ip, new_code,MCOUNT_INSN_SIZE ); +mutex_unlock (&text_mutex); + }return 0 ; }
@@ -142,7 +147,7 @@ int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long ad* just modify the code directly. */ if (addr ==MCOUNT_ADDR ) -return ftrace_modify_code_direct (ip, old, new); +return ftrace_modify_code_direct (ip, old, new, mod);/* * x86 overrides ftrace_replace_code -- this function will never be used
@@ -161,7 +166,7 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) new =ftrace_call_replace (ip, addr);/* Should only be called when module is loaded */ -return ftrace_modify_code_direct (rec->ip, old, new); +return ftrace_modify_code_direct (rec->ip, old, new,NULL ); }/*
Quoted 1 line
-- Steve
Signature · 2 lines
Sincerely yours, Mike.