Skip to contentSkip to conversation

x86/module: rework ROX cache to avoid writable copy

PATCH Applied Mike Rapoport 7 participants 27 messages Jan 26, 2025 – Aug 12

Subsystems (13)
lore
Share & tools
lore

Mike Rapoport

author maintainer
#0 lore raw

v3 cover x86/module: rework ROX cache to avoid writable copy

From: "Mike Rapoport (Microsoft)" <[email protected]>

Hi,

Following Peter's comments [1] these patches rework handling of ROX caches for module text allocations.

Instead of using a writable copy that really complicates alternatives patching, temporarily remap parts of a large ROX page as RW for the time of module formation and then restore it's ROX protections when the module is ready.

To keep the ROX memory mapped with large pages, make set_memory_rox() capable of restoring large pages (more details are in patch 3).

Since this is really about x86, I believe this should go in via tip tree.

The patches also available in git
https://git.kernel.org/rppt/h/execmem/x86-rox/v10

v3 changes:
* instead of adding a new module state handle ROX restoration locally in load_module() as Petr suggested

v2: https://lore.kernel.org/all/[email protected]
* only collapse large mappings in set_memory_rox()
* simplify RW <-> ROX remapping
* don't remove ROX cache pages from the direct map (patch 4)

v1: https://lore.kernel.org/all/[email protected]

[1] https://lore.kernel.org/all/[email protected]

Kirill A. Shutemov (1):
x86/mm/pat: restore large ROX pages after fragmentation

Mike Rapoport (Microsoft) (8):
x86/mm/pat: cpa-test: fix length for CPA_ARRAY test x86/mm/pat: drop duplicate variable in cpa_flush()
execmem: don't remove ROX cache from the direct map
execmem: add API for temporal remapping as RW and restoring ROX afterwards
module: switch to execmem API for remapping as RW and restoring ROX Revert "x86/module: prepare module loading for ROX allocations of text"
module: drop unused module_writable_address()
x86: re-enable EXECMEM_ROX support

 arch/um/kernel/um_arch.c             |  11 +-
 arch/x86/Kconfig                     |   1 +
 arch/x86/entry/vdso/vma.c            |   3 +-
 arch/x86/include/asm/alternative.h   |  14 +-
 arch/x86/include/asm/pgtable_types.h |   2 +
 arch/x86/kernel/alternative.c        | 181 +++++++++-------------
 arch/x86/kernel/ftrace.c             |  30 ++--
 arch/x86/kernel/module.c             |  45 ++----
 arch/x86/mm/pat/cpa-test.c           |   2 +-
 arch/x86/mm/pat/set_memory.c         | 220 ++++++++++++++++++++++++++-
 include/linux/execmem.h              |  31 ++++
 include/linux/module.h               |  16 --
 include/linux/moduleloader.h         |   4 -
 include/linux/vm_event_item.h        |   2 +
 kernel/module/main.c                 |  78 +++-------
 kernel/module/strict_rwx.c           |   9 +-
 mm/execmem.c                         |  39 +++--
 mm/vmstat.c                          |   2 +
 18 files changed, 422 insertions(+), 268 deletions(-)

base-commit: ffd294d346d185b70e28b1a28abe367bbfe53c04

Signature · 1 line
2.45.2
author maintainer
#1 lore raw

PATCH v3 1/9 x86/mm/pat: cpa-test: fix length for CPA_ARRAY test

From: "Mike Rapoport (Microsoft)" <[email protected]>

The CPA_ARRAY test always uses len[1] as numpages argument to change_page_attr_set() although the addresses array is different each iteration of the test loop.

Replace len[1] with len[i] to have numpages matching the addresses array.

Fixes ecc729f1f471 ("x86/mm/cpa: Add ARRAY and PAGES_ARRAY selftests")
---
 arch/x86/mm/pat/cpa-test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/mm/pat/cpa-test.c +1 -1
index 3d2f7f0a6ed1..ad3c1feec990 100644
@@ -183,7 +183,7 @@ static int pageattr_test(void)
 			break;
 
 		case 1:
-			err = change_page_attr_set(addrs, len[1], PAGE_CPA_TEST, 1);
+			err = change_page_attr_set(addrs, len[i], PAGE_CPA_TEST, 1);
 			break;
 
 		case 2:
Signature · 1 line
2.45.2
#2 lore raw

[tip: x86/mm] x86/mm/pat: cpa-test: fix length for CPA_ARRAY test

The following commit has been merged into the x86/mm branch of tip:

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     33ea120582a638b2f2e380a50686c2b1d7cce795
Gitweb:        https://git.kernel.org/tip/33ea120582a638b2f2e380a50686c2b1d7cce795
Author:        Mike Rapoport (Microsoft) <[email protected]>
AuthorDate:    Sun, 26 Jan 2025 09:47:25 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 03 Feb 2025 11:46:01 +01:00

x86/mm/pat: cpa-test: fix length for CPA_ARRAY test

The CPA_ARRAY test always uses len[1] as numpages argument to change_page_attr_set() although the addresses array is different each iteration of the test loop.

Replace len[1] with len[i] to have numpages matching the addresses array.

Fixes ecc729f1f471 ("x86/mm/cpa: Add ARRAY and PAGES_ARRAY selftests")
---
 arch/x86/mm/pat/cpa-test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/mm/pat/cpa-test.c +1 -1
index 3d2f7f0..ad3c1fe 100644
@@ -183,7 +183,7 @@ static int pageattr_test(void)
 			break;
 
 		case 1:
-			err = change_page_attr_set(addrs, len[1], PAGE_CPA_TEST, 1);
+			err = change_page_attr_set(addrs, len[i], PAGE_CPA_TEST, 1);
 			break;
 
 		case 2:
author maintainer
#3 lore raw

PATCH v3 2/9 x86/mm/pat: drop duplicate variable in cpa_flush()

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

There is a 'struct cpa_data *data' parameter in cpa_flush() that is assigned to a local 'struct cpa_data *cpa' variable.

Rename the parameter from 'data' to 'cpa' and drop declaration of the local 'cpa' variable.

---
 arch/x86/mm/pat/set_memory.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/mm/pat/set_memory.c +1 -2
index 95bc50a8541c..d43b919b11ae 100644
@@ -396,9 +396,8 @@ static void __cpa_flush_tlb(void *data)
 		flush_tlb_one_kernel(fix_addr(__cpa_addr(cpa, i)));
 }
 
-static void cpa_flush(struct cpa_data *data, int cache)
+static void cpa_flush(struct cpa_data *cpa, int cache)
 {
-	struct cpa_data *cpa = data;
 	unsigned int i;
 
 	BUG_ON(irqs_disabled() && !early_boot_irqs_disabled);
Signature · 1 line
2.45.2
#4 lore raw

[tip: x86/mm] x86/mm/pat: drop duplicate variable in cpa_flush()

The following commit has been merged into the x86/mm branch of tip:

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     4ee788eb0781ba082709c1ac1d5146ebcc40b967
Gitweb:        https://git.kernel.org/tip/4ee788eb0781ba082709c1ac1d5146ebcc40b967
Author:        Mike Rapoport (Microsoft) <[email protected]>
AuthorDate:    Sun, 26 Jan 2025 09:47:26 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 03 Feb 2025 11:46:01 +01:00

x86/mm/pat: drop duplicate variable in cpa_flush()

There is a 'struct cpa_data *data' parameter in cpa_flush() that is assigned to a local 'struct cpa_data *cpa' variable.

Rename the parameter from 'data' to 'cpa' and drop declaration of the local 'cpa' variable.

---
 arch/x86/mm/pat/set_memory.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/mm/pat/set_memory.c +1 -2
index ef4514d..1f7698c 100644
@@ -394,9 +394,8 @@ static void __cpa_flush_tlb(void *data)
 		flush_tlb_one_kernel(fix_addr(__cpa_addr(cpa, i)));
 }
 
-static void cpa_flush(struct cpa_data *data, int cache)
+static void cpa_flush(struct cpa_data *cpa, int cache)
 {
-	struct cpa_data *cpa = data;
 	unsigned int i;
 
 	BUG_ON(irqs_disabled() && !early_boot_irqs_disabled);
author maintainer
#5 lore raw

PATCH v3 3/9 x86/mm/pat: restore large ROX pages after fragmentation

In reply to Mike Rapoport

From: "Kirill A. Shutemov" <[email protected]>

Change of attributes of the pages may lead to fragmentation of direct mapping over time and performance degradation when these pages contain executable code.

With current code it's one way road: kernel tries to avoid splitting large pages, but it doesn't restore them back even if page attributes got compatible again.

Any change to the mapping may potentially allow to restore large page.

Add a hook to cpa_flush() path that will check if the pages in the range that were just touched can be mapped at PMD level. If the collapse at the PMD level succeeded, also attempt to collapse PUD level.

The collapse logic runs only when a set_memory_ method explicitly sets CPA_COLLAPSE flag, for now this is only enabled in set_memory_rox().

CPUs don't like[1] to have to have TLB entries of different size for the same memory, but looks like it's okay as long as these entries have matching attributes[2]. Therefore it's critical to flush TLB before any following changes to the mapping.

Note that we already allow for multiple TLB entries of different sizes for the same memory now in split_large_page() path. It's not a new situation.

set_memory_4k() provides a way to use 4k pages on purpose. Kernel must not remap such pages as large. Re-use one of software PTE bits to indicate such pages.

[1] See Erratum 383 of AMD Family 10h Processors
[2] https://lore.kernel.org/linux-mm/[email protected]/

[[email protected]:
* s/restore/collapse/
* update formatting per peterz
* use 'struct ptdesc' instead of 'struct page' for list of page tables to be freed
* try to collapse PMD first and if it succeeds move on to PUD as peterz suggested
* flush TLB twice: for changes done in the original CPA call and after collapsing of large pages
* update commit message
]

---
 arch/x86/include/asm/pgtable_types.h |   2 +
 arch/x86/mm/pat/set_memory.c         | 217 ++++++++++++++++++++++++++-
 include/linux/vm_event_item.h        |   2 +
 mm/vmstat.c                          |   2 +
 4 files changed, 219 insertions(+), 4 deletions(-)
arch/x86/include/asm/pgtable_types.h +2 -0
index 4b804531b03c..c90e9c51edb7 100644
@@ -33,6 +33,7 @@
 #define _PAGE_BIT_CPA_TEST	_PAGE_BIT_SOFTW1
 #define _PAGE_BIT_UFFD_WP	_PAGE_BIT_SOFTW2 /* userfaultfd wrprotected */
 #define _PAGE_BIT_SOFT_DIRTY	_PAGE_BIT_SOFTW3 /* software dirty tracking */
+#define _PAGE_BIT_KERNEL_4K	_PAGE_BIT_SOFTW3 /* page must not be converted to large */
 #define _PAGE_BIT_DEVMAP	_PAGE_BIT_SOFTW4
 
 #ifdef CONFIG_X86_64
@@ -64,6 +65,7 @@
 #define _PAGE_PAT_LARGE (_AT(pteval_t, 1) << _PAGE_BIT_PAT_LARGE)
 #define _PAGE_SPECIAL	(_AT(pteval_t, 1) << _PAGE_BIT_SPECIAL)
 #define _PAGE_CPA_TEST	(_AT(pteval_t, 1) << _PAGE_BIT_CPA_TEST)
+#define _PAGE_KERNEL_4K	(_AT(pteval_t, 1) << _PAGE_BIT_KERNEL_4K)
 #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS
 #define _PAGE_PKEY_BIT0	(_AT(pteval_t, 1) << _PAGE_BIT_PKEY_BIT0)
 #define _PAGE_PKEY_BIT1	(_AT(pteval_t, 1) << _PAGE_BIT_PKEY_BIT1)
arch/x86/mm/pat/set_memory.c +213 -4

277 lines not on this page · raw message

include/linux/vm_event_item.h +2 -0
index f70d0958095c..5a37cb2b6f93 100644
@@ -151,6 +151,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 #ifdef CONFIG_X86
 		DIRECT_MAP_LEVEL2_SPLIT,
 		DIRECT_MAP_LEVEL3_SPLIT,
+		DIRECT_MAP_LEVEL2_COLLAPSE,
+		DIRECT_MAP_LEVEL3_COLLAPSE,
 #endif
 #ifdef CONFIG_PER_VMA_LOCK_STATS
 		VMA_LOCK_SUCCESS,
mm/vmstat.c +2 -0
index 16bfe1c694dd..88998725f1c5 100644
@@ -1435,6 +1435,8 @@ const char * const vmstat_text[] = {
 #ifdef CONFIG_X86
 	"direct_map_level2_splits",
 	"direct_map_level3_splits",
+	"direct_map_level2_collapses",
+	"direct_map_level3_collapses",
 #endif
 #ifdef CONFIG_PER_VMA_LOCK_STATS
 	"vma_lock_success",
Signature · 1 line
2.45.2
#6 lore raw

[tip: x86/mm] x86/mm/pat: restore large ROX pages after fragmentation

The following commit has been merged into the x86/mm branch of tip:

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     41d88484c71cd4f659348da41b7b5b3dbd3be1f6
Gitweb:        https://git.kernel.org/tip/41d88484c71cd4f659348da41b7b5b3dbd3be1f6
Author:        Kirill A. Shutemov <[email protected]>
AuthorDate:    Sun, 26 Jan 2025 09:47:27 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 03 Feb 2025 11:46:01 +01:00

x86/mm/pat: restore large ROX pages after fragmentation

Change of attributes of the pages may lead to fragmentation of direct mapping over time and performance degradation when these pages contain executable code.

With current code it's one way road: kernel tries to avoid splitting large pages, but it doesn't restore them back even if page attributes got compatible again.

Any change to the mapping may potentially allow to restore large page.

Add a hook to cpa_flush() path that will check if the pages in the range that were just touched can be mapped at PMD level. If the collapse at the PMD level succeeded, also attempt to collapse PUD level.

The collapse logic runs only when a set_memory_ method explicitly sets CPA_COLLAPSE flag, for now this is only enabled in set_memory_rox().

CPUs don't like[1] to have to have TLB entries of different size for the same memory, but looks like it's okay as long as these entries have matching attributes[2]. Therefore it's critical to flush TLB before any following changes to the mapping.

Note that we already allow for multiple TLB entries of different sizes for the same memory now in split_large_page() path. It's not a new situation.

set_memory_4k() provides a way to use 4k pages on purpose. Kernel must not remap such pages as large. Re-use one of software PTE bits to indicate such pages.

[1] See Erratum 383 of AMD Family 10h Processors
[2] https://lore.kernel.org/linux-mm/[email protected]/

[[email protected]:
* s/restore/collapse/
* update formatting per peterz
* use 'struct ptdesc' instead of 'struct page' for list of page tables to be freed
* try to collapse PMD first and if it succeeds move on to PUD as peterz suggested
* flush TLB twice: for changes done in the original CPA call and after collapsing of large pages
* update commit message
]

---
 arch/x86/include/asm/pgtable_types.h |   2 +-
 arch/x86/mm/pat/set_memory.c         | 217 +++++++++++++++++++++++++-
 include/linux/vm_event_item.h        |   2 +-
 mm/vmstat.c                          |   2 +-
 4 files changed, 219 insertions(+), 4 deletions(-)
arch/x86/include/asm/pgtable_types.h +2 -0
index 4b80453..c90e9c5 100644
@@ -33,6 +33,7 @@
 #define _PAGE_BIT_CPA_TEST	_PAGE_BIT_SOFTW1
 #define _PAGE_BIT_UFFD_WP	_PAGE_BIT_SOFTW2 /* userfaultfd wrprotected */
 #define _PAGE_BIT_SOFT_DIRTY	_PAGE_BIT_SOFTW3 /* software dirty tracking */
+#define _PAGE_BIT_KERNEL_4K	_PAGE_BIT_SOFTW3 /* page must not be converted to large */
 #define _PAGE_BIT_DEVMAP	_PAGE_BIT_SOFTW4
 
 #ifdef CONFIG_X86_64
@@ -64,6 +65,7 @@
 #define _PAGE_PAT_LARGE (_AT(pteval_t, 1) << _PAGE_BIT_PAT_LARGE)
 #define _PAGE_SPECIAL	(_AT(pteval_t, 1) << _PAGE_BIT_SPECIAL)
 #define _PAGE_CPA_TEST	(_AT(pteval_t, 1) << _PAGE_BIT_CPA_TEST)
+#define _PAGE_KERNEL_4K	(_AT(pteval_t, 1) << _PAGE_BIT_KERNEL_4K)
 #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS
 #define _PAGE_PKEY_BIT0	(_AT(pteval_t, 1) << _PAGE_BIT_PKEY_BIT0)
 #define _PAGE_PKEY_BIT1	(_AT(pteval_t, 1) << _PAGE_BIT_PKEY_BIT1)
arch/x86/mm/pat/set_memory.c +213 -4

277 lines not on this page · raw message

include/linux/vm_event_item.h +2 -0
index f70d095..5a37cb2 100644
@@ -151,6 +151,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 #ifdef CONFIG_X86
 		DIRECT_MAP_LEVEL2_SPLIT,
 		DIRECT_MAP_LEVEL3_SPLIT,
+		DIRECT_MAP_LEVEL2_COLLAPSE,
+		DIRECT_MAP_LEVEL3_COLLAPSE,
 #endif
 #ifdef CONFIG_PER_VMA_LOCK_STATS
 		VMA_LOCK_SUCCESS,
mm/vmstat.c +2 -0
index 16bfe1c..8899872 100644
@@ -1435,6 +1435,8 @@ const char * const vmstat_text[] = {
 #ifdef CONFIG_X86
 	"direct_map_level2_splits",
 	"direct_map_level3_splits",
+	"direct_map_level2_collapses",
+	"direct_map_level3_collapses",
 #endif
 #ifdef CONFIG_PER_VMA_LOCK_STATS
 	"vma_lock_success",
author maintainer
#7 lore raw

PATCH v3 4/9 execmem: don't remove ROX cache from the direct map

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

The memory allocated for the ROX cache was removed from the direct map to reduce amount of direct map updates, however this cannot be tolerated by /proc/kcore that accesses module memory using vread_iter() and the latter does vmalloc_to_page() and copy_page_to_iter_nofault().

Instead of removing ROX cache memory from the direct map and mapping it as ROX in vmalloc space, simply call set_memory_rox() that will take care of proper permissions on both vmalloc and in the direct map.

---
 mm/execmem.c | 17 ++++-------------
 1 file changed, 4 insertions(+), 13 deletions(-)
mm/execmem.c +4 -13
index 317b6a8d35be..04b0bf1b5025 100644
@@ -257,7 +257,6 @@ static void *__execmem_cache_alloc(struct execmem_range *range, size_t size)
 static int execmem_cache_populate(struct execmem_range *range, size_t size)
 {
 	unsigned long vm_flags = VM_ALLOW_HUGE_VMAP;
-	unsigned long start, end;
 	struct vm_struct *vm;
 	size_t alloc_size;
 	int err = -ENOMEM;
@@ -275,26 +274,18 @@ static int execmem_cache_populate(struct execmem_range *range, size_t size)
 	/* fill memory with instructions that will trap */
 	execmem_fill_trapping_insns(p, alloc_size, /* writable = */ true);
 
-	start = (unsigned long)p;
-	end = start + alloc_size;
-
-	vunmap_range(start, end);
-
-	err = execmem_set_direct_map_valid(vm, false);
-	if (err)
-		goto err_free_mem;
-
-	err = vmap_pages_range_noflush(start, end, range->pgprot, vm->pages,
-				       PMD_SHIFT);
+	err = set_memory_rox((unsigned long)p, vm->nr_pages);
 	if (err)
 		goto err_free_mem;
 
 	err = execmem_cache_add(p, alloc_size);
 	if (err)
-		goto err_free_mem;
+		goto err_reset_direct_map;
 
 	return 0;
 
+err_reset_direct_map:
+	execmem_set_direct_map_valid(vm, true);
 err_free_mem:
 	vfree(p);
 	return err;
Signature · 1 line
2.45.2
#8 lore raw

[tip: x86/mm] execmem: don't remove ROX cache from the direct map

The following commit has been merged into the x86/mm branch of tip:

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     925f426451182a9f3e0f7f0e7e928f32f81a966a
Gitweb:        https://git.kernel.org/tip/925f426451182a9f3e0f7f0e7e928f32f81a966a
Author:        Mike Rapoport (Microsoft) <[email protected]>
AuthorDate:    Sun, 26 Jan 2025 09:47:28 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 03 Feb 2025 11:46:01 +01:00

execmem: don't remove ROX cache from the direct map

The memory allocated for the ROX cache was removed from the direct map to reduce amount of direct map updates, however this cannot be tolerated by /proc/kcore that accesses module memory using vread_iter() and the latter does vmalloc_to_page() and copy_page_to_iter_nofault().

Instead of removing ROX cache memory from the direct map and mapping it as ROX in vmalloc space, simply call set_memory_rox() that will take care of proper permissions on both vmalloc and in the direct map.

---
 mm/execmem.c | 17 ++++-------------
 1 file changed, 4 insertions(+), 13 deletions(-)
mm/execmem.c +4 -13
index 317b6a8..04b0bf1 100644
@@ -257,7 +257,6 @@ out_unlock:
 static int execmem_cache_populate(struct execmem_range *range, size_t size)
 {
 	unsigned long vm_flags = VM_ALLOW_HUGE_VMAP;
-	unsigned long start, end;
 	struct vm_struct *vm;
 	size_t alloc_size;
 	int err = -ENOMEM;
@@ -275,26 +274,18 @@ static int execmem_cache_populate(struct execmem_range *range, size_t size)
 	/* fill memory with instructions that will trap */
 	execmem_fill_trapping_insns(p, alloc_size, /* writable = */ true);
 
-	start = (unsigned long)p;
-	end = start + alloc_size;
-
-	vunmap_range(start, end);
-
-	err = execmem_set_direct_map_valid(vm, false);
-	if (err)
-		goto err_free_mem;
-
-	err = vmap_pages_range_noflush(start, end, range->pgprot, vm->pages,
-				       PMD_SHIFT);
+	err = set_memory_rox((unsigned long)p, vm->nr_pages);
 	if (err)
 		goto err_free_mem;
 
 	err = execmem_cache_add(p, alloc_size);
 	if (err)
-		goto err_free_mem;
+		goto err_reset_direct_map;
 
 	return 0;
 
+err_reset_direct_map:
+	execmem_set_direct_map_valid(vm, true);
 err_free_mem:
 	vfree(p);
 	return err;
author maintainer
#9 lore raw

PATCH v3 5/9 execmem: add API for temporal remapping as RW and restoring ROX afterwards

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Using a writable copy for ROX memory is cumbersome and error prone.

Add API that allow temporarily remapping of ranges in the ROX cache as writable and then restoring their read-only-execute permissions.

This API will be later used in modules code and will allow removing nasty games with writable copy in alternatives patching on x86.

The restoring of the ROX permissions relies on the ability of architecture to reconstruct large pages in its set_memory_rox() method.

---
 include/linux/execmem.h | 31 +++++++++++++++++++++++++++++++
 mm/execmem.c            | 22 ++++++++++++++++++++++
 2 files changed, 53 insertions(+)
include/linux/execmem.h +31 -0
index 64130ae19690..65655a5d1be2 100644
@@ -65,6 +65,37 @@ enum execmem_range_flags {
  * Architectures that use EXECMEM_ROX_CACHE must implement this.
  */
 void execmem_fill_trapping_insns(void *ptr, size_t size, bool writable);
+
+/**
+ * execmem_make_temp_rw - temporarily remap region with read-write
+ *			  permissions
+ * @ptr:	address of the region to remap
+ * @size:	size of the region to remap
+ *
+ * Remaps a part of the cached large page in the ROX cache in the range
+ * [@ptr, @ptr + @size) as writable and not executable. The caller must
+ * have exclusive ownership of this range and ensure nothing will try to
+ * execute code in this range.
+ *
+ * Return: 0 on success or negative error code on failure.
+ */
+int execmem_make_temp_rw(void *ptr, size_t size);
+
+/**
+ * execmem_restore_rox - restore read-only-execute permissions
+ * @ptr:	address of the region to remap
+ * @size:	size of the region to remap
+ *
+ * Restores read-only-execute permissions on a range [@ptr, @ptr + @size)
+ * after it was temporarily remapped as writable. Relies on architecture
+ * implementation of set_memory_rox() to restore mapping using large pages.
+ *
+ * Return: 0 on success or negative error code on failure.
+ */
+int execmem_restore_rox(void *ptr, size_t size);
+#else
+static inline int execmem_make_temp_rw(void *ptr, size_t size) { return 0; }
+static inline int execmem_restore_rox(void *ptr, size_t size) { return 0; }
 #endif
 
 /**
mm/execmem.c +22 -0
index 04b0bf1b5025..e6c4f5076ca8 100644
@@ -335,6 +335,28 @@ static bool execmem_cache_free(void *ptr)
 
 	return true;
 }
+
+int execmem_make_temp_rw(void *ptr, size_t size)
+{
+	unsigned int nr = PAGE_ALIGN(size) >> PAGE_SHIFT;
+	unsigned long addr = (unsigned long)ptr;
+	int ret;
+
+	ret = set_memory_nx(addr, nr);
+	if (ret)
+		return ret;
+
+	return set_memory_rw(addr, nr);
+}
+
+int execmem_restore_rox(void *ptr, size_t size)
+{
+	unsigned int nr = PAGE_ALIGN(size) >> PAGE_SHIFT;
+	unsigned long addr = (unsigned long)ptr;
+
+	return set_memory_rox(addr, nr);
+}
+
 #else /* CONFIG_ARCH_HAS_EXECMEM_ROX */
 static void *execmem_cache_alloc(struct execmem_range *range, size_t size)
 {
Signature · 1 line
2.45.2
#10 lore raw

[tip: x86/mm] execmem: add API for temporal remapping as RW and restoring ROX afterwards

The following commit has been merged into the x86/mm branch of tip:

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     05e555b817262b5df6aa3a73df8b3dc9d388a3b4
Gitweb:        https://git.kernel.org/tip/05e555b817262b5df6aa3a73df8b3dc9d388a3b4
Author:        Mike Rapoport (Microsoft) <[email protected]>
AuthorDate:    Sun, 26 Jan 2025 09:47:29 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 03 Feb 2025 11:46:02 +01:00

execmem: add API for temporal remapping as RW and restoring ROX afterwards

Using a writable copy for ROX memory is cumbersome and error prone.

Add API that allow temporarily remapping of ranges in the ROX cache as writable and then restoring their read-only-execute permissions.

This API will be later used in modules code and will allow removing nasty games with writable copy in alternatives patching on x86.

The restoring of the ROX permissions relies on the ability of architecture to reconstruct large pages in its set_memory_rox() method.

---
 include/linux/execmem.h | 31 +++++++++++++++++++++++++++++++
 mm/execmem.c            | 22 ++++++++++++++++++++++
 2 files changed, 53 insertions(+)
include/linux/execmem.h +31 -0
index 64130ae..65655a5 100644
@@ -65,6 +65,37 @@ enum execmem_range_flags {
  * Architectures that use EXECMEM_ROX_CACHE must implement this.
  */
 void execmem_fill_trapping_insns(void *ptr, size_t size, bool writable);
+
+/**
+ * execmem_make_temp_rw - temporarily remap region with read-write
+ *			  permissions
+ * @ptr:	address of the region to remap
+ * @size:	size of the region to remap
+ *
+ * Remaps a part of the cached large page in the ROX cache in the range
+ * [@ptr, @ptr + @size) as writable and not executable. The caller must
+ * have exclusive ownership of this range and ensure nothing will try to
+ * execute code in this range.
+ *
+ * Return: 0 on success or negative error code on failure.
+ */
+int execmem_make_temp_rw(void *ptr, size_t size);
+
+/**
+ * execmem_restore_rox - restore read-only-execute permissions
+ * @ptr:	address of the region to remap
+ * @size:	size of the region to remap
+ *
+ * Restores read-only-execute permissions on a range [@ptr, @ptr + @size)
+ * after it was temporarily remapped as writable. Relies on architecture
+ * implementation of set_memory_rox() to restore mapping using large pages.
+ *
+ * Return: 0 on success or negative error code on failure.
+ */
+int execmem_restore_rox(void *ptr, size_t size);
+#else
+static inline int execmem_make_temp_rw(void *ptr, size_t size) { return 0; }
+static inline int execmem_restore_rox(void *ptr, size_t size) { return 0; }
 #endif
 
 /**
mm/execmem.c +22 -0
index 04b0bf1..e6c4f50 100644
@@ -335,6 +335,28 @@ static bool execmem_cache_free(void *ptr)
 
 	return true;
 }
+
+int execmem_make_temp_rw(void *ptr, size_t size)
+{
+	unsigned int nr = PAGE_ALIGN(size) >> PAGE_SHIFT;
+	unsigned long addr = (unsigned long)ptr;
+	int ret;
+
+	ret = set_memory_nx(addr, nr);
+	if (ret)
+		return ret;
+
+	return set_memory_rw(addr, nr);
+}
+
+int execmem_restore_rox(void *ptr, size_t size)
+{
+	unsigned int nr = PAGE_ALIGN(size) >> PAGE_SHIFT;
+	unsigned long addr = (unsigned long)ptr;
+
+	return set_memory_rox(addr, nr);
+}
+
 #else /* CONFIG_ARCH_HAS_EXECMEM_ROX */
 static void *execmem_cache_alloc(struct execmem_range *range, size_t size)
 {
author maintainer
#11 lore raw

PATCH v3 6/9 module: switch to execmem API for remapping as RW and restoring ROX

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Instead of using writable copy for module text sections, temporarily remap the memory allocated from execmem's ROX cache as writable and restore its ROX permissions after the module is formed.

This will allow removing nasty games with writable copy in alternatives patching on x86.

---
 include/linux/module.h       |  8 +---
 include/linux/moduleloader.h |  4 --
 kernel/module/main.c         | 78 ++++++++++--------------------------
 kernel/module/strict_rwx.c   |  9 +++--
 4 files changed, 27 insertions(+), 72 deletions(-)
include/linux/module.h +1 -7
index b3a643435357..6a24e9395cb2 100644
@@ -367,7 +367,6 @@ enum mod_mem_type {
 
 struct module_memory {
 	void *base;
-	void *rw_copy;
 	bool is_rox;
 	unsigned int size;
 
@@ -769,14 +768,9 @@ static inline bool is_livepatch_module(struct module *mod)
 
 void set_module_sig_enforced(void);
 
-void *__module_writable_address(struct module *mod, void *loc);
-
 static inline void *module_writable_address(struct module *mod, void *loc)
 {
-	if (!IS_ENABLED(CONFIG_ARCH_HAS_EXECMEM_ROX) || !mod ||
-	    mod->state != MODULE_STATE_UNFORMED)
-		return loc;
-	return __module_writable_address(mod, loc);
+	return loc;
 }
 
 #else /* !CONFIG_MODULES... */
include/linux/moduleloader.h +0 -4
index 1f5507ba5a12..e395461d59e5 100644
@@ -108,10 +108,6 @@ int module_finalize(const Elf_Ehdr *hdr,
 		    const Elf_Shdr *sechdrs,
 		    struct module *mod);
 
-int module_post_finalize(const Elf_Ehdr *hdr,
-			 const Elf_Shdr *sechdrs,
-			 struct module *mod);
-
 #ifdef CONFIG_MODULES
 void flush_module_init_free_work(void);
 #else
kernel/module/main.c +21 -57

164 lines not on this page · raw message

kernel/module/strict_rwx.c +5 -4
index 239e5013359d..ce47b6346f27 100644
@@ -9,6 +9,7 @@
 #include <linux/mm.h>
 #include <linux/vmalloc.h>
 #include <linux/set_memory.h>
+#include <linux/execmem.h>
 #include "internal.h"
 
 static int module_set_memory(const struct module *mod, enum mod_mem_type type,
@@ -32,12 +33,12 @@ static int module_set_memory(const struct module *mod, enum mod_mem_type type,
 int module_enable_text_rox(const struct module *mod)
 {
 	for_class_mod_mem_type(type, text) {
+		const struct module_memory *mem = &mod->mem[type];
 		int ret;
 
-		if (mod->mem[type].is_rox)
-			continue;
-
-		if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX))
+		if (mem->is_rox)
+			ret = execmem_restore_rox(mem->base, mem->size);
+		else if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX))
 			ret = module_set_memory(mod, type, set_memory_rox);
 		else
 			ret = module_set_memory(mod, type, set_memory_x);
Signature · 1 line
2.45.2
maintainer
#12 lore raw

Re: [PATCH v3 6/9] module: switch to execmem API for remapping as RW and restoring ROX

Quoted 10 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Instead of using writable copy for module text sections, temporarily remap the memory allocated from execmem's ROX cache as writable and restore its ROX permissions after the module is formed.

This will allow removing nasty games with writable copy in alternatives patching on x86.

[...]

kernel/module/main.c static int module_memory_alloc(struct module *mod, enum mod_mem_type type) · line 1280

Quoted 10 lines
kernel/module/main.c +10 -0
+static void module_memory_restore_rox(struct module *mod)
+{
+	for_class_mod_mem_type(type, text) {
+		struct module_memory *mem = &mod->mem[type];
+
+		if (mem->is_rox)
+			execmem_restore_rox(mem->base, mem->size);
+	}
+}
+

Can the execmem_restore_rox() call here fail? I realize that there isn't much that the module loader can do if that happens, but should it be perhaps logged as a warning?

Signature · 2 lines
Thanks,
Petr
author maintainer
#13 lore raw

Re: [PATCH v3 6/9] module: switch to execmem API for remapping as RW and restoring ROX

Quoted 14 lines · Petr Pavlu
Quoted 10 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Instead of using writable copy for module text sections, temporarily remap the memory allocated from execmem's ROX cache as writable and restore its ROX permissions after the module is formed.

This will allow removing nasty games with writable copy in alternatives patching on x86.

[...]

Quoted 14 lines

kernel/module/main.c static int module_memory_alloc(struct module *mod, enum mod_mem_type type) · line 1280

Quoted 10 lines
kernel/module/main.c +10 -0
+static void module_memory_restore_rox(struct module *mod)
+{
+	for_class_mod_mem_type(type, text) {
+		struct module_memory *mem = &mod->mem[type];
+
+		if (mem->is_rox)
+			execmem_restore_rox(mem->base, mem->size);
+	}
+}
+

Can the execmem_restore_rox() call here fail? I realize that there isn't much that the module loader can do if that happens, but should it be perhaps logged as a warning?

It won't fail at this point. set_memory APIs may fail if they need to split a large page and could not allocate a new page table, but here all the splits were already done at module_memory_alloc() time.

Quoted 3 lines
Signature · 2 lines
Thanks,
Petr
Signature · 2 lines
Sincerely yours,
Mike.
maintainer Acked-by
#14 lore raw

Re: [PATCH v3 6/9] module: switch to execmem API for remapping as RW and restoring ROX

Quoted 30 lines · Mike Rapoport
Quoted 14 lines · Petr Pavlu
Quoted 10 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Instead of using writable copy for module text sections, temporarily remap the memory allocated from execmem's ROX cache as writable and restore its ROX permissions after the module is formed.

This will allow removing nasty games with writable copy in alternatives patching on x86.

[...]

Quoted 14 lines

kernel/module/main.c static int module_memory_alloc(struct module *mod, enum mod_mem_type type) · line 1280

Quoted 10 lines
kernel/module/main.c +10 -0
+static void module_memory_restore_rox(struct module *mod)
+{
+	for_class_mod_mem_type(type, text) {
+		struct module_memory *mem = &mod->mem[type];
+
+		if (mem->is_rox)
+			execmem_restore_rox(mem->base, mem->size);
+	}
+}
+

Can the execmem_restore_rox() call here fail? I realize that there isn't much that the module loader can do if that happens, but should it be perhaps logged as a warning?

Quoted 3 lines

It won't fail at this point. set_memory APIs may fail if they need to split a large page and could not allocate a new page table, but here all the splits were already done at module_memory_alloc() time.

Ok, thanks for the explanation.

-- Petr

#15 lore raw

[tip: x86/mm] module: switch to execmem API for remapping as RW and restoring ROX

In reply to Mike Rapoport
The following commit has been merged into the x86/mm branch of tip:

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     c287c072332905b7d878a8aade86cfef6b396343
Gitweb:        https://git.kernel.org/tip/c287c072332905b7d878a8aade86cfef6b396343
Author:        Mike Rapoport (Microsoft) <[email protected]>
AuthorDate:    Sun, 26 Jan 2025 09:47:30 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 03 Feb 2025 11:46:02 +01:00

module: switch to execmem API for remapping as RW and restoring ROX

Instead of using writable copy for module text sections, temporarily remap the memory allocated from execmem's ROX cache as writable and restore its ROX permissions after the module is formed.

This will allow removing nasty games with writable copy in alternatives patching on x86.

---
 include/linux/module.h       |  8 +----
 include/linux/moduleloader.h |  4 +--
 kernel/module/main.c         | 78 +++++++++--------------------------
 kernel/module/strict_rwx.c   |  9 ++--
 4 files changed, 27 insertions(+), 72 deletions(-)
include/linux/module.h +1 -7
index 23792d5..ddf27ed 100644
@@ -367,7 +367,6 @@ enum mod_mem_type {
 
 struct module_memory {
 	void *base;
-	void *rw_copy;
 	bool is_rox;
 	unsigned int size;
 
@@ -769,14 +768,9 @@ static inline bool is_livepatch_module(struct module *mod)
 
 void set_module_sig_enforced(void);
 
-void *__module_writable_address(struct module *mod, void *loc);
-
 static inline void *module_writable_address(struct module *mod, void *loc)
 {
-	if (!IS_ENABLED(CONFIG_ARCH_HAS_EXECMEM_ROX) || !mod ||
-	    mod->state != MODULE_STATE_UNFORMED)
-		return loc;
-	return __module_writable_address(mod, loc);
+	return loc;
 }
 
 #else /* !CONFIG_MODULES... */
include/linux/moduleloader.h +0 -4
index 1f5507b..e395461 100644
@@ -108,10 +108,6 @@ int module_finalize(const Elf_Ehdr *hdr,
 		    const Elf_Shdr *sechdrs,
 		    struct module *mod);
 
-int module_post_finalize(const Elf_Ehdr *hdr,
-			 const Elf_Shdr *sechdrs,
-			 struct module *mod);
-
 #ifdef CONFIG_MODULES
 void flush_module_init_free_work(void);
 #else
kernel/module/main.c +21 -57

164 lines not on this page · raw message

kernel/module/strict_rwx.c +5 -4
index 74834ba..03f4142 100644
@@ -9,6 +9,7 @@
 #include <linux/mm.h>
 #include <linux/vmalloc.h>
 #include <linux/set_memory.h>
+#include <linux/execmem.h>
 #include "internal.h"
 
 static int module_set_memory(const struct module *mod, enum mod_mem_type type,
@@ -32,12 +33,12 @@ static int module_set_memory(const struct module *mod, enum mod_mem_type type,
 int module_enable_text_rox(const struct module *mod)
 {
 	for_class_mod_mem_type(type, text) {
+		const struct module_memory *mem = &mod->mem[type];
 		int ret;
 
-		if (mod->mem[type].is_rox)
-			continue;
-
-		if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX))
+		if (mem->is_rox)
+			ret = execmem_restore_rox(mem->base, mem->size);
+		else if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX))
 			ret = module_set_memory(mod, type, set_memory_rox);
 		else
 			ret = module_set_memory(mod, type, set_memory_x);
author maintainer
#16 lore raw

PATCH v3 7/9 Revert "x86/module: prepare module loading for ROX allocations of text"

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

The module code does not create a writable copy of the executable memory anymore so there is no need to handle it in module relocation and alternatives patching.

This reverts commit 9bfc4824fd4836c16bb44f922bfaffba5da3e4f3.

---
 arch/um/kernel/um_arch.c           |  11 +-
 arch/x86/entry/vdso/vma.c          |   3 +-
 arch/x86/include/asm/alternative.h |  14 +--
 arch/x86/kernel/alternative.c      | 181 ++++++++++++-----------------
 arch/x86/kernel/ftrace.c           |  30 +++--
 arch/x86/kernel/module.c           |  45 +++----
 6 files changed, 117 insertions(+), 167 deletions(-)
arch/um/kernel/um_arch.c +5 -6
index 8037a967225d..d2cc2c69a8c4 100644
@@ -440,25 +440,24 @@ void __init arch_cpu_finalize_init(void)
 	os_check_bugs();
 }
 
-void apply_seal_endbr(s32 *start, s32 *end, struct module *mod)
+void apply_seal_endbr(s32 *start, s32 *end)
 {
 }
 
-void apply_retpolines(s32 *start, s32 *end, struct module *mod)
+void apply_retpolines(s32 *start, s32 *end)
 {
 }
 
-void apply_returns(s32 *start, s32 *end, struct module *mod)
+void apply_returns(s32 *start, s32 *end)
 {
 }
 
 void apply_fineibt(s32 *start_retpoline, s32 *end_retpoline,
-		   s32 *start_cfi, s32 *end_cfi, struct module *mod)
+		   s32 *start_cfi, s32 *end_cfi)
 {
 }
 
-void apply_alternatives(struct alt_instr *start, struct alt_instr *end,
-			struct module *mod)
+void apply_alternatives(struct alt_instr *start, struct alt_instr *end)
 {
 }
 
arch/x86/entry/vdso/vma.c +1 -2
index 39e6efc1a9ca..bfc7cabf4017 100644
@@ -48,8 +48,7 @@ int __init init_vdso_image(const struct vdso_image *image)
 
 	apply_alternatives((struct alt_instr *)(image->data + image->alt),
 			   (struct alt_instr *)(image->data + image->alt +
-						image->alt_len),
-			   NULL);
+						image->alt_len));
 
 	return 0;
 }
arch/x86/include/asm/alternative.h +7 -7
index dc03a647776d..ca9ae606aab9 100644
@@ -96,16 +96,16 @@ extern struct alt_instr __alt_instructions[], __alt_instructions_end[];
  * instructions were patched in already:
  */
 extern int alternatives_patched;
-struct module;
 
 extern void alternative_instructions(void);
-extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end,
-			       struct module *mod);
-extern void apply_retpolines(s32 *start, s32 *end, struct module *mod);
-extern void apply_returns(s32 *start, s32 *end, struct module *mod);
-extern void apply_seal_endbr(s32 *start, s32 *end, struct module *mod);
+extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end);
+extern void apply_retpolines(s32 *start, s32 *end);
+extern void apply_returns(s32 *start, s32 *end);
+extern void apply_seal_endbr(s32 *start, s32 *end);
 extern void apply_fineibt(s32 *start_retpoline, s32 *end_retpoine,
-			  s32 *start_cfi, s32 *end_cfi, struct module *mod);
+			  s32 *start_cfi, s32 *end_cfi);
+
+struct module;
 
 struct callthunk_sites {
 	s32				*call_start, *call_end;
arch/x86/kernel/alternative.c +76 -105

535 lines not on this page · raw message

arch/x86/kernel/ftrace.c +14 -16
index 4dd0ad6c94d6..adb09f78edb2 100644
@@ -118,13 +118,10 @@ ftrace_modify_code_direct(unsigned long ip, const char *old_code,
 		return ret;
 
 	/* replace the text with the new text */
-	if (ftrace_poke_late) {
+	if (ftrace_poke_late)
 		text_poke_queue((void *)ip, new_code, MCOUNT_INSN_SIZE, NULL);
-	} else {
-		mutex_lock(&text_mutex);
-		text_poke((void *)ip, new_code, MCOUNT_INSN_SIZE);
-		mutex_unlock(&text_mutex);
-	}
+	else
+		text_poke_early((void *)ip, new_code, MCOUNT_INSN_SIZE);
 	return 0;
 }
 
@@ -321,7 +318,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	unsigned const char op_ref[] = { 0x48, 0x8b, 0x15 };
 	unsigned const char retq[] = { RET_INSN_OPCODE, INT3_INSN_OPCODE };
 	union ftrace_op_code_union op_ptr;
-	void *ret;
+	int ret;
 
 	if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) {
 		start_offset = (unsigned long)ftrace_regs_caller;
@@ -352,15 +349,15 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	npages = DIV_ROUND_UP(*tramp_size, PAGE_SIZE);
 
 	/* Copy ftrace_caller onto the trampoline memory */
-	ret = text_poke_copy(trampoline, (void *)start_offset, size);
-	if (WARN_ON(!ret))
+	ret = copy_from_kernel_nofault(trampoline, (void *)start_offset, size);
+	if (WARN_ON(ret < 0))
 		goto fail;
 
 	ip = trampoline + size;
 	if (cpu_feature_enabled(X86_FEATURE_RETHUNK))
 		__text_gen_insn(ip, JMP32_INSN_OPCODE, ip, x86_return_thunk, JMP32_INSN_SIZE);
 	else
-		text_poke_copy(ip, retq, sizeof(retq));
+		memcpy(ip, retq, sizeof(retq));
 
 	/* No need to test direct calls on created trampolines */
 	if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) {
@@ -368,7 +365,8 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 		ip = trampoline + (jmp_offset - start_offset);
 		if (WARN_ON(*(char *)ip != 0x75))
 			goto fail;
-		if (!text_poke_copy(ip, x86_nops[2], 2))
+		ret = copy_from_kernel_nofault(ip, x86_nops[2], 2);
+		if (ret < 0)
 			goto fail;
 	}
 
@@ -381,7 +379,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	 */
 
 	ptr = (unsigned long *)(trampoline + size + RET_SIZE);
-	text_poke_copy(ptr, &ops, sizeof(unsigned long));
+	*ptr = (unsigned long)ops;
 
 	op_offset -= start_offset;
 	memcpy(&op_ptr, trampoline + op_offset, OP_REF_SIZE);
@@ -397,7 +395,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	op_ptr.offset = offset;
 
 	/* put in the new offset to the ftrace_ops */
-	text_poke_copy(trampoline + op_offset, &op_ptr, OP_REF_SIZE);
+	memcpy(trampoline + op_offset, &op_ptr, OP_REF_SIZE);
 
 	/* put in the call to the function */
 	mutex_lock(&text_mutex);
@@ -407,9 +405,9 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	 * the depth accounting before the call already.
 	 */
 	dest = ftrace_ops_get_func(ops);
-	text_poke_copy_locked(trampoline + call_offset,
-	      text_gen_insn(CALL_INSN_OPCODE, trampoline + call_offset, dest),
-	      CALL_INSN_SIZE, false);
+	memcpy(trampoline + call_offset,
+	       text_gen_insn(CALL_INSN_OPCODE, trampoline + call_offset, dest),
+	       CALL_INSN_SIZE);
 	mutex_unlock(&text_mutex);
 
 	/* ALLOC_TRAMP flags lets us know we created it */
arch/x86/kernel/module.c +14 -31

102 lines not on this page · raw message

Signature · 1 line
2.45.2
#17 lore raw

[tip: x86/mm] Revert "x86/module: prepare module loading for ROX allocations of text"

The following commit has been merged into the x86/mm branch of tip:

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     1d7e707af446134dd272ea8a89018c63cc17bb6a
Gitweb:        https://git.kernel.org/tip/1d7e707af446134dd272ea8a89018c63cc17bb6a
Author:        Mike Rapoport (Microsoft) <[email protected]>
AuthorDate:    Sun, 26 Jan 2025 09:47:31 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 03 Feb 2025 11:46:02 +01:00

Revert "x86/module: prepare module loading for ROX allocations of text"

The module code does not create a writable copy of the executable memory anymore so there is no need to handle it in module relocation and alternatives patching.

This reverts commit 9bfc4824fd4836c16bb44f922bfaffba5da3e4f3.

---
 arch/um/kernel/um_arch.c           |  11 +--
 arch/x86/entry/vdso/vma.c          |   3 +-
 arch/x86/include/asm/alternative.h |  14 +-
 arch/x86/kernel/alternative.c      | 181 +++++++++++-----------------
 arch/x86/kernel/ftrace.c           |  30 ++---
 arch/x86/kernel/module.c           |  45 ++-----
 6 files changed, 117 insertions(+), 167 deletions(-)
arch/um/kernel/um_arch.c +5 -6
index 79ea97d..8be9197 100644
@@ -440,25 +440,24 @@ void __init arch_cpu_finalize_init(void)
 	os_check_bugs();
 }
 
-void apply_seal_endbr(s32 *start, s32 *end, struct module *mod)
+void apply_seal_endbr(s32 *start, s32 *end)
 {
 }
 
-void apply_retpolines(s32 *start, s32 *end, struct module *mod)
+void apply_retpolines(s32 *start, s32 *end)
 {
 }
 
-void apply_returns(s32 *start, s32 *end, struct module *mod)
+void apply_returns(s32 *start, s32 *end)
 {
 }
 
 void apply_fineibt(s32 *start_retpoline, s32 *end_retpoline,
-		   s32 *start_cfi, s32 *end_cfi, struct module *mod)
+		   s32 *start_cfi, s32 *end_cfi)
 {
 }
 
-void apply_alternatives(struct alt_instr *start, struct alt_instr *end,
-			struct module *mod)
+void apply_alternatives(struct alt_instr *start, struct alt_instr *end)
 {
 }
 
arch/x86/entry/vdso/vma.c +1 -2
index 39e6efc..bfc7cab 100644
@@ -48,8 +48,7 @@ int __init init_vdso_image(const struct vdso_image *image)
 
 	apply_alternatives((struct alt_instr *)(image->data + image->alt),
 			   (struct alt_instr *)(image->data + image->alt +
-						image->alt_len),
-			   NULL);
+						image->alt_len));
 
 	return 0;
 }
arch/x86/include/asm/alternative.h +7 -7
index e3903b7..a214166 100644
@@ -87,16 +87,16 @@ extern struct alt_instr __alt_instructions[], __alt_instructions_end[];
  * instructions were patched in already:
  */
 extern int alternatives_patched;
-struct module;
 
 extern void alternative_instructions(void);
-extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end,
-			       struct module *mod);
-extern void apply_retpolines(s32 *start, s32 *end, struct module *mod);
-extern void apply_returns(s32 *start, s32 *end, struct module *mod);
-extern void apply_seal_endbr(s32 *start, s32 *end, struct module *mod);
+extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end);
+extern void apply_retpolines(s32 *start, s32 *end);
+extern void apply_returns(s32 *start, s32 *end);
+extern void apply_seal_endbr(s32 *start, s32 *end);
 extern void apply_fineibt(s32 *start_retpoline, s32 *end_retpoine,
-			  s32 *start_cfi, s32 *end_cfi, struct module *mod);
+			  s32 *start_cfi, s32 *end_cfi);
+
+struct module;
 
 struct callthunk_sites {
 	s32				*call_start, *call_end;
arch/x86/kernel/alternative.c +76 -105

535 lines not on this page · raw message

arch/x86/kernel/ftrace.c +14 -16
index 166bc0e..cace6e8 100644
@@ -118,13 +118,10 @@ ftrace_modify_code_direct(unsigned long ip, const char *old_code,
 		return ret;
 
 	/* replace the text with the new text */
-	if (ftrace_poke_late) {
+	if (ftrace_poke_late)
 		text_poke_queue((void *)ip, new_code, MCOUNT_INSN_SIZE, NULL);
-	} else {
-		mutex_lock(&text_mutex);
-		text_poke((void *)ip, new_code, MCOUNT_INSN_SIZE);
-		mutex_unlock(&text_mutex);
-	}
+	else
+		text_poke_early((void *)ip, new_code, MCOUNT_INSN_SIZE);
 	return 0;
 }
 
@@ -321,7 +318,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	unsigned const char op_ref[] = { 0x48, 0x8b, 0x15 };
 	unsigned const char retq[] = { RET_INSN_OPCODE, INT3_INSN_OPCODE };
 	union ftrace_op_code_union op_ptr;
-	void *ret;
+	int ret;
 
 	if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) {
 		start_offset = (unsigned long)ftrace_regs_caller;
@@ -352,15 +349,15 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	npages = DIV_ROUND_UP(*tramp_size, PAGE_SIZE);
 
 	/* Copy ftrace_caller onto the trampoline memory */
-	ret = text_poke_copy(trampoline, (void *)start_offset, size);
-	if (WARN_ON(!ret))
+	ret = copy_from_kernel_nofault(trampoline, (void *)start_offset, size);
+	if (WARN_ON(ret < 0))
 		goto fail;
 
 	ip = trampoline + size;
 	if (cpu_feature_enabled(X86_FEATURE_RETHUNK))
 		__text_gen_insn(ip, JMP32_INSN_OPCODE, ip, x86_return_thunk, JMP32_INSN_SIZE);
 	else
-		text_poke_copy(ip, retq, sizeof(retq));
+		memcpy(ip, retq, sizeof(retq));
 
 	/* No need to test direct calls on created trampolines */
 	if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) {
@@ -368,7 +365,8 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 		ip = trampoline + (jmp_offset - start_offset);
 		if (WARN_ON(*(char *)ip != 0x75))
 			goto fail;
-		if (!text_poke_copy(ip, x86_nops[2], 2))
+		ret = copy_from_kernel_nofault(ip, x86_nops[2], 2);
+		if (ret < 0)
 			goto fail;
 	}
 
@@ -381,7 +379,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	 */
 
 	ptr = (unsigned long *)(trampoline + size + RET_SIZE);
-	text_poke_copy(ptr, &ops, sizeof(unsigned long));
+	*ptr = (unsigned long)ops;
 
 	op_offset -= start_offset;
 	memcpy(&op_ptr, trampoline + op_offset, OP_REF_SIZE);
@@ -397,7 +395,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	op_ptr.offset = offset;
 
 	/* put in the new offset to the ftrace_ops */
-	text_poke_copy(trampoline + op_offset, &op_ptr, OP_REF_SIZE);
+	memcpy(trampoline + op_offset, &op_ptr, OP_REF_SIZE);
 
 	/* put in the call to the function */
 	mutex_lock(&text_mutex);
@@ -407,9 +405,9 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
 	 * the depth accounting before the call already.
 	 */
 	dest = ftrace_ops_get_func(ops);
-	text_poke_copy_locked(trampoline + call_offset,
-	      text_gen_insn(CALL_INSN_OPCODE, trampoline + call_offset, dest),
-	      CALL_INSN_SIZE, false);
+	memcpy(trampoline + call_offset,
+	       text_gen_insn(CALL_INSN_OPCODE, trampoline + call_offset, dest),
+	       CALL_INSN_SIZE);
 	mutex_unlock(&text_mutex);
 
 	/* ALLOC_TRAMP flags lets us know we created it */
arch/x86/kernel/module.c +14 -31

102 lines not on this page · raw message

author maintainer
#18 lore raw

PATCH v3 8/9 module: drop unused module_writable_address()

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

module_writable_address() is unused and can be removed.

---
 include/linux/module.h | 10 ----------
 1 file changed, 10 deletions(-)
include/linux/module.h +0 -10
index 6a24e9395cb2..d2cf30be10cc 100644
@@ -768,11 +768,6 @@ static inline bool is_livepatch_module(struct module *mod)
 
 void set_module_sig_enforced(void);
 
-static inline void *module_writable_address(struct module *mod, void *loc)
-{
-	return loc;
-}
-
 #else /* !CONFIG_MODULES... */
 
 static inline struct module *__module_address(unsigned long addr)
@@ -880,11 +875,6 @@ static inline bool module_is_coming(struct module *mod)
 {
 	return false;
 }
-
-static inline void *module_writable_address(struct module *mod, void *loc)
-{
-	return loc;
-}
 #endif /* CONFIG_MODULES */
 
 #ifdef CONFIG_SYSFS
Signature · 1 line
2.45.2
#19 lore raw

[tip: x86/mm] module: drop unused module_writable_address()

The following commit has been merged into the x86/mm branch of tip:

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     602df3712979de594d268e8cbca46a93126c977d
Gitweb:        https://git.kernel.org/tip/602df3712979de594d268e8cbca46a93126c977d
Author:        Mike Rapoport (Microsoft) <[email protected]>
AuthorDate:    Sun, 26 Jan 2025 09:47:32 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 03 Feb 2025 11:46:03 +01:00

module: drop unused module_writable_address()

module_writable_address() is unused and can be removed.

---
 include/linux/module.h | 10 ----------
 1 file changed, 10 deletions(-)
include/linux/module.h +0 -10
index ddf27ed..a76928c 100644
@@ -768,11 +768,6 @@ static inline bool is_livepatch_module(struct module *mod)
 
 void set_module_sig_enforced(void);
 
-static inline void *module_writable_address(struct module *mod, void *loc)
-{
-	return loc;
-}
-
 #else /* !CONFIG_MODULES... */
 
 static inline struct module *__module_address(unsigned long addr)
@@ -880,11 +875,6 @@ static inline bool module_is_coming(struct module *mod)
 {
 	return false;
 }
-
-static inline void *module_writable_address(struct module *mod, void *loc)
-{
-	return loc;
-}
 #endif /* CONFIG_MODULES */
 
 #ifdef CONFIG_SYSFS
author maintainer
#20 lore raw

PATCH v3 9/9 x86: re-enable EXECMEM_ROX support

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

after rework of execmem ROX caches

---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
index ef6cfea9df73..9d7bd0ae48c4 100644
@@ -83,6 +83,7 @@ config X86
 	select ARCH_HAS_DMA_OPS			if GART_IOMMU || XEN
 	select ARCH_HAS_EARLY_DEBUG		if KGDB
 	select ARCH_HAS_ELF_RANDOMIZE
+	select ARCH_HAS_EXECMEM_ROX		if X86_64
 	select ARCH_HAS_FAST_MULTIPLIER
 	select ARCH_HAS_FORTIFY_SOURCE
 	select ARCH_HAS_GCOV_PROFILE_ALL
Signature · 1 line
2.45.2
#21 lore raw

[tip: x86/mm] x86: re-enable EXECMEM_ROX support

The following commit has been merged into the x86/mm branch of tip:

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     64f6a4e10c05ed527f0f24b7954964255e0d3535
Gitweb:        https://git.kernel.org/tip/64f6a4e10c05ed527f0f24b7954964255e0d3535
Author:        Mike Rapoport (Microsoft) <[email protected]>
AuthorDate:    Sun, 26 Jan 2025 09:47:33 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 03 Feb 2025 11:46:03 +01:00

x86: re-enable EXECMEM_ROX support

after rework of execmem ROX caches

---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
index 87198d9..6df7779 100644
@@ -85,6 +85,7 @@ config X86
 	select ARCH_HAS_DMA_OPS			if GART_IOMMU || XEN
 	select ARCH_HAS_EARLY_DEBUG		if KGDB
 	select ARCH_HAS_ELF_RANDOMIZE
+	select ARCH_HAS_EXECMEM_ROX		if X86_64
 	select ARCH_HAS_FAST_MULTIPLIER
 	select ARCH_HAS_FORTIFY_SOURCE
 	select ARCH_HAS_GCOV_PROFILE_ALL
#22 lore raw

Re: [PATCH v3 9/9] x86: re-enable EXECMEM_ROX support

In reply to Mike Rapoport

Hi,

A kernel BUG in __text_poke() is observed inside a Ubuntu 26.04 guest (v7.0.0.28 and also with upstream v7.2-4c5). Host is running on v7.2-rc5.

SETUP
-----

512 QEMU Q35 guests (2 vCPUs, 1G RAM each) boot concurrently across 200 iterations. The crash is seen during module loading in early boot. 2-7 guests out of ~100,000 boot attempts crash with the kernel BUG.

Reproduced on:
- AMD EPYC, QEMU Q35 guests
- Intel Icelake, QEMU Q35 guests

Apparently it was seen once in the non-guest Ubuntu 26.04 environment as well
https://discourse.ubuntu.com/t/failed-boot-one-time-on-ubuntu-unity-26-04-educational-mission/85300

SYMPTOMS
--------

Either of the below two crash signatures is observed:

Signature 1 — BUG_ON(!pages[0]) in __text_poke():

kernel BUG at arch/x86/kernel/alternative.c:2564!

  Oops: invalid opcode: 0000 [#1] SMP NOPTI
  CPU: 0 UID: 0 PID: 741 Comm: (udev-worker) Not tainted 7.0.0-28-generic #28-Ubuntu PREEMPT(lazy)
  Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)
  RIP: 0010:__text_poke+0x2a4/0x400
  Call Trace:
   <TASK>
   smp_text_poke_batch_finish+0x25f/0x5b0
   smp_text_poke_single+0x13/0x30
   __static_call_transform+0x10e/0x210
   arch_static_call_transform+0x5c/0xc0
   __static_call_init+0x154/0x300
   static_call_module_notify+0x144/0x170
   notifier_call_chain+0x62/0xf0
   blocking_notifier_call_chain_robust+0x62/0xc0
   load_module+0x5ef/0x960
  Modules linked in: ppdev(+) kvm(+) irqbypass ...

Signature 2 — BUG_ON(memcmp()) in __text_poke():

kernel BUG at arch/x86/kernel/alternative.c:2632!

  Oops: invalid opcode: 0000 [#1] SMP NOPTI
  CPU: 0 UID: 0 PID: 742 Comm: (udev-worker) Not tainted 7.0.0-28-generic #28-Ubuntu PREEMPT(lazy)
  Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)
  RIP: 0010:__text_poke+0x380/0x400
  Call Trace:
   <TASK>
   smp_text_poke_batch_finish+0x323/0x5b0
   smp_text_poke_single+0x13/0x30
   __static_call_transform+0x10e/0x210
   arch_static_call_transform+0x5c/0xc0
   __static_call_init+0x154/0x300
   static_call_module_notify+0x144/0x170
   notifier_call_chain+0x62/0xf0
   blocking_notifier_call_chain_robust+0x62/0xc0
   load_module+0x5ef/0x960
  Modules linked in: kvm(+) irqbypass ...

INSTRUMENTATION
---------------

The following instrumentation was applied to capture the page table state at the moment of failure:

In mm/vmalloc.c::vmalloc_to_page()

    ptep = pte_offset_kernel(pmd, addr);
    pte = ptep_get(ptep);
    if (pte_present(pte))
        page = pte_page(pte);
    else
        pr_err("vmalloc_to_page: pte_not_present for addr=%lx "
               "PGD=%lx P4D=%lx PUD=%lx PMD=%lx PTE=%lx\n",
               addr, pgd_val(*pgd), p4d_val(*p4d), pud_val(*pud),
               pmd_val(*pmd), pte_val(pte));

In arch/x86/kernel/alternative.c::__text_poke()

    /* BUG 1: pages[0] NULL */
    if (!pages[0] || ...) {
        pr_err("%s: vmalloc_to_page failed for addr=%pK pages[0]=%pK pages[1]=%pK cross=%d\n", ...);
        BUG();
    }

    ...

    /* BUG 2: memcmp mismatch */
    if (memcmp(addr, src, len)) {
        struct page *page_now = vmalloc_to_page(addr);
        pr_err("%s: memcmp failed at addr=%pK len=%zu\n", __func__, addr, len);
        pr_err("%s: expected (src): %*ph\n", __func__, (int)len, src);
        pr_err("%s: found    (dst): %*ph\n", __func__, (int)len, (u8 *)addr);
        pr_err("%s: pfn at poke time (vmalloc_to_page): %lx\n", __func__, page_to_pfn(pages[0]));
        pr_err("%s: pfn at readback time (vmalloc_to_page): %lx\n", __func__, page_to_pfn(page_now));
        text_poke_dump_pagetable((unsigned long)addr); /* reads CR3 */
        BUG();
    }

Signature 1 — vmalloc_to_page() returned NULL:

vmalloc_to_page: pte_not_present for addr=ffffffffc0697979 PGD=28242067 P4D=28243067 PUD=28245067 PMD=3be001a1 PTE=0 SMP alternatives: __text_poke: vmalloc_to_page failed for addr=ffffffffc0697979 pages[0]=0000000000000000 pages[1]=0000000000000000 cross=0

PMD was not a leaf, and the code reached a point to check pte_present() and finds the PTE to be not present. But in the dump in else case, pmd_val(*pmd) shows that it is a PMD leaf (PSE bit set).

Signature 2 — vmalloc_to_page() returned a wrong page:

SMP alternatives: __text_poke: memcmp failed at addr=ffffffffc06fc4bd len=1 SMP alternatives: __text_poke: expected (src): cc
SMP alternatives: __text_poke: found (dst): e8
SMP alternatives: __text_poke: pfn at poke time (vmalloc_to_page): 3884a SMP alternatives: __text_poke: pfn at readback time (vmalloc_to_page): 52fc SMP alternatives: __text_poke: page table dump for addr:

    PGD 1ba42067  P4D 1ba43067  PUD 1ba45067  PMD 52001a1

vmalloc_to_page() returned pfn=0x3884a when pages[0] was queried. The write of 0xcc landed there. When vmalloc_to_page() was called again during the memcmp readback in the instrumentation code, the vmap mapping resolved through PMD=52001a1 to a different pfn=0x52fc, which contains 0xe8 (existing instruction bytes). This is silent memory corruption — 0xcc was written to the wrong physical page (pfn=0x3884a).

BISECTION
---------

The following experiments all ran 200 iterations of 512 concurrent guest boots with zero crashes.

1. Remove "select ARCH_HAS_EXECMEM_ROX" from arch/x86/Kconfig

2. Boot with "nohugevmalloc"

3. Remove VM_ALLOW_HUGE_VMAP from vm_flags in execmem_cache_populate()

- unsigned long vm_flags = VM_ALLOW_HUGE_VMAP;
+ unsigned long vm_flags = 0;

4. Remove CPA_COLLAPSE from set_memory_rox()

return change_page_attr_set_clr(&addr, numpages, __pgprot(0), clr, 0,
- CPA_COLLAPSE, NULL);
+ 0, NULL);

Without CPA_COLLAPSE, set_memory_rox() changes 4K PTE permissions in-place and the PMD stays permanently as a PTE-page pointer after the first split — no re-collapse, no future race window. This confirms that CPA_COLLAPSE is a necessary part of the race sequence.

5. Hold pgd_lock around vmalloc_to_page() in __text_poke()

if (!core_kernel_text((unsigned long)addr)) {
+ spin_lock(&pgd_lock);
pages[0] = vmalloc_to_page(addr);
if (cross_page_boundary)
pages[1] = vmalloc_to_page(addr + PAGE_SIZE);
+ spin_unlock(&pgd_lock);
} else {

__split_large_page() holds pgd_lock while installing the new PMD and populating all 512 4K PTEs. Holding pgd_lock around vmalloc_to_page() prevents CPA from modifying the PMD concurrently with the page table walk. This confirms the race is precisely between vmalloc_to_page() reading the PMD/PTE and __split_large_page() modifying it under pgd_lock.

Note: this is a diagnostic experiment, not a production fix.

The experiments point to a race between vmalloc_to_page() in __text_poke() and __split_large_page() modifying the shared PMD under pgd_lock. __split_large_page() holds pgd_lock while modifying the PMD; vmalloc_to_page() holds no lock and can observe the PMD in an intermediate state.

What would be the correct fix for this issue?

Happy to test any proposed fix or run further experiments.

Regards,
Nikunj

#23 lore raw

Re: [PATCH v3 9/9] x86: re-enable EXECMEM_ROX support

Quoted 181 lines · Nikunj A Dadhania

Hi,

A kernel BUG in __text_poke() is observed inside a Ubuntu 26.04 guest (v7.0.0.28 and also with upstream v7.2-4c5). Host is running on v7.2-rc5.

SETUP
-----

512 QEMU Q35 guests (2 vCPUs, 1G RAM each) boot concurrently across 200 iterations. The crash is seen during module loading in early boot. 2-7 guests out of ~100,000 boot attempts crash with the kernel BUG.

Reproduced on:
- AMD EPYC, QEMU Q35 guests
- Intel Icelake, QEMU Q35 guests

Apparently it was seen once in the non-guest Ubuntu 26.04 environment as well
https://discourse.ubuntu.com/t/failed-boot-one-time-on-ubuntu-unity-26-04-educational-mission/85300

SYMPTOMS
--------

Either of the below two crash signatures is observed:

Signature 1 — BUG_ON(!pages[0]) in __text_poke():

kernel BUG at arch/x86/kernel/alternative.c:2564!

  Oops: invalid opcode: 0000 [#1] SMP NOPTI
  CPU: 0 UID: 0 PID: 741 Comm: (udev-worker) Not tainted 7.0.0-28-generic #28-Ubuntu PREEMPT(lazy)
  Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)
  RIP: 0010:__text_poke+0x2a4/0x400
  Call Trace:
   <TASK>
   smp_text_poke_batch_finish+0x25f/0x5b0
   smp_text_poke_single+0x13/0x30
   __static_call_transform+0x10e/0x210
   arch_static_call_transform+0x5c/0xc0
   __static_call_init+0x154/0x300
   static_call_module_notify+0x144/0x170
   notifier_call_chain+0x62/0xf0
   blocking_notifier_call_chain_robust+0x62/0xc0
   load_module+0x5ef/0x960
  Modules linked in: ppdev(+) kvm(+) irqbypass ...

Signature 2 — BUG_ON(memcmp()) in __text_poke():

kernel BUG at arch/x86/kernel/alternative.c:2632!

  Oops: invalid opcode: 0000 [#1] SMP NOPTI
  CPU: 0 UID: 0 PID: 742 Comm: (udev-worker) Not tainted 7.0.0-28-generic #28-Ubuntu PREEMPT(lazy)
  Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)
  RIP: 0010:__text_poke+0x380/0x400
  Call Trace:
   <TASK>
   smp_text_poke_batch_finish+0x323/0x5b0
   smp_text_poke_single+0x13/0x30
   __static_call_transform+0x10e/0x210
   arch_static_call_transform+0x5c/0xc0
   __static_call_init+0x154/0x300
   static_call_module_notify+0x144/0x170
   notifier_call_chain+0x62/0xf0
   blocking_notifier_call_chain_robust+0x62/0xc0
   load_module+0x5ef/0x960
  Modules linked in: kvm(+) irqbypass ...

INSTRUMENTATION
---------------

The following instrumentation was applied to capture the page table state at the moment of failure:

In mm/vmalloc.c::vmalloc_to_page()

    ptep = pte_offset_kernel(pmd, addr);
    pte = ptep_get(ptep);
    if (pte_present(pte))
        page = pte_page(pte);
    else
        pr_err("vmalloc_to_page: pte_not_present for addr=%lx "
               "PGD=%lx P4D=%lx PUD=%lx PMD=%lx PTE=%lx\n",
               addr, pgd_val(*pgd), p4d_val(*p4d), pud_val(*pud),
               pmd_val(*pmd), pte_val(pte));

In arch/x86/kernel/alternative.c::__text_poke()

    /* BUG 1: pages[0] NULL */
    if (!pages[0] || ...) {
        pr_err("%s: vmalloc_to_page failed for addr=%pK pages[0]=%pK pages[1]=%pK cross=%d\n", ...);
        BUG();
    }

    ...

    /* BUG 2: memcmp mismatch */
    if (memcmp(addr, src, len)) {
        struct page *page_now = vmalloc_to_page(addr);
        pr_err("%s: memcmp failed at addr=%pK len=%zu\n", __func__, addr, len);
        pr_err("%s: expected (src): %*ph\n", __func__, (int)len, src);
        pr_err("%s: found    (dst): %*ph\n", __func__, (int)len, (u8 *)addr);
        pr_err("%s: pfn at poke time (vmalloc_to_page): %lx\n", __func__, page_to_pfn(pages[0]));
        pr_err("%s: pfn at readback time (vmalloc_to_page): %lx\n", __func__, page_to_pfn(page_now));
        text_poke_dump_pagetable((unsigned long)addr); /* reads CR3 */
        BUG();
    }

Signature 1 — vmalloc_to_page() returned NULL:

vmalloc_to_page: pte_not_present for addr=ffffffffc0697979 PGD=28242067 P4D=28243067 PUD=28245067 PMD=3be001a1 PTE=0 SMP alternatives: __text_poke: vmalloc_to_page failed for addr=ffffffffc0697979 pages[0]=0000000000000000 pages[1]=0000000000000000 cross=0

PMD was not a leaf, and the code reached a point to check pte_present() and finds the PTE to be not present. But in the dump in else case, pmd_val(*pmd) shows that it is a PMD leaf (PSE bit set).

Signature 2 — vmalloc_to_page() returned a wrong page:

SMP alternatives: __text_poke: memcmp failed at addr=ffffffffc06fc4bd len=1 SMP alternatives: __text_poke: expected (src): cc
SMP alternatives: __text_poke: found (dst): e8
SMP alternatives: __text_poke: pfn at poke time (vmalloc_to_page): 3884a SMP alternatives: __text_poke: pfn at readback time (vmalloc_to_page): 52fc SMP alternatives: __text_poke: page table dump for addr:

    PGD 1ba42067  P4D 1ba43067  PUD 1ba45067  PMD 52001a1

vmalloc_to_page() returned pfn=0x3884a when pages[0] was queried. The write of 0xcc landed there. When vmalloc_to_page() was called again during the memcmp readback in the instrumentation code, the vmap mapping resolved through PMD=52001a1 to a different pfn=0x52fc, which contains 0xe8 (existing instruction bytes). This is silent memory corruption — 0xcc was written to the wrong physical page (pfn=0x3884a).

BISECTION
---------

The following experiments all ran 200 iterations of 512 concurrent guest boots with zero crashes.

1. Remove "select ARCH_HAS_EXECMEM_ROX" from arch/x86/Kconfig

2. Boot with "nohugevmalloc"

3. Remove VM_ALLOW_HUGE_VMAP from vm_flags in execmem_cache_populate()

- unsigned long vm_flags = VM_ALLOW_HUGE_VMAP;
+ unsigned long vm_flags = 0;

4. Remove CPA_COLLAPSE from set_memory_rox()

return change_page_attr_set_clr(&addr, numpages, __pgprot(0), clr, 0,
- CPA_COLLAPSE, NULL);
+ 0, NULL);

Without CPA_COLLAPSE, set_memory_rox() changes 4K PTE permissions in-place and the PMD stays permanently as a PTE-page pointer after the first split — no re-collapse, no future race window. This confirms that CPA_COLLAPSE is a necessary part of the race sequence.

5. Hold pgd_lock around vmalloc_to_page() in __text_poke()

if (!core_kernel_text((unsigned long)addr)) {
+ spin_lock(&pgd_lock);
pages[0] = vmalloc_to_page(addr);
if (cross_page_boundary)
pages[1] = vmalloc_to_page(addr + PAGE_SIZE);
+ spin_unlock(&pgd_lock);
} else {

__split_large_page() holds pgd_lock while installing the new PMD and populating all 512 4K PTEs. Holding pgd_lock around vmalloc_to_page() prevents CPA from modifying the PMD concurrently with the page table walk. This confirms the race is precisely between vmalloc_to_page() reading the PMD/PTE and __split_large_page() modifying it under pgd_lock.

Note: this is a diagnostic experiment, not a production fix.

The experiments point to a race between vmalloc_to_page() in __text_poke() and __split_large_page() modifying the shared PMD under pgd_lock. __split_large_page() holds pgd_lock while modifying the PMD; vmalloc_to_page() holds no lock and can observe the PMD in an intermediate state.

Quoted 3 lines

What would be the correct fix for this issue?

Happy to test any proposed fix or run further experiments.

https://lore.kernel.org/linux-mm/anCK3eWFMwZqq5ka@pedro-suse/

Signature · 1 line
Pedro
maintainer
#25 lore raw
In reply to Mike Rapoport
Quoted 3 lines · Mike Rapoport

v3 changes:
* instead of adding a new module state handle ROX restoration locally in load_module() as Petr suggested

OK, updated the patches I have pending for post -rc1.

Tested-by on the v3 series
#26 lore raw
In reply to Mike Rapoport

Hi Mike,

I bisected issues with the kernel booting correctly to an earlier revision of this series [2]. I have now tested this series in the same environment and found that it functions correctly with no issues, so feel free to add:

Cheers, Lorenzo

[2]:https://lore.kernel.org/linux-mm/[email protected]/

Quoted 71 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Hi,

Following Peter's comments [1] these patches rework handling of ROX caches for module text allocations.

Instead of using a writable copy that really complicates alternatives patching, temporarily remap parts of a large ROX page as RW for the time of module formation and then restore it's ROX protections when the module is ready.

To keep the ROX memory mapped with large pages, make set_memory_rox() capable of restoring large pages (more details are in patch 3).

Since this is really about x86, I believe this should go in via tip tree.

The patches also available in git
https://git.kernel.org/rppt/h/execmem/x86-rox/v10

v3 changes:
* instead of adding a new module state handle ROX restoration locally in load_module() as Petr suggested

v2: https://lore.kernel.org/all/[email protected]
* only collapse large mappings in set_memory_rox()
* simplify RW <-> ROX remapping
* don't remove ROX cache pages from the direct map (patch 4)

v1: https://lore.kernel.org/all/[email protected]

[1] https://lore.kernel.org/all/[email protected]

Kirill A. Shutemov (1):
x86/mm/pat: restore large ROX pages after fragmentation

Mike Rapoport (Microsoft) (8):
x86/mm/pat: cpa-test: fix length for CPA_ARRAY test x86/mm/pat: drop duplicate variable in cpa_flush()
execmem: don't remove ROX cache from the direct map
execmem: add API for temporal remapping as RW and restoring ROX afterwards
module: switch to execmem API for remapping as RW and restoring ROX Revert "x86/module: prepare module loading for ROX allocations of text"
module: drop unused module_writable_address()
x86: re-enable EXECMEM_ROX support

 arch/um/kernel/um_arch.c             |  11 +-
 arch/x86/Kconfig                     |   1 +
 arch/x86/entry/vdso/vma.c            |   3 +-
 arch/x86/include/asm/alternative.h   |  14 +-
 arch/x86/include/asm/pgtable_types.h |   2 +
 arch/x86/kernel/alternative.c        | 181 +++++++++-------------
 arch/x86/kernel/ftrace.c             |  30 ++--
 arch/x86/kernel/module.c             |  45 ++----
 arch/x86/mm/pat/cpa-test.c           |   2 +-
 arch/x86/mm/pat/set_memory.c         | 220 ++++++++++++++++++++++++++-
 include/linux/execmem.h              |  31 ++++
 include/linux/module.h               |  16 --
 include/linux/moduleloader.h         |   4 -
 include/linux/vm_event_item.h        |   2 +
 kernel/module/main.c                 |  78 +++-------
 kernel/module/strict_rwx.c           |   9 +-
 mm/execmem.c                         |  39 +++--
 mm/vmstat.c                          |   2 +
 18 files changed, 422 insertions(+), 268 deletions(-)

base-commit: ffd294d346d185b70e28b1a28abe367bbfe53c04 --
2.45.2