Skip to contentSkip to conversation

arm64: mm: Introducing ROX CACHE to ARM64 systems with bbml2 no abort

RFC PATCH In review Adrian Barnaś 4 participants 27 messages Jun 11 – Jul 6

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover arm64: mm: Introducing ROX CACHE to ARM64 systems with bbml2 no abort

Hi All,

I would like to propose the introduction of the EXECMEM_ROX_CACHE feature for ARM64.

When the rodata=on kernel parameter is set, all executable and read-only aliases in the linear map are forced to be read-only. Originally, this forced PTE-level mappings across the entire linear map, causing a significant performance regression (greater than 10%) for some workloads due to increased TLB miss rates and deeper page table walk.

The feat_bbml2_no_abort (FEAT_BBML3 in ARMv9.7) feature can be utilized to mitigate this regression. Because we can split memory mappings without triggering TLB conflict aborts, kernel memory permission adjustments become possible after early boot without forcing PTE-level mappings across the entire linear map.

However, when applying read-only permissions to kernel module section the linear map can still became fragmented due to the scattered physical layout of the underlying pages. To address this, EXECMEM_ROX_CACHE, which was initially enabled on the x86 architecture [1] for this purpose, can be used on ARM64 as well.

EXECMEM_ROX_CACHE works by preallocating PMD-sized contiguous blocks to act as a cache for .module.text memory. These blocks are initially poisoned and made read-only-execute (which simultaneously makes the linear alias of this region read-only). When loading a .module.text section into memory, the requested cache region is made RW, the bytes are copied, and ROX permissions are restored.

To take full advantage of this approach, after restoring RO permissions on the PMD-sized linear alias block, the PTE mappings are coalesced back into a single PMD entry.

Testing on an Android device running 6.18 based kernel with rodata=on shows an average 20% reduction of level 3 page table entries for the linear mapping.

This implementation currently works around some limitations of the set_memory_xx API, which might be relevant when considering the refactoring proposed here [2]:

* Because the execmem_cache operates outside the linear map, its permissions could theoretically remain untouched (poisoned and RO) until the cache block is fully emptied and freed. However, we currently lack an API to interact exclusively with the vmalloc area (e.g., setting it to RW) without simultaneously setting the linear alias to RW.

* Additionally, set_direct_map_valid() has somewhat confusing semantics. It is used while "cleaning" a cache block (after confirming it is entirely empty). The linear map region should be returned to its default state to restore writability, but set_valid might just set the "valid" attribute (as is the case for ARM64, which I have temporarily addressed here with a workaround).

I would be glad to hear your feedback on these changes.

Best regards,
Adrian

[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://lore.kernel.org/all/[email protected]/

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc "Mike Rapoport (Microsoft)" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Yang Shi <[email protected]>
Cc Brendan Jackman <[email protected]>

--->8

Adrian Barnaś (6):
arm64: mm: explicitly declare module and ftrace execmem regions
arm64: mm: allow huge vmap permission adjustments with bbml2_no_abort
arm64: mm: fix restoring linear map permissions on execmem cache clean
arm64: mm: add helper to fill execmem with trapping instructions
arm64: execmem: enable EXECMEM_ROX_CACHE on supported CPUs
arm64: mm: support PMD page coalescing in the linear map

 arch/arm64/Kconfig           |  1 +
 arch/arm64/include/asm/mmu.h |  1 +
 arch/arm64/mm/init.c         | 54 +++++++++++++++++++-
 arch/arm64/mm/mmu.c          | 95 ++++++++++++++++++++++++++++++++++++
 arch/arm64/mm/pageattr.c     | 46 +++++++++++++----
 5 files changed, 186 insertions(+), 11 deletions(-)

--
2.54.0.1136.gdb2ca164c4-goog

#1 lore raw

RFC PATCH 1/6 arm64: mm: explicitly declare module and ftrace execmem regions

Replace the reliance on the EXECMEM_DEFAULT fallback by explicitly defining the execution memory (execmem) regions for MODULE_TEXT, MODULE_DATA, and FTRACE in execmem_arch_setup().

---
 arch/arm64/mm/init.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
arch/arm64/mm/init.c +17 -1
index 96711b8578fd..c673a9a839dd 100644
@@ -519,7 +519,7 @@ struct execmem_info __init *execmem_arch_setup(void)
 
 	execmem_info = (struct execmem_info){
 		.ranges = {
-			[EXECMEM_DEFAULT] = {
+			[EXECMEM_MODULE_TEXT] = {
 				.start	= start,
 				.end	= end,
 				.pgprot	= PAGE_KERNEL,
@@ -533,12 +533,28 @@ struct execmem_info __init *execmem_arch_setup(void)
 				.pgprot	= PAGE_KERNEL_ROX,
 				.alignment = 1,
 			},
+			[EXECMEM_FTRACE] = {
+				.start	= VMALLOC_START,
+				.end	= VMALLOC_END,
+				.pgprot	= PAGE_KERNEL,
+				.alignment = 1,
+				.fallback_start	= fallback_start,
+				.fallback_end	= fallback_end,
+			},
 			[EXECMEM_BPF] = {
 				.start	= VMALLOC_START,
 				.end	= VMALLOC_END,
 				.pgprot	= PAGE_KERNEL,
 				.alignment = 1,
 			},
+			[EXECMEM_MODULE_DATA] = {
+				.start  = start,
+				.end    = end,
+				.pgprot = PAGE_KERNEL,
+				.alignment = 1,
+				.fallback_start	= fallback_start,
+				.fallback_end	= fallback_end,
+			},
 		},
 	};
 
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
#2 lore raw

Re: [RFC PATCH 1/6] arm64: mm: explicitly declare module and ftrace execmem regions

Quoted 3 lines · On Thu Jun 11, 2026 at 1:01 PM UTC, =?UTF-8?q?Adrian=20Barna=C5=9B?= wrote:

Replace the reliance on the EXECMEM_DEFAULT fallback by explicitly defining the execution memory (execmem) regions for MODULE_TEXT, MODULE_DATA, and FTRACE in execmem_arch_setup().

Please can you explain in the commit message _why_ you do this. This way reviewers don't have to make a mental note and come back later after reading the rest of the patchset. And once it's commited, it will save readers from having to chase down the contextual commits to figure out what's going on.

#3 lore raw

RFC PATCH 2/6 arm64: mm: allow huge vmap permission adjustments with bbml2_no_abort

In reply to Adrian Barnaś

Remove the protection against huge vmap permission adjustments on systems that support the bbml2_no_abort CPU feature.

Splitting live kernel VA section mappings into page mappings was restricted because it could cause TLB Conflict Aborts. This forced permission adjustments on memory allocated with VM_ALLOW_HUGE_VMAP to be rejected, resulting in performance drops (e.g., when enforcing rodata=on disables huge mappings).

The bbml2_no_abort feature (which mirrors the architectural guarantees of FEAT_BBML3) ensures that changing between table and block sizes without following a break-before-make sequence will not generate a TLB Conflict Abort. This hardware guarantee makes it safe to allow dynamic permission adjustments on huge vmap regions.

---
 arch/arm64/mm/pageattr.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
arch/arm64/mm/pageattr.c +14 -8
index 358d1dc9a576..88720bbba892 100644
@@ -157,23 +157,29 @@ static int change_memory_common(unsigned long addr, int numpages,
 	}
 
 	/*
-	 * Kernel VA mappings are always live, and splitting live section
-	 * mappings into page mappings may cause TLB conflicts. This means
-	 * we have to ensure that changing the permission bits of the range
-	 * we are operating on does not result in such splitting.
-	 *
 	 * Let's restrict ourselves to mappings created by vmalloc (or vmap).
-	 * Disallow VM_ALLOW_HUGE_VMAP mappings to guarantee that only page
-	 * mappings are updated and splitting is never needed.
 	 *
 	 * So check whether the [addr, addr + size) interval is entirely
 	 * covered by precisely one VM area that has the VM_ALLOC flag set.
 	 */
 	area = find_vm_area((void *)addr);
+
 	if (!area ||
 	    ((unsigned long)kasan_reset_tag((void *)end) >
 	     (unsigned long)kasan_reset_tag(area->addr) + area->size) ||
-	    ((area->flags & (VM_ALLOC | VM_ALLOW_HUGE_VMAP)) != VM_ALLOC))
+	    !(area->flags & VM_ALLOC))
+		return -EINVAL;
+
+	/*
+	 * Kernel VA mappings are always live, and splitting live section
+	 * mappings into page mappings may cause TLB conflicts if bbml2_noabort
+	 * is not present.
+	 *
+	 * While bbml2_noabort is not present disallow VM_ALLOW_HUGE_VMAP mappings
+	 * to guarantee that only page mappings are updated and splitting is not
+	 * needed.
+	 */
+	if (!system_supports_bbml2_noabort() && (area->flags & (VM_ALLOW_HUGE_VMAP)))
 		return -EINVAL;
 
 	if (!numpages)
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Reviewed-by on v1 2/6
#4 lore raw

Re: [RFC PATCH 2/6] arm64: mm: allow huge vmap permission adjustments with bbml2_no_abort

Quoted 9 lines · Adrian Barnaś

Remove the protection against huge vmap permission adjustments on systems that support the bbml2_no_abort CPU feature.

Splitting live kernel VA section mappings into page mappings was restricted because it could cause TLB Conflict Aborts. This forced permission adjustments on memory allocated with VM_ALLOW_HUGE_VMAP to be rejected, resulting in performance drops (e.g., when enforcing rodata=on disables huge mappings).

Quoted 5 lines

The bbml2_no_abort feature (which mirrors the architectural guarantees of FEAT_BBML3) ensures that changing between table and block sizes without following a break-before-make sequence will not generate a TLB Conflict Abort. This hardware guarantee makes it safe to allow dynamic permission adjustments on huge vmap regions.

FYI Linu Cherian has a series that renames bbml2_no_abort to bbml3. I think he's planning to post at -rc1. Would be good to rebase this on top once merged.

Quoted 35 lines
---
 arch/arm64/mm/pageattr.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
arch/arm64/mm/pageattr.c +4 -8
index 358d1dc9a576..88720bbba892 100644
@@ -157,23 +157,29 @@ static int change_memory_common(unsigned long addr, int numpages,
 	}
 
 	/*
-	 * Kernel VA mappings are always live, and splitting live section
-	 * mappings into page mappings may cause TLB conflicts. This means
-	 * we have to ensure that changing the permission bits of the range
-	 * we are operating on does not result in such splitting.
-	 *
 	 * Let's restrict ourselves to mappings created by vmalloc (or vmap).
-	 * Disallow VM_ALLOW_HUGE_VMAP mappings to guarantee that only page
-	 * mappings are updated and splitting is never needed.
 	 *
 	 * So check whether the [addr, addr + size) interval is entirely
 	 * covered by precisely one VM area that has the VM_ALLOC flag set.
 	 */
 	area = find_vm_area((void *)addr);
+
 	if (!area ||
 	    ((unsigned long)kasan_reset_tag((void *)end) >
 	     (unsigned long)kasan_reset_tag(area->addr) + area->size) ||
-	    ((area->flags & (VM_ALLOC | VM_ALLOW_HUGE_VMAP)) != VM_ALLOC))
+	    !(area->flags & VM_ALLOC))
+		return -EINVAL;
+

arch/arm64/mm/pageattr.c static int change_memory_common(unsigned long addr, int numpages, · line 182

Quoted 10 lines
arch/arm64/mm/pageattr.c +10 -0
+	/*
+	 * Kernel VA mappings are always live, and splitting live section
+	 * mappings into page mappings may cause TLB conflicts if bbml2_noabort
+	 * is not present.
+	 *
+	 * While bbml2_noabort is not present disallow VM_ALLOW_HUGE_VMAP mappings
+	 * to guarantee that only page mappings are updated and splitting is not
+	 * needed.
+	 */
+	if (!system_supports_bbml2_noabort() && (area->flags & (VM_ALLOW_HUGE_VMAP)))

nit: no need for the parentheses around VM_ALLOW_HUGE_VMAP.

With that:

Quoted 3 lines
arch/arm64/mm/pageattr.c +0 -0
 		return -EINVAL;
 
 	if (!numpages)
#5 lore raw

Re: [RFC PATCH 2/6] arm64: mm: allow huge vmap permission adjustments with bbml2_no_abort

Quoted 10 lines · Ryan Roberts
Quoted 9 lines · Adrian Barnaś

Remove the protection against huge vmap permission adjustments on systems that support the bbml2_no_abort CPU feature.

Splitting live kernel VA section mappings into page mappings was restricted because it could cause TLB Conflict Aborts. This forced permission adjustments on memory allocated with VM_ALLOW_HUGE_VMAP to be rejected, resulting in performance drops (e.g., when enforcing rodata=on disables huge mappings).

Quoted 9 lines
Quoted 5 lines

The bbml2_no_abort feature (which mirrors the architectural guarantees of FEAT_BBML3) ensures that changing between table and block sizes without following a break-before-make sequence will not generate a TLB Conflict Abort. This hardware guarantee makes it safe to allow dynamic permission adjustments on huge vmap regions.

FYI Linu Cherian has a series that renames bbml2_no_abort to bbml3. I think he's planning to post at -rc1. Would be good to rebase this on top once merged.

I will keep an eye on next releases.

Quoted 48 lines
Quoted 35 lines
---
 arch/arm64/mm/pageattr.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
arch/arm64/mm/pageattr.c +4 -8
index 358d1dc9a576..88720bbba892 100644
@@ -157,23 +157,29 @@ static int change_memory_common(unsigned long addr, int numpages,
 	}
 
 	/*
-	 * Kernel VA mappings are always live, and splitting live section
-	 * mappings into page mappings may cause TLB conflicts. This means
-	 * we have to ensure that changing the permission bits of the range
-	 * we are operating on does not result in such splitting.
-	 *
 	 * Let's restrict ourselves to mappings created by vmalloc (or vmap).
-	 * Disallow VM_ALLOW_HUGE_VMAP mappings to guarantee that only page
-	 * mappings are updated and splitting is never needed.
 	 *
 	 * So check whether the [addr, addr + size) interval is entirely
 	 * covered by precisely one VM area that has the VM_ALLOC flag set.
 	 */
 	area = find_vm_area((void *)addr);
+
 	if (!area ||
 	    ((unsigned long)kasan_reset_tag((void *)end) >
 	     (unsigned long)kasan_reset_tag(area->addr) + area->size) ||
-	    ((area->flags & (VM_ALLOC | VM_ALLOW_HUGE_VMAP)) != VM_ALLOC))
+	    !(area->flags & VM_ALLOC))
+		return -EINVAL;
+

arch/arm64/mm/pageattr.c static int change_memory_common(unsigned long addr, int numpages, · line 182

Quoted 10 lines
arch/arm64/mm/pageattr.c +10 -0
+	/*
+	 * Kernel VA mappings are always live, and splitting live section
+	 * mappings into page mappings may cause TLB conflicts if bbml2_noabort
+	 * is not present.
+	 *
+	 * While bbml2_noabort is not present disallow VM_ALLOW_HUGE_VMAP mappings
+	 * to guarantee that only page mappings are updated and splitting is not
+	 * needed.
+	 */
+	if (!system_supports_bbml2_noabort() && (area->flags & (VM_ALLOW_HUGE_VMAP)))

nit: no need for the parentheses around VM_ALLOW_HUGE_VMAP.

Quoted 8 lines

With that:

Quoted 3 lines
arch/arm64/mm/pageattr.c +0 -0
 		return -EINVAL;
 
 	if (!numpages)

Thanks,
Adrian

#6 lore raw

RFC PATCH 3/6 arm64: mm: fix restoring linear map permissions on execmem cache clean

In reply to Adrian Barnaś

Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.

An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.

Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.

---
 arch/arm64/mm/pageattr.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
arch/arm64/mm/pageattr.c +16 -1
index 88720bbba892..eaefdf90b0d5 100644
@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages)
 					__pgprot(PTE_PXN));
 }
 
+static int set_memory_default(unsigned long addr, int numpages)
+{
+	return __change_memory_common(addr, PAGE_SIZE * numpages,
+				      __pgprot(PTE_VALID),
+				      __pgprot(PTE_RDONLY));
+}
+
 int set_memory_valid(unsigned long addr, int numpages, int enable)
 {
 	if (enable)
@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
 	if (!can_set_direct_map())
 		return 0;
 
-	return set_memory_valid(addr, nr, valid);
+	/*
+	 * Execmem cache uses this function to reset permissions on linear mapping
+	 * when freeing unused cache block. On x86 it makes memory RW which is
+	 * desirable. On ARM64 set_memory_valid() just change valid bit which
+	 * leave direct mapping read-only so use set_memory_default instead.
+	 */
+
+	return valid ? set_memory_default(addr, nr) :
+		       set_memory_valid(addr, nr, false);
 }
 
 #ifdef CONFIG_DEBUG_PAGEALLOC
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
#7 lore raw

Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean

Quoted 35 lines · On Thu Jun 11, 2026 at 1:01 PM UTC, =?UTF-8?q?Adrian=20Barna=C5=9B?= wrote:

Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.

An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.

Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.

---
 arch/arm64/mm/pageattr.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
arch/arm64/mm/pageattr.c +7 -0
index 88720bbba892..eaefdf90b0d5 100644
@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages)
 					__pgprot(PTE_PXN));
 }
 
+static int set_memory_default(unsigned long addr, int numpages)
+{
+	return __change_memory_common(addr, PAGE_SIZE * numpages,
+				      __pgprot(PTE_VALID),
+				      __pgprot(PTE_RDONLY));
+}
+
 int set_memory_valid(unsigned long addr, int numpages, int enable)
 {
 	if (enable)

arch/arm64/mm/pageattr.c int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) · line 375

Quoted 8 lines
arch/arm64/mm/pageattr.c +4 -1
@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
 	if (!can_set_direct_map())
 		return 0;
 
-	return set_memory_valid(addr, nr, valid);
+	/*
+	 * Execmem cache uses this function to reset permissions on linear mapping
+	 * when freeing unused cache block. On x86 it makes memory RW which is
+	 * desirable. 

Hm, maybe desirable for execmem but that doesn't really mean the x86 behaviour is correct. Maybe it makes more sense to change the x86 to align with the arm64 behaviour here?

BTW we should probably document this API a little bit, I never thought abut what "valid" actually means until now. I had thought of it as "I can access this memory" but that's an unclear concept and now I realise "valid" is a technical concept in Arm that's confusing. And it's extra confusing if the kernel API uses "valid" to mean a _different_ thing.

Also, shouldn't execmem be using set_memory_default_noflush() before freeing anyway? I guess that shoudl even be documented as "if you change anything you need to call this before you free the page".

Quoted 9 lines

On ARM64 set_memory_valid() just change valid bit which

arch/arm64/mm/pageattr.c +5 -0
+	 * leave direct mapping read-only so use set_memory_default instead.
+	 */
+
+	return valid ? set_memory_default(addr, nr) :
+		       set_memory_valid(addr, nr, false);
 }
 
 #ifdef CONFIG_DEBUG_PAGEALLOC
#8 lore raw

Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean

Quoted 49 lines · Brendan Jackman
Quoted 35 lines · On Thu Jun 11, 2026 at 1:01 PM UTC, =?UTF-8?q?Adrian=20Barna=C5=9B?= wrote:

Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.

An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.

Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.

---
 arch/arm64/mm/pageattr.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
arch/arm64/mm/pageattr.c +7 -0
index 88720bbba892..eaefdf90b0d5 100644
@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages)
 					__pgprot(PTE_PXN));
 }
 
+static int set_memory_default(unsigned long addr, int numpages)
+{
+	return __change_memory_common(addr, PAGE_SIZE * numpages,
+				      __pgprot(PTE_VALID),
+				      __pgprot(PTE_RDONLY));
+}
+
 int set_memory_valid(unsigned long addr, int numpages, int enable)
 {
 	if (enable)

arch/arm64/mm/pageattr.c int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) · line 375

Quoted 8 lines
arch/arm64/mm/pageattr.c +4 -1
@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
 	if (!can_set_direct_map())
 		return 0;
 
-	return set_memory_valid(addr, nr, valid);
+	/*
+	 * Execmem cache uses this function to reset permissions on linear mapping
+	 * when freeing unused cache block. On x86 it makes memory RW which is
+	 * desirable. 

Hm, maybe desirable for execmem but that doesn't really mean the x86 behaviour is correct. Maybe it makes more sense to change the x86 to align with the arm64 behaviour here?

Quoted 5 lines

BTW we should probably document this API a little bit, I never thought abut what "valid" actually means until now. I had thought of it as "I can access this memory" but that's an unclear concept and now I realise "valid" is a technical concept in Arm that's confusing. And it's extra confusing if the kernel API uses "valid" to mean a _different_ thing.

I've got confused too and that's how set_direct_map_valid() got into x86 with a different semantics than on arm64.

What execmem really needs is set_direct_map_default() variant that gets nr_pages.

AFAIR, set_direct_map_default() has a single 'page' parameter because it was added to reset permissions for the direct map alias for vmalloc()'ed pages before there was VMALLOC_HUGE and each page had to be reset independently anyway.

Maybe it's time to add nr_pages to set_direct_map_valid().

Quoted 3 lines

Also, shouldn't execmem be using set_memory_default_noflush() before freeing anyway? I guess that shoudl even be documented as "if you change anything you need to call this before you free the page".

It does on x86 because there set_direct_map_valid() is the same as set_direct_map_default().

Quoted 11 lines
Quoted 9 lines

On ARM64 set_memory_valid() just change valid bit which

arch/arm64/mm/pageattr.c +5 -0
+	 * leave direct mapping read-only so use set_memory_default instead.
+	 */
+
+	return valid ? set_memory_default(addr, nr) :
+		       set_memory_valid(addr, nr, false);
 }
 
 #ifdef CONFIG_DEBUG_PAGEALLOC
Signature · 2 lines
Sincerely yours,
Mike.
#9 lore raw

Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean

Quoted 84 lines · Mike Rapoport
Quoted 49 lines · Brendan Jackman
Quoted 35 lines · On Thu Jun 11, 2026 at 1:01 PM UTC, =?UTF-8?q?Adrian=20Barna=C5=9B?= wrote:

Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.

An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.

Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.

---
 arch/arm64/mm/pageattr.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
arch/arm64/mm/pageattr.c +7 -0
index 88720bbba892..eaefdf90b0d5 100644
@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages)
 					__pgprot(PTE_PXN));
 }
 
+static int set_memory_default(unsigned long addr, int numpages)
+{
+	return __change_memory_common(addr, PAGE_SIZE * numpages,
+				      __pgprot(PTE_VALID),
+				      __pgprot(PTE_RDONLY));
+}
+
 int set_memory_valid(unsigned long addr, int numpages, int enable)
 {
 	if (enable)

arch/arm64/mm/pageattr.c int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) · line 375

Quoted 8 lines
arch/arm64/mm/pageattr.c +4 -1
@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
 	if (!can_set_direct_map())
 		return 0;
 
-	return set_memory_valid(addr, nr, valid);
+	/*
+	 * Execmem cache uses this function to reset permissions on linear mapping
+	 * when freeing unused cache block. On x86 it makes memory RW which is
+	 * desirable.

Hm, maybe desirable for execmem but that doesn't really mean the x86 behaviour is correct. Maybe it makes more sense to change the x86 to align with the arm64 behaviour here?

Quoted 5 lines

BTW we should probably document this API a little bit, I never thought abut what "valid" actually means until now. I had thought of it as "I can access this memory" but that's an unclear concept and now I realise "valid" is a technical concept in Arm that's confusing. And it's extra confusing if the kernel API uses "valid" to mean a _different_ thing.

I've got confused too and that's how set_direct_map_valid() got into x86 with a different semantics than on arm64.

What execmem really needs is set_direct_map_default() variant that gets nr_pages.

AFAIR, set_direct_map_default() has a single 'page' parameter because it was added to reset permissions for the direct map alias for vmalloc()'ed pages before there was VMALLOC_HUGE and each page had to be reset independently anyway.

Maybe it's time to add nr_pages to set_direct_map_valid().

Quoted 3 lines

Also, shouldn't execmem be using set_memory_default_noflush() before freeing anyway? I guess that shoudl even be documented as "if you change anything you need to call this before you free the page".

It does on x86 because there set_direct_map_valid() is the same as set_direct_map_default().

Quoted 8 lines
Quoted 8 lines

On ARM64 set_memory_valid() just change valid bit which

arch/arm64/mm/pageattr.c +5 -0
+	 * leave direct mapping read-only so use set_memory_default instead.
+	 */
+
+	return valid ? set_memory_default(addr, nr) :
+		       set_memory_valid(addr, nr, false);
 }
Quoted 7 lines
Quoted 3 lines
Quoted 1 line

#ifdef CONFIG_DEBUG_PAGEALLOC

Signature · 2 lines
Sincerely yours,
Mike.

Hi Mike, Brendan,

Thanks for taking a look at the RFC.

I was also quite confused by this initially. I spent some time debugging until I realized why unloading all the modules was causing the kernel to crash.

The reason I took this approach was that I wanted to send out a working prototype for arm64 that wouldn't interfere with the existing, working implementation on x86.

Following your suggestion, I can put together a preparatory patch series to refactor the set_direct_map_* APIs to accept a nr_pages parameter. This refactoring would also allow us to drop the redundant set_area_direct_map helper. I could then rebase the rox_cache series on top of that.

Does this sound like a good path forward?

Thanks,
Adrian

#10 lore raw

Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean

Hi Adrian,

Quoted 29 lines · Adrian Barnaś
Quoted 17 lines · Mike Rapoport
Quoted 10 lines

Hm, maybe desirable for execmem but that doesn't really mean the x86 behaviour is correct. Maybe it makes more sense to change the x86 to align with the arm64 behaviour here?

BTW we should probably document this API a little bit, I never thought abut what "valid" actually means until now. I had thought of it as "I can access this memory" but that's an unclear concept and now I realise "valid" is a technical concept in Arm that's confusing. And it's extra confusing if the kernel API uses "valid" to mean a _different_ thing.

I've got confused too and that's how set_direct_map_valid() got into x86 with a different semantics than on arm64.

What execmem really needs is set_direct_map_default() variant that gets nr_pages.

Quoted 6 lines

AFAIR, set_direct_map_default() has a single 'page' parameter because it was added to reset permissions for the direct map alias for vmalloc()'ed pages before there was VMALLOC_HUGE and each page had to be reset independently anyway.

Maybe it's time to add nr_pages to set_direct_map_valid().

I was also quite confused by this initially. I spent some time debugging until I realized why unloading all the modules was causing the kernel to crash.

Quoted 6 lines

The reason I took this approach was that I wanted to send out a working prototype for arm64 that wouldn't interfere with the existing, working implementation on x86.

Following your suggestion, I can put together a preparatory patch series to refactor the set_direct_map_* APIs to accept a nr_pages parameter. This

There was a patch Nikita sent a while ago that does something similar:

https://lore.kernel.org/all/[email protected]

I believe you can start from there.

Quoted 1 line

refactoring would also allow us to drop the redundant set_area_direct_map

We can't drop set_area_direct_map() because vmalloc pages might be not physically contiguous.

Quoted 6 lines

helper. I could then rebase the rox_cache series on top of that.

Does this sound like a good path forward?

Thanks,
Adrian

Signature · 2 lines
Sincerely yours,
Mike.
#11 lore raw

Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean

Quoted 36 lines · Mike Rapoport

Hi Adrian,

Quoted 33 lines · Adrian Barnaś
Quoted 17 lines · Mike Rapoport
Quoted 10 lines

Hm, maybe desirable for execmem but that doesn't really mean the x86 behaviour is correct. Maybe it makes more sense to change the x86 to align with the arm64 behaviour here?

BTW we should probably document this API a little bit, I never thought abut what "valid" actually means until now. I had thought of it as "I can access this memory" but that's an unclear concept and now I realise "valid" is a technical concept in Arm that's confusing. And it's extra confusing if the kernel API uses "valid" to mean a _different_ thing.

I've got confused too and that's how set_direct_map_valid() got into x86 with a different semantics than on arm64.

What execmem really needs is set_direct_map_default() variant that gets nr_pages.

Quoted 6 lines

AFAIR, set_direct_map_default() has a single 'page' parameter because it was added to reset permissions for the direct map alias for vmalloc()'ed pages before there was VMALLOC_HUGE and each page had to be reset independently anyway.

Maybe it's time to add nr_pages to set_direct_map_valid().

I was also quite confused by this initially. I spent some time debugging until I realized why unloading all the modules was causing the kernel to crash.

The reason I took this approach was that I wanted to send out a working prototype for arm64 that wouldn't interfere with the existing, working implementation on x86.

Quoted 9 lines
Quoted 2 lines

Following your suggestion, I can put together a preparatory patch series to refactor the set_direct_map_* APIs to accept a nr_pages parameter. This

There was a patch Nikita sent a while ago that does something similar:

https://lore.kernel.org/all/[email protected]

I believe you can start from there.

I will pick the 01 patch from there to my series.

Quoted 5 lines
Quoted 1 line

refactoring would also allow us to drop the redundant set_area_direct_map

We can't drop set_area_direct_map() because vmalloc pages might be not physically contiguous.

Agree.

Quoted 10 lines
Quoted 6 lines

helper. I could then rebase the rox_cache series on top of that.

Does this sound like a good path forward?

Thanks,
Adrian

Signature · 2 lines
Sincerely yours,
Mike.

Thanks,
Adrian

#12 lore raw

Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean

In reply to Adrian Barnaś

Hi Adrian,

Quoted 5 lines · Adrian Barnaś

Following your suggestion, I can put together a preparatory patch series to refactor the set_direct_map_* APIs to accept a nr_pages parameter. This refactoring would also allow us to drop the redundant set_area_direct_map helper. I could then rebase the rox_cache series on top of that.

I've been looking at something related and I was already half way through adding numpages to set_direct_map when I remembered this conversation :)

Here's what I have for now:

https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=direct-map/v0

This is only lightly build tested for arm64 and x86.

As this is WIP, things may change there, but I hope it can save you some time with your arm64 ROX cache work.

Quoted 4 lines

Does this sound like a good path forward?

Thanks,
Adrian

Signature · 2 lines
Sincerely yours,
Mike.
#13 lore raw

Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean

Quoted 9 lines · Mike Rapoport

Hi Adrian,

Quoted 5 lines · Adrian Barnaś

Following your suggestion, I can put together a preparatory patch series to refactor the set_direct_map_* APIs to accept a nr_pages parameter. This refactoring would also allow us to drop the redundant set_area_direct_map helper. I could then rebase the rox_cache series on top of that.

Quoted 7 lines

I've been looking at something related and I was already half way through adding numpages to set_direct_map when I remembered this conversation :)

Here's what I have for now:

https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=direct-map/v0

I did roughly the same on my local tree.

Quoted 3 lines

This is only lightly build tested for arm64 and x86.

As this is WIP, things may change there, but I hope it can save you some time with your arm64 ROX cache work.

I will fetch your commits and I will keep an eye on your tree if anything changes.

Thanks,
Adrian

#14 lore raw

Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean

In reply to Adrian Barnaś
Quoted 21 lines · Adrian Barnaś

Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.

An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.

Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.

---
 arch/arm64/mm/pageattr.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
arch/arm64/mm/pageattr.c +0 -0
index 88720bbba892..eaefdf90b0d5 100644

arch/arm64/mm/pageattr.c int set_memory_x(unsigned long addr, int numpages) · line 246

Quoted 8 lines
arch/arm64/mm/pageattr.c +5 -0
@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages)
 					__pgprot(PTE_PXN));
 }
 
+static int set_memory_default(unsigned long addr, int numpages)
+{
+	return __change_memory_common(addr, PAGE_SIZE * numpages,
+				      __pgprot(PTE_VALID),
+				      __pgprot(PTE_RDONLY));

This is not sufficient to convert an invalid entry to valid. As well as setting the PTE_VALID bit, you would also need to clear the PTE_PRESENT_INVALID and set PTE_MAYBE_NG.

e.g:

int set_memory_valid(unsigned long addr, int numpages, int enable)

{
	if (enable)
		return __change_memory_common(addr, PAGE_SIZE * numpages,
					__pgprot(PTE_PRESENT_VALID_KERNEL),
					__pgprot(PTE_PRESENT_INVALID));
Quoted 9 lines
arch/arm64/mm/pageattr.c +2 -0
+}
+
 int set_memory_valid(unsigned long addr, int numpages, int enable)
 {
 	if (enable)
@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
 	if (!can_set_direct_map())
 		return 0;
 

arch/arm64/mm/pageattr.c int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) · line 380

Quoted 10 lines
arch/arm64/mm/pageattr.c +9 -1
-	return set_memory_valid(addr, nr, valid);
+	/*
+	 * Execmem cache uses this function to reset permissions on linear mapping
+	 * when freeing unused cache block. On x86 it makes memory RW which is
+	 * desirable. On ARM64 set_memory_valid() just change valid bit which
+	 * leave direct mapping read-only so use set_memory_default instead.
+	 */
+
+	return valid ? set_memory_default(addr, nr) :
+		       set_memory_valid(addr, nr, false);

Surely execmem should just be using set_direct_map_default_noflush() if that's the behaviour it wants?

I think that the current implementation of set_direct_map_default_noflush() doesn't undo the effects of set_memory_nx() / set_memory_x(). That might be worth checking?

Thanks,
Ryan

Quoted 3 lines
arch/arm64/mm/pageattr.c +0 -0
 }
 
 #ifdef CONFIG_DEBUG_PAGEALLOC
#15 lore raw

Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean

Quoted 68 lines · Ryan Roberts
Quoted 21 lines · Adrian Barnaś

Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.

An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.

Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.

---
 arch/arm64/mm/pageattr.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
arch/arm64/mm/pageattr.c +0 -0
index 88720bbba892..eaefdf90b0d5 100644

arch/arm64/mm/pageattr.c int set_memory_x(unsigned long addr, int numpages) · line 246

Quoted 8 lines
arch/arm64/mm/pageattr.c +5 -0
@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages)
 					__pgprot(PTE_PXN));
 }
 
+static int set_memory_default(unsigned long addr, int numpages)
+{
+	return __change_memory_common(addr, PAGE_SIZE * numpages,
+				      __pgprot(PTE_VALID),
+				      __pgprot(PTE_RDONLY));

This is not sufficient to convert an invalid entry to valid. As well as setting the PTE_VALID bit, you would also need to clear the PTE_PRESENT_INVALID and set PTE_MAYBE_NG.

e.g:

int set_memory_valid(unsigned long addr, int numpages, int enable)

{
	if (enable)
		return __change_memory_common(addr, PAGE_SIZE * numpages,
					__pgprot(PTE_PRESENT_VALID_KERNEL),
					__pgprot(PTE_PRESENT_INVALID));
Quoted 9 lines
arch/arm64/mm/pageattr.c +2 -0
+}
+
 int set_memory_valid(unsigned long addr, int numpages, int enable)
 {
 	if (enable)
@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
 	if (!can_set_direct_map())
 		return 0;
 

arch/arm64/mm/pageattr.c int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) · line 380

Quoted 10 lines
arch/arm64/mm/pageattr.c +9 -1
-	return set_memory_valid(addr, nr, valid);
+	/*
+	 * Execmem cache uses this function to reset permissions on linear mapping
+	 * when freeing unused cache block. On x86 it makes memory RW which is
+	 * desirable. On ARM64 set_memory_valid() just change valid bit which
+	 * leave direct mapping read-only so use set_memory_default instead.
+	 */
+
+	return valid ? set_memory_default(addr, nr) :
+		       set_memory_valid(addr, nr, false);

Surely execmem should just be using set_direct_map_default_noflush() if that's the behaviour it wants?

Quoted 3 lines

I think that the current implementation of set_direct_map_default_noflush() doesn't undo the effects of set_memory_nx() / set_memory_x(). That might be worth checking?

It's also worth mentioning that set_direct_map_valid_noflush() has "noflush" in the name, implies it doesn't expect/require any TLB flushing to occur. But the implementation will perform tlb flushing for any case that is not just a invalid->valid transition (which for the existing impl is the case when valid=true and for your changes is never the case - see __change_memory_common).

But execmem doesn't do any tlb flushing so it looks to me like it actually requires that set_direct_map_valid_noflush() handles the tlb flushing? All seems a bit fishy and probably warrants a cleanup to make things clearer.

Quoted 9 lines

Thanks,
Ryan

Quoted 3 lines
arch/arm64/mm/pageattr.c +0 -0
 }
 
 #ifdef CONFIG_DEBUG_PAGEALLOC
#16 lore raw

Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean

Quoted 38 lines · Ryan Roberts
Quoted 37 lines · Ryan Roberts
Quoted 21 lines · Adrian Barnaś

Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.

An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.

Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.

---
 arch/arm64/mm/pageattr.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
arch/arm64/mm/pageattr.c +0 -0
index 88720bbba892..eaefdf90b0d5 100644

arch/arm64/mm/pageattr.c int set_memory_x(unsigned long addr, int numpages) · line 246

Quoted 8 lines
arch/arm64/mm/pageattr.c +5 -0
@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages)
 					__pgprot(PTE_PXN));
 }
 
+static int set_memory_default(unsigned long addr, int numpages)
+{
+	return __change_memory_common(addr, PAGE_SIZE * numpages,
+				      __pgprot(PTE_VALID),
+				      __pgprot(PTE_RDONLY));

This is not sufficient to convert an invalid entry to valid. As well as setting the PTE_VALID bit, you would also need to clear the PTE_PRESENT_INVALID and set PTE_MAYBE_NG.

e.g:

Quoted 7 lines
Quoted 7 lines

int set_memory_valid(unsigned long addr, int numpages, int enable)

{
	if (enable)
		return __change_memory_common(addr, PAGE_SIZE * numpages,
					__pgprot(PTE_PRESENT_VALID_KERNEL),
					__pgprot(PTE_PRESENT_INVALID));

Thanks, I will fix that

Quoted 34 lines
Quoted 24 lines
Quoted 10 lines
arch/arm64/mm/pageattr.c +2 -0
+}
+
 int set_memory_valid(unsigned long addr, int numpages, int enable)
 {
 	if (enable)
@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
 	if (!can_set_direct_map())
 		return 0;
 

arch/arm64/mm/pageattr.c int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) · line 380

Quoted 10 lines
arch/arm64/mm/pageattr.c +9 -1
-	return set_memory_valid(addr, nr, valid);
+	/*
+	 * Execmem cache uses this function to reset permissions on linear mapping
+	 * when freeing unused cache block. On x86 it makes memory RW which is
+	 * desirable. On ARM64 set_memory_valid() just change valid bit which
+	 * leave direct mapping read-only so use set_memory_default instead.
+	 */
+
+	return valid ? set_memory_default(addr, nr) :
+		       set_memory_valid(addr, nr, false);

Surely execmem should just be using set_direct_map_default_noflush() if that's the behaviour it wants?

Quoted 3 lines

I think that the current implementation of set_direct_map_default_noflush() doesn't undo the effects of set_memory_nx() / set_memory_x(). That might be worth checking?

It's also worth mentioning that set_direct_map_valid_noflush() has "noflush" in the name, implies it doesn't expect/require any TLB flushing to occur. But the implementation will perform tlb flushing for any case that is not just a invalid->valid transition (which for the existing impl is the case when valid=true and for your changes is never the case - see __change_memory_common).

Quoted 4 lines

But execmem doesn't do any tlb flushing so it looks to me like it actually requires that set_direct_map_valid_noflush() handles the tlb flushing? All seems a bit fishy and probably warrants a cleanup to make things clearer.

I think that the clean approach would be to have the set_direct_map_default function (without noflush) that does flushing as needed (like the current one on ARM64). I am not entirely sure but x86 seems to handle permission faults gracefully while on ARM64 those would cause panics. Is that correct?

Quoted 10 lines
Quoted 9 lines

Thanks,
Ryan

Quoted 3 lines
arch/arm64/mm/pageattr.c +0 -0
 }
 
 #ifdef CONFIG_DEBUG_PAGEALLOC

Thanks,
Adrian

#17 lore raw

RFC PATCH 4/6 arm64: mm: add helper to fill execmem with trapping instructions

In reply to Adrian Barnaś

Implement the architecture-specific execmem_fill_trapping_insns() helper to poison executable memory regions.

When CONFIG_ARCH_HAS_EXECMEM_ROX is enabled, the execmem subsystem requires a way to fill unused or freed executable memory with architecture-specific trapping instructions. This implementation fills the specified region with AARCH64_BREAK_FAULT instructions and flushes the icache to ensure the traps are immediately visible to execution.

---
 arch/arm64/mm/init.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/arm64/mm/init.c +14 -0
index c673a9a839dd..71aa745e0bef 100644
@@ -408,6 +408,20 @@ void dump_mem_limit(void)
 }
 
 #ifdef CONFIG_EXECMEM
+
+#ifdef CONFIG_ARCH_HAS_EXECMEM_ROX
+void execmem_fill_trapping_insns(void *ptr, size_t size)
+{
+	int nr_inst = size / AARCH64_INSN_SIZE;
+	__le32 *updptr = ptr;
+
+	for (int i = 0; i < nr_inst; i++)
+		updptr[i] = cpu_to_le32(AARCH64_BREAK_FAULT);
+
+	flush_icache_range((unsigned long)ptr, (unsigned long)ptr + size);
+}
+#endif
+
 static u64 module_direct_base __ro_after_init = 0;
 static u64 module_plt_base __ro_after_init = 0;
 
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
#18 lore raw

Re: [RFC PATCH 4/6] arm64: mm: add helper to fill execmem with trapping instructions

Quoted 19 lines · Adrian Barnaś

Implement the architecture-specific execmem_fill_trapping_insns() helper to poison executable memory regions.

When CONFIG_ARCH_HAS_EXECMEM_ROX is enabled, the execmem subsystem requires a way to fill unused or freed executable memory with architecture-specific trapping instructions. This implementation fills the specified region with AARCH64_BREAK_FAULT instructions and flushes the icache to ensure the traps are immediately visible to execution.

---
 arch/arm64/mm/init.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/arm64/mm/init.c +0 -0
index c673a9a839dd..71aa745e0bef 100644

arch/arm64/mm/init.c void dump_mem_limit(void) · line 415

Quoted 8 lines
arch/arm64/mm/init.c +5 -0
@@ -408,6 +408,20 @@ void dump_mem_limit(void)
 }
 
 #ifdef CONFIG_EXECMEM
+
+#ifdef CONFIG_ARCH_HAS_EXECMEM_ROX
+void execmem_fill_trapping_insns(void *ptr, size_t size)
+{
+	int nr_inst = size / AARCH64_INSN_SIZE;

The x86 instruction is 1 byte, so it can exactly fill any provided buffer. For arm64, the instruction is 4 bytes so we can only exactly fill the buffer if it's size is 4 byte aligned.

I'm guessing that in practice, size will always be page aligned so we are good? Perhaps worth a WARN_ON_ONCE() though?

Thanks,
Ryan

Quoted 12 lines
arch/arm64/mm/init.c +9 -0
+	__le32 *updptr = ptr;
+
+	for (int i = 0; i < nr_inst; i++)
+		updptr[i] = cpu_to_le32(AARCH64_BREAK_FAULT);
+
+	flush_icache_range((unsigned long)ptr, (unsigned long)ptr + size);
+}
+#endif
+
 static u64 module_direct_base __ro_after_init = 0;
 static u64 module_plt_base __ro_after_init = 0;
 
#19 lore raw

Re: [RFC PATCH 4/6] arm64: mm: add helper to fill execmem with trapping instructions

Quoted 33 lines · Ryan Roberts
Quoted 19 lines · Adrian Barnaś

Implement the architecture-specific execmem_fill_trapping_insns() helper to poison executable memory regions.

When CONFIG_ARCH_HAS_EXECMEM_ROX is enabled, the execmem subsystem requires a way to fill unused or freed executable memory with architecture-specific trapping instructions. This implementation fills the specified region with AARCH64_BREAK_FAULT instructions and flushes the icache to ensure the traps are immediately visible to execution.

---
 arch/arm64/mm/init.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/arm64/mm/init.c +0 -0
index c673a9a839dd..71aa745e0bef 100644

arch/arm64/mm/init.c void dump_mem_limit(void) · line 415

Quoted 8 lines
arch/arm64/mm/init.c +5 -0
@@ -408,6 +408,20 @@ void dump_mem_limit(void)
 }
 
 #ifdef CONFIG_EXECMEM
+
+#ifdef CONFIG_ARCH_HAS_EXECMEM_ROX
+void execmem_fill_trapping_insns(void *ptr, size_t size)
+{
+	int nr_inst = size / AARCH64_INSN_SIZE;

The x86 instruction is 1 byte, so it can exactly fill any provided buffer. For arm64, the instruction is 4 bytes so we can only exactly fill the buffer if it's size is 4 byte aligned.

Quoted 1 line

I'm guessing that in practice, size will always be page aligned so we are good?

The size is always page aligned:

void *execmem_alloc(enum execmem_type type, size_t size)

{
	...

	size = PAGE_ALIGN(size);
Signature · 2 lines
Sincerely yours,
Mike.
#20 lore raw

RFC PATCH 5/6 arm64: execmem: enable EXECMEM_ROX_CACHE on supported CPUs

In reply to Adrian Barnaś

Enable EXECMEM_ROX_CACHE support for ARM64 systems that implement the bbml2_no_abort CPU feature.

Using the ROX cache brings a performance boost by reducing linear region fragmentation caused by strict memory permissions (e.g., W^X enforcement). Grouping executable code (which is read-only in the linear region alias) into PMD-sized block mappings reduces TLB pressure and page table size.

This is only enabled on systems with bbml2_no_abort, as splitting these large blocks to make pages writable during module loading would otherwise risk triggering TLB Conflict Aborts.

---
 arch/arm64/Kconfig   |  1 +
 arch/arm64/mm/init.c | 22 +++++++++++++++++++++-
 2 files changed, 22 insertions(+), 1 deletion(-)
arch/arm64/Kconfig +1 -0
index 38dba5f7e4d2..79c347ab841e 100644
@@ -285,6 +285,7 @@ config ARM64
 	select USER_STACKTRACE_SUPPORT
 	select VDSO_GETRANDOM
 	select VMAP_STACK
+	select ARCH_HAS_EXECMEM_ROX
 	help
 	  ARM 64-bit (AArch64) Linux support.
 
arch/arm64/mm/init.c +21 -1
index 71aa745e0bef..8269d7747b84 100644
@@ -420,6 +420,12 @@ void execmem_fill_trapping_insns(void *ptr, size_t size)
 
 	flush_icache_range((unsigned long)ptr, (unsigned long)ptr + size);
 }
+
+#define MODULE_TEXT_FLAG	EXECMEM_ROX_CACHE
+#define MODULE_TEXT_PGPROT	PAGE_KERNEL_ROX
+#else
+#define MODULE_TEXT_FLAG	(0)
+#define MODULE_TEXT_PGPROT	PAGE_KERNEL
 #endif
 
 static u64 module_direct_base __ro_after_init = 0;
@@ -511,6 +517,8 @@ struct execmem_info __init *execmem_arch_setup(void)
 {
 	unsigned long fallback_start = 0, fallback_end = 0;
 	unsigned long start = 0, end = 0;
+	enum execmem_range_flags module_text_flags = 0;
+	pgprot_t module_text_pgprot = PAGE_KERNEL;
 
 	module_init_limits();
 
@@ -531,12 +539,24 @@ struct execmem_info __init *execmem_arch_setup(void)
 		end = module_plt_base + SZ_2G;
 	}
 
+	/*
+	 * The ROX Cache requires bbml2_no_abort because it uses large block
+	 * mappings. On systems without this guarantee, splitting these blocks
+	 * to make pages writable for module loading can trigger TLB Conflict
+	 * Aborts.
+	 */
+	if (system_supports_bbml2_noabort()) {
+		module_text_flags = MODULE_TEXT_FLAG;
+		module_text_pgprot = MODULE_TEXT_PGPROT;
+	}
+
 	execmem_info = (struct execmem_info){
 		.ranges = {
 			[EXECMEM_MODULE_TEXT] = {
 				.start	= start,
 				.end	= end,
-				.pgprot	= PAGE_KERNEL,
+				.flags = module_text_flags,
+				.pgprot	= module_text_pgprot,
 				.alignment = 1,
 				.fallback_start	= fallback_start,
 				.fallback_end	= fallback_end,
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
#21 lore raw

Re: [RFC PATCH 5/6] arm64: execmem: enable EXECMEM_ROX_CACHE on supported CPUs

Quoted 7 lines · Adrian Barnaś

Enable EXECMEM_ROX_CACHE support for ARM64 systems that implement the bbml2_no_abort CPU feature.

Using the ROX cache brings a performance boost by reducing linear region fragmentation caused by strict memory permissions (e.g., W^X enforcement). Grouping executable code (which is read-only in the linear region alias) into PMD-sized block mappings reduces TLB pressure and page table size.

Do you have any data on fragmentation reduction and/or performance improvement in practice due to this change?

Quoted 15 lines

This is only enabled on systems with bbml2_no_abort, as splitting these large blocks to make pages writable during module loading would otherwise risk triggering TLB Conflict Aborts.

Quoted 1 line
---
 arch/arm64/Kconfig   |  1 +
 arch/arm64/mm/init.c | 22 +++++++++++++++++++++-
 2 files changed, 22 insertions(+), 1 deletion(-)
arch/arm64/Kconfig +0 -0
index 38dba5f7e4d2..79c347ab841e 100644

arch/arm64/Kconfig config ARM64 · line 288

Quoted 4 lines
arch/arm64/Kconfig +1 -0
@@ -285,6 +285,7 @@ config ARM64
 	select USER_STACKTRACE_SUPPORT
 	select VDSO_GETRANDOM
 	select VMAP_STACK
+	select ARCH_HAS_EXECMEM_ROX

nit: This list is sorted in alphabetical order; please maintain that ordering.

Quoted 33 lines
arch/arm64/Kconfig +0 -0
 	help
 	  ARM 64-bit (AArch64) Linux support.
 
arch/arm64/mm/init.c +8 -0
index 71aa745e0bef..8269d7747b84 100644
@@ -420,6 +420,12 @@ void execmem_fill_trapping_insns(void *ptr, size_t size)
 
 	flush_icache_range((unsigned long)ptr, (unsigned long)ptr + size);
 }
+
+#define MODULE_TEXT_FLAG	EXECMEM_ROX_CACHE
+#define MODULE_TEXT_PGPROT	PAGE_KERNEL_ROX
+#else
+#define MODULE_TEXT_FLAG	(0)
+#define MODULE_TEXT_PGPROT	PAGE_KERNEL
 #endif
 
 static u64 module_direct_base __ro_after_init = 0;
@@ -511,6 +517,8 @@ struct execmem_info __init *execmem_arch_setup(void)
 {
 	unsigned long fallback_start = 0, fallback_end = 0;
 	unsigned long start = 0, end = 0;
+	enum execmem_range_flags module_text_flags = 0;
+	pgprot_t module_text_pgprot = PAGE_KERNEL;
 
 	module_init_limits();
 
@@ -531,12 +539,24 @@ struct execmem_info __init *execmem_arch_setup(void)
 		end = module_plt_base + SZ_2G;
 	}
 

arch/arm64/mm/init.c struct execmem_info __init *execmem_arch_setup(void) · line 551

Quoted 10 lines
arch/arm64/mm/init.c +10 -0
+	/*
+	 * The ROX Cache requires bbml2_no_abort because it uses large block
+	 * mappings. On systems without this guarantee, splitting these blocks
+	 * to make pages writable for module loading can trigger TLB Conflict
+	 * Aborts.
+	 */
+	if (system_supports_bbml2_noabort()) {
+		module_text_flags = MODULE_TEXT_FLAG;
+		module_text_pgprot = MODULE_TEXT_PGPROT;
+	}

Perhaps this is a bit clearer? Then you don't need the MODULE_TEXT_* macros:

	if (IS_ENABLED(CONFIG_ARCH_HAS_EXECMEM_ROX) &&
	    system_supports_bbml2_noabort()) {
		module_text_flags = EXECMEM_ROX_CACHE;
		module_text_pgprot = PAGE_KERNEL_ROX;
	}

Thanks,
Ryan

Quoted 12 lines
arch/arm64/mm/init.c +3 -1
+
 	execmem_info = (struct execmem_info){
 		.ranges = {
 			[EXECMEM_MODULE_TEXT] = {
 				.start	= start,
 				.end	= end,
-				.pgprot	= PAGE_KERNEL,
+				.flags = module_text_flags,
+				.pgprot	= module_text_pgprot,
 				.alignment = 1,
 				.fallback_start	= fallback_start,
 				.fallback_end	= fallback_end,
#22 lore raw

RFC PATCH 6/6 arm64: mm: support PMD page coalescing in the linear map

In reply to Adrian Barnaś

Implement PMD block coalescing to merge fragmented linear mapping regions back into huge pages when restoring the read-only attribute.

When memory allocated with VM_ALLOW_HUGE_VMAP (such as for the execmem ROX cache) has its permissions modified, the PMD block mapping is split into individual PTEs. Without this change, when that memory have its RO attribute subsequently cleared and set the mapping remains permanently fragmented into 4K pages.

---
 arch/arm64/include/asm/mmu.h |  1 +
 arch/arm64/mm/mmu.c          | 95 ++++++++++++++++++++++++++++++++++++
 arch/arm64/mm/pageattr.c     |  7 +++
 3 files changed, 103 insertions(+)
arch/arm64/include/asm/mmu.h +1 -0
index 137a173df1ff..19158bacb2df 100644
@@ -80,6 +80,7 @@ extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot);
 extern void mark_linear_text_alias_ro(void);
 extern int split_kernel_leaf_mapping(unsigned long start, unsigned long end);
 extern void linear_map_maybe_split_to_ptes(void);
+void try_collapse_kernel_pmd(unsigned long addr);
 
 /*
  * This check is triggered during the early boot before the cpufeature
arch/arm64/mm/mmu.c +95 -0

101 lines not on this page · raw message

arch/arm64/mm/pageattr.c +7 -0
index eaefdf90b0d5..11e0b60264c3 100644
@@ -200,6 +200,13 @@ static int change_memory_common(unsigned long addr, int numpages,
 			if (ret)
 				return ret;
 		}
+		/*
+		 * When setting a read-only flag on the linear region, the memory
+		 * may have been backed by a PMD before being split. Try to
+		 * collapse it back into a PMD to restore huge page performance.
+		 */
+		if (pgprot_val(set_mask) == PTE_RDONLY && area->flags & VM_ALLOW_HUGE_VMAP)
+			try_collapse_kernel_pmd((u64)page_address(area->pages[0]));
 	}
 
 	/*
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
#23 lore raw

Re: [RFC PATCH 6/6] arm64: mm: support PMD page coalescing in the linear map

Quoted 8 lines · Adrian Barnaś

Implement PMD block coalescing to merge fragmented linear mapping regions back into huge pages when restoring the read-only attribute.

When memory allocated with VM_ALLOW_HUGE_VMAP (such as for the execmem ROX cache) has its permissions modified, the PMD block mapping is split into individual PTEs. Without this change, when that memory have its RO attribute subsequently cleared and set the mapping remains permanently fragmented into 4K pages.

I'd like to make sure I understand this correctly: So the execmem ROX cache is allocated using vmalloc_huge such that all it's backing memory are PMD-sized pages. It is initially poisoned with a faulting instruction and marked ROX. When a module is loaded, it's text is copied into a portion of the ROX cache and executed from there? To do that, the required number of (4K) pages are allocated from ROX cache, and the permissions are changed on that sub-region to RW, meaning we have to split the mapping from a PMD to (cont)PTEs in both vmalloc space and in the linear map. After the text is copied, the sub-region is switched back to ROX. But without that change, the 2M region is now mapped by (cont)PTEs for all of time?

Quoted 25 lines
---
 arch/arm64/include/asm/mmu.h |  1 +
 arch/arm64/mm/mmu.c          | 95 ++++++++++++++++++++++++++++++++++++
 arch/arm64/mm/pageattr.c     |  7 +++
 3 files changed, 103 insertions(+)
arch/arm64/include/asm/mmu.h +1 -0
index 137a173df1ff..19158bacb2df 100644
@@ -80,6 +80,7 @@ extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot);
 extern void mark_linear_text_alias_ro(void);
 extern int split_kernel_leaf_mapping(unsigned long start, unsigned long end);
 extern void linear_map_maybe_split_to_ptes(void);
+void try_collapse_kernel_pmd(unsigned long addr);
 
 /*
  * This check is triggered during the early boot before the cpufeature
arch/arm64/mm/mmu.c +0 -0
index a6a00accf4f9..d74226fa1c9b 100644

arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 778

Quoted 10 lines
arch/arm64/mm/mmu.c +7 -0
@@ -769,6 +769,101 @@ static inline bool force_pte_mapping(void)
 
 static DEFINE_MUTEX(pgtable_split_lock);
 
+static inline bool __pte_can_be_collapsed(pte_t pte, unsigned long pfn, pgprot_t prot)
+{
+	if (!pte_valid(pte))
+		return false;
+	if (pte_pfn(pte) != pfn)
+		return false;
+	if ((pgprot_val(pte_pgprot(pte)) & ~PTE_CONT) != pgprot_val(prot))

Do we want to permit differing values for access and dirty? We do for user space, but I think for kernel, those bits are always set? In which case, what you're doing is correct.

Quoted 25 lines
arch/arm64/mm/mmu.c +25 -0
+		return false;
+
+	return true;
+}
+
+static void __try_collapse_pmd(pmd_t *pmdp, pmd_t pmd, unsigned long addr)
+{
+	pte_t *ptep;
+	pte_t first_pte;
+	unsigned long pfn;
+	pgprot_t prot;
+	int i;
+
+	ptep = (pte_t *)pmd_page_vaddr(pmd);
+	first_pte = __ptep_get(ptep);
+
+	if (!pte_valid(first_pte))
+		return;
+
+	prot = pte_pgprot(first_pte);
+	prot = __pgprot(pgprot_val(prot) & ~PTE_CONT);
+	pfn = pte_pfn(first_pte);
+
+	if (!IS_ALIGNED(pfn, PMD_SIZE >> PAGE_SHIFT))
+		return;

arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 813

Quoted 10 lines
arch/arm64/mm/mmu.c +10 -0
+
+	for (i = 1; i < PTRS_PER_PTE; i++) {
+		if (!__pte_can_be_collapsed(__ptep_get(ptep + i), pfn + i, prot))
+			return;
+	}
+
+	set_pmd(pmdp, pmd_mkhuge(pfn_pmd(pfn, prot)));
+
+	__flush_tlb_kernel_pgtable(addr);
+

nit: suggest adding the same comment that other sites has, since this is not obvious:

/* See comment in pud_free_pmd_page for static key logic */

arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 819

Quoted 6 lines
arch/arm64/mm/mmu.c +6 -0
+	if (static_branch_unlikely(&arm64_ptdump_lock_key)) {
+		mmap_read_lock(&init_mm);
+		mmap_read_unlock(&init_mm);
+	}
+
+	pte_free_kernel(NULL, ptep);

Ooh, fun. Per my comments below we definitely have opportunity for racy pte table accesses (beyond ptdump). I _think_ if you fix that then this will be safe.

Quoted 7 lines
arch/arm64/mm/mmu.c +7 -0
+}
+
+void try_collapse_kernel_pmd(unsigned long addr)
+{
+	pgd_t *pgdp;
+	p4d_t *p4dp;
+	pud_t *pudp;

arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 836

Quoted 10 lines
arch/arm64/mm/mmu.c +10 -0
+	pmd_t *pmdp;
+	pmd_t pmd;
+
+	/*
+	 * collapse_pmd expects exact address of block to be collapsed
+	 */
+	if (WARN_ON(ALIGN_DOWN(addr, PMD_SIZE) != addr))
+		return;
+
+	mutex_lock(&pgtable_split_lock);

I don't think this is safe in general. Let's say we have a 2M region split into 512 x 4K PTEs. It's possible that the first 1M is one object and the second 1M is another object. Different CPUs could set_memory_*() on those 2 objects concurrently. If one of them then calls this function, we could end up collapsing the whole 2M while the other is trying to modify the PTEs and they will race.

Note that splitting _is_ safe (and protected by this lock) because you'd have 2 objects backed by the same PMD, so they would both have to split before modifying the PTEs.

I think you'd need to ensure mutual exclusion at a higher level if doing this; probably execmem is the place that can ensure that no objects within a 2M region are racily trying to modify their permissions?

arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 839

Quoted 3 lines
arch/arm64/mm/mmu.c +3 -0
+
+	pgdp = pgd_offset_k(addr);
+	if (pgd_none(READ_ONCE(*pgdp)))

nit: use the getters (e.g. pXdp_get()) instead of READ_ONCE().

Quoted 35 lines
arch/arm64/mm/mmu.c +27 -0
+		goto out;
+
+	p4dp = p4d_offset(pgdp, addr);
+	if (p4d_none(READ_ONCE(*p4dp)))
+		goto out;
+
+	pudp = pud_offset(p4dp, addr);
+	if (pud_none(READ_ONCE(*pudp)))
+		goto out;
+
+	if (pud_leaf(READ_ONCE(*pudp)))
+		goto out;
+
+	pmdp = pmd_offset(pudp, addr);
+	pmd = pmdp_get(pmdp);
+
+	if (!pmd_table(pmd))
+		goto out;
+
+	lazy_mmu_mode_enable();
+	__try_collapse_pmd(pmdp, pmd, addr);
+	lazy_mmu_mode_disable();
+
+out:
+	mutex_unlock(&pgtable_split_lock);
+}
+
 int split_kernel_leaf_mapping(unsigned long start, unsigned long end)
 {
 	int ret;
arch/arm64/mm/pageattr.c +0 -0
index eaefdf90b0d5..11e0b60264c3 100644

arch/arm64/mm/pageattr.c static int change_memory_common(unsigned long addr, int numpages, · line 202

Quoted 3 lines
arch/arm64/mm/pageattr.c +0 -0
@@ -200,6 +200,13 @@ static int change_memory_common(unsigned long addr, int numpages,
 			if (ret)
 				return ret;
 		}

The loop here in the unchanged code, calls __change_memory_common() for each PAGE_SIZE page in the linear alias. Perhaps it should be area->page_order aware? That way, if we are changing the permissions of the whole 2M chunk of vmalloc space and it's backed by a 2M page, then we won't split the mapping to PTEs in the linear map. Similarly, if we are changing permissions on a sub-region of the 2M area, we might be able to split only as far as contpte and still have some performance benefit?

arch/arm64/mm/pageattr.c static int change_memory_common(unsigned long addr, int numpages, · line 209

Quoted 7 lines
arch/arm64/mm/pageattr.c +7 -0
+		/*
+		 * When setting a read-only flag on the linear region, the memory
+		 * may have been backed by a PMD before being split. Try to
+		 * collapse it back into a PMD to restore huge page performance.
+		 */
+		if (pgprot_val(set_mask) == PTE_RDONLY && area->flags & VM_ALLOW_HUGE_VMAP)
+			try_collapse_kernel_pmd((u64)page_address(area->pages[0]));

try_collapse_kernel_pmd() requires a PMD-aligned address. VM_ALLOW_HUGE_VMAP doesn't guarrantee that - it only says that it's _allowed_ to be huge (and 64K would still be huge). vmalloc may have failed to allocate a PMD-sized page and reverted to something smaller (64K contpte or 4K). You need to look at area->page_order, I think.

You're only calling this for the linear alias. Don't you want to call it for the vmalloc range too? I assume the module text executes using the vmalloc address so you'd want it mapped by a single PMD for best performance?

But in general, I don't really like the idea of special-casing a collapse to pmd here for just execmem. I think we should either investigate a general purpose collapse mechanism or execmem should have extra hooks added to request specific collapse.

Thanks,
Ryan

Quoted 3 lines
arch/arm64/mm/pageattr.c +0 -0
 	}
 
 	/*
#24 lore raw

Re: [RFC PATCH 6/6] arm64: mm: support PMD page coalescing in the linear map

Quoted 10 lines · Ryan Roberts
Quoted 8 lines · Adrian Barnaś

Implement PMD block coalescing to merge fragmented linear mapping regions back into huge pages when restoring the read-only attribute.

When memory allocated with VM_ALLOW_HUGE_VMAP (such as for the execmem ROX cache) has its permissions modified, the PMD block mapping is split into individual PTEs. Without this change, when that memory have its RO attribute subsequently cleared and set the mapping remains permanently fragmented into 4K pages.

Quoted 10 lines

I'd like to make sure I understand this correctly: So the execmem ROX cache is allocated using vmalloc_huge such that all it's backing memory are PMD-sized pages. It is initially poisoned with a faulting instruction and marked ROX. When a module is loaded, it's text is copied into a portion of the ROX cache and executed from there? To do that, the required number of (4K) pages are allocated from ROX cache, and the permissions are changed on that sub-region to RW, meaning we have to split the mapping from a PMD to (cont)PTEs in both vmalloc space and in the linear map. After the text is copied, the sub-region is switched back to ROX. But without that change, the 2M region is now mapped by (cont)PTEs for all of time?

That is correct.

Quoted 170 lines
Quoted 26 lines
---
 arch/arm64/include/asm/mmu.h |  1 +
 arch/arm64/mm/mmu.c          | 95 ++++++++++++++++++++++++++++++++++++
 arch/arm64/mm/pageattr.c     |  7 +++
 3 files changed, 103 insertions(+)
arch/arm64/include/asm/mmu.h +1 -0
index 137a173df1ff..19158bacb2df 100644
@@ -80,6 +80,7 @@ extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot);
 extern void mark_linear_text_alias_ro(void);
 extern int split_kernel_leaf_mapping(unsigned long start, unsigned long end);
 extern void linear_map_maybe_split_to_ptes(void);
+void try_collapse_kernel_pmd(unsigned long addr);
 
 /*
  * This check is triggered during the early boot before the cpufeature
arch/arm64/mm/mmu.c +0 -0
index a6a00accf4f9..d74226fa1c9b 100644

arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 778

Quoted 10 lines
arch/arm64/mm/mmu.c +7 -0
@@ -769,6 +769,101 @@ static inline bool force_pte_mapping(void)
 
 static DEFINE_MUTEX(pgtable_split_lock);
 
+static inline bool __pte_can_be_collapsed(pte_t pte, unsigned long pfn, pgprot_t prot)
+{
+	if (!pte_valid(pte))
+		return false;
+	if (pte_pfn(pte) != pfn)
+		return false;
+	if ((pgprot_val(pte_pgprot(pte)) & ~PTE_CONT) != pgprot_val(prot))

Do we want to permit differing values for access and dirty? We do for user space, but I think for kernel, those bits are always set? In which case, what you're doing is correct.

Quoted 25 lines
arch/arm64/mm/mmu.c +25 -0
+		return false;
+
+	return true;
+}
+
+static void __try_collapse_pmd(pmd_t *pmdp, pmd_t pmd, unsigned long addr)
+{
+	pte_t *ptep;
+	pte_t first_pte;
+	unsigned long pfn;
+	pgprot_t prot;
+	int i;
+
+	ptep = (pte_t *)pmd_page_vaddr(pmd);
+	first_pte = __ptep_get(ptep);
+
+	if (!pte_valid(first_pte))
+		return;
+
+	prot = pte_pgprot(first_pte);
+	prot = __pgprot(pgprot_val(prot) & ~PTE_CONT);
+	pfn = pte_pfn(first_pte);
+
+	if (!IS_ALIGNED(pfn, PMD_SIZE >> PAGE_SHIFT))
+		return;

arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 813

Quoted 10 lines
arch/arm64/mm/mmu.c +10 -0
+
+	for (i = 1; i < PTRS_PER_PTE; i++) {
+		if (!__pte_can_be_collapsed(__ptep_get(ptep + i), pfn + i, prot))
+			return;
+	}
+
+	set_pmd(pmdp, pmd_mkhuge(pfn_pmd(pfn, prot)));
+
+	__flush_tlb_kernel_pgtable(addr);
+

nit: suggest adding the same comment that other sites has, since this is not obvious:

/* See comment in pud_free_pmd_page for static key logic */

arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 819

Quoted 6 lines
arch/arm64/mm/mmu.c +6 -0
+	if (static_branch_unlikely(&arm64_ptdump_lock_key)) {
+		mmap_read_lock(&init_mm);
+		mmap_read_unlock(&init_mm);
+	}
+
+	pte_free_kernel(NULL, ptep);

Ooh, fun. Per my comments below we definitely have opportunity for racy pte table accesses (beyond ptdump). I _think_ if you fix that then this will be safe.

Quoted 7 lines
arch/arm64/mm/mmu.c +7 -0
+}
+
+void try_collapse_kernel_pmd(unsigned long addr)
+{
+	pgd_t *pgdp;
+	p4d_t *p4dp;
+	pud_t *pudp;

arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 836

Quoted 10 lines
arch/arm64/mm/mmu.c +10 -0
+	pmd_t *pmdp;
+	pmd_t pmd;
+
+	/*
+	 * collapse_pmd expects exact address of block to be collapsed
+	 */
+	if (WARN_ON(ALIGN_DOWN(addr, PMD_SIZE) != addr))
+		return;
+
+	mutex_lock(&pgtable_split_lock);

I don't think this is safe in general. Let's say we have a 2M region split into 512 x 4K PTEs. It's possible that the first 1M is one object and the second 1M is another object. Different CPUs could set_memory_*() on those 2 objects concurrently. If one of them then calls this function, we could end up collapsing the whole 2M while the other is trying to modify the PTEs and they will race.

Note that splitting _is_ safe (and protected by this lock) because you'd have 2 objects backed by the same PMD, so they would both have to split before modifying the PTEs.

I think you'd need to ensure mutual exclusion at a higher level if doing this; probably execmem is the place that can ensure that no objects within a 2M region are racily trying to modify their permissions?

arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 839

Quoted 3 lines
arch/arm64/mm/mmu.c +3 -0
+
+	pgdp = pgd_offset_k(addr);
+	if (pgd_none(READ_ONCE(*pgdp)))

nit: use the getters (e.g. pXdp_get()) instead of READ_ONCE().

Quoted 35 lines
arch/arm64/mm/mmu.c +27 -0
+		goto out;
+
+	p4dp = p4d_offset(pgdp, addr);
+	if (p4d_none(READ_ONCE(*p4dp)))
+		goto out;
+
+	pudp = pud_offset(p4dp, addr);
+	if (pud_none(READ_ONCE(*pudp)))
+		goto out;
+
+	if (pud_leaf(READ_ONCE(*pudp)))
+		goto out;
+
+	pmdp = pmd_offset(pudp, addr);
+	pmd = pmdp_get(pmdp);
+
+	if (!pmd_table(pmd))
+		goto out;
+
+	lazy_mmu_mode_enable();
+	__try_collapse_pmd(pmdp, pmd, addr);
+	lazy_mmu_mode_disable();
+
+out:
+	mutex_unlock(&pgtable_split_lock);
+}
+
 int split_kernel_leaf_mapping(unsigned long start, unsigned long end)
 {
 	int ret;
arch/arm64/mm/pageattr.c +0 -0
index eaefdf90b0d5..11e0b60264c3 100644

arch/arm64/mm/pageattr.c static int change_memory_common(unsigned long addr, int numpages, · line 202

Quoted 3 lines
arch/arm64/mm/pageattr.c +0 -0
@@ -200,6 +200,13 @@ static int change_memory_common(unsigned long addr, int numpages,
 			if (ret)
 				return ret;
 		}
Quoted 8 lines

The loop here in the unchanged code, calls __change_memory_common() for each PAGE_SIZE page in the linear alias. Perhaps it should be area->page_order aware? That way, if we are changing the permissions of the whole 2M chunk of vmalloc space and it's backed by a 2M page, then we won't split the mapping to PTEs in the linear map. Similarly, if we are changing permissions on a sub-region of the 2M area, we might be able to split only as far as contpte and still have some performance benefit?

This is out of scope of execmem (as you noted we are mostly changing permision for only a part of the block) but your sugestion seems to be valid for other cases.

Quoted 5 lines
Quoted 5 lines
arch/arm64/mm/pageattr.c +5 -0
+		/*
+		 * When setting a read-only flag on the linear region, the memory
+		 * may have been backed by a PMD before being split. Try to
+		 * collapse it back into a PMD to restore huge page performance.
+		 */
Quoted 9 lines

arch/arm64/mm/pageattr.c static int change_memory_common(unsigned long addr, int numpages, · line 209

Quoted 2 lines
arch/arm64/mm/pageattr.c +2 -0
+		if (pgprot_val(set_mask) == PTE_RDONLY && area->flags & VM_ALLOW_HUGE_VMAP)
+			try_collapse_kernel_pmd((u64)page_address(area->pages[0]));

try_collapse_kernel_pmd() requires a PMD-aligned address. VM_ALLOW_HUGE_VMAP doesn't guarrantee that - it only says that it's _allowed_ to be huge (and 64K would still be huge). vmalloc may have failed to allocate a PMD-sized page and reverted to something smaller (64K contpte or 4K). You need to look at area->page_order, I think.

Agree.

Quoted 4 lines

You're only calling this for the linear alias. Don't you want to call it for the vmalloc range too? I assume the module text executes using the vmalloc address so you'd want it mapped by a single PMD for best performance?

I think it is a good idea to have them both (linear alias and vmalloc area) mapped as PMD.

Quoted 5 lines

But in general, I don't really like the idea of special-casing a collapse to pmd here for just execmem. I think we should either investigate a general purpose collapse mechanism or execmem should have extra hooks added to request specific collapse.>

Quoted 8 lines

Thanks,
Ryan

Quoted 3 lines
arch/arm64/mm/pageattr.c +0 -0
 	}
 
 	/*

I will rethink this based on your suggestion and try to properly fix the concurrency issues.

I am also wondering if we should avoid splitting linear aliases altogether. I think I understand why it was done originally (to restrict writing through
linear mapping aliases for read-only vmalloc areas). However, it is not necessary to make them writable when the vmalloc area is writable. Maybe we should create a special case for execmem to prevent this. WDYT?

Thank you for the review,
Adrian

#25 lore raw

Re: [RFC PATCH 6/6] arm64: mm: support PMD page coalescing in the linear map

In reply to Ryan Roberts

Hi Mike,

Quoted 11 lines · Ryan Roberts

I don't think this is safe in general. Let's say we have a 2M region split into 512 x 4K PTEs. It's possible that the first 1M is one object and the second 1M is another object. Different CPUs could set_memory_*() on those 2 objects concurrently. If one of them then calls this function, we could end up collapsing the whole 2M while the other is trying to modify the PTEs and they will race.

Note that splitting _is_ safe (and protected by this lock) because you'd have 2 objects backed by the same PMD, so they would both have to split before modifying the PTEs.

Quoted 6 lines

I think you'd need to ensure mutual exclusion at a higher level if doing this; probably execmem is the place that can ensure that no objects within a 2M region are racily trying to modify their permissions?

Thanks,
Ryan

I wonder what do you think about enforcing mutual exclusion for permission changes to the ROX_CACHE block as Ryan sugest?

Would it be appropriate to add locks (over &execmem_cache.mutex) to execmem_force_rw and execmem_restore_rox?

I see that x86 has a lock that wraps entire process of updating attributes (including spliting and collapsing pages):

```                                                                        
        spin_lock(&cpa_lock);                                              
        ret = __change_page_attr(cpa, primary);                            
        spin_unlock(&cpa_lock);                                            
```                                                                        As I understand it makes it safe already.                               

However, on ARM64 there is a split lock (which works fine for spliting) but for collapsing at least on my implementation it is not sufficient as Ryan pointed out.

Best regards
Adrian

#26 lore raw

Re: [RFC PATCH 6/6] arm64: mm: support PMD page coalescing in the linear map

Quoted 25 lines · Adrian Barnaś

Hi Mike,

Quoted 11 lines · Ryan Roberts

I don't think this is safe in general. Let's say we have a 2M region split into 512 x 4K PTEs. It's possible that the first 1M is one object and the second 1M is another object. Different CPUs could set_memory_*() on those 2 objects concurrently. If one of them then calls this function, we could end up collapsing the whole 2M while the other is trying to modify the PTEs and they will race.

Note that splitting _is_ safe (and protected by this lock) because you'd have 2 objects backed by the same PMD, so they would both have to split before modifying the PTEs.

Quoted 6 lines

I think you'd need to ensure mutual exclusion at a higher level if doing this; probably execmem is the place that can ensure that no objects within a 2M region are racily trying to modify their permissions?

Thanks,
Ryan

I wonder what do you think about enforcing mutual exclusion for permission changes to the ROX_CACHE block as Ryan sugest?

Quoted 13 lines

Would it be appropriate to add locks (over &execmem_cache.mutex) to execmem_force_rw and execmem_restore_rox?
I
see that x86 has a lock that wraps entire process of updating attributes (including spliting and collapsing pages):
```
spin_lock(&cpa_lock);
ret = __change_page_attr(cpa, primary);
spin_unlock(&cpa_lock); ```
As I understand it makes it safe already.
However, on ARM64 there is a split lock (which works fine for spliting) but for collapsing at least on my implementation it is not sufficient as Ryan pointed out.

I think the lock should be at set_memory level so that set_memory callers shouldn't care about the differences in arch implementations.

I also suspect that the lockless page table walk in update_range_prot() isn't safe today.

Quoted 2 lines

Best regards
Adrian

Signature · 2 lines
Sincerely yours,
Mike.