arm64: mm: Introducing ROX CACHE to ARM64 systems with bbml2 no abort
Subsystems (2)
cover arm64: mm: Introducing ROX CACHE to ARM64 systems with bbml2 no abort
Hi All,
I would like to propose the introduction of the EXECMEM_ROX_CACHE feature for ARM64.
When the rodata=on kernel parameter is set, all executable and read-only aliases in the linear map are forced to be read-only. Originally, this forced PTE-level mappings across the entire linear map, causing a significant performance regression (greater than 10%) for some workloads due to increased TLB miss rates and deeper page table walk.
The feat_bbml2_no_abort (FEAT_BBML3 in ARMv9.7) feature can be utilized to mitigate this regression. Because we can split memory mappings without triggering TLB conflict aborts, kernel memory permission adjustments become possible after early boot without forcing PTE-level mappings across the entire linear map.
However, when applying read-only permissions to kernel module section the linear map can still became fragmented due to the scattered physical layout of the underlying pages. To address this, EXECMEM_ROX_CACHE, which was initially enabled on the x86 architecture [1] for this purpose, can be used on ARM64 as well.
EXECMEM_ROX_CACHE works by preallocating PMD-sized contiguous blocks to act as a cache for .module.text memory. These blocks are initially poisoned and made read-only-execute (which simultaneously makes the linear alias of this region read-only). When loading a .module.text section into memory, the requested cache region is made RW, the bytes are copied, and ROX permissions are restored.
To take full advantage of this approach, after restoring RO permissions on the PMD-sized linear alias block, the PTE mappings are coalesced back into a single PMD entry.
Testing on an Android device running 6.18 based kernel with rodata=on shows an average 20% reduction of level 3 page table entries for the linear mapping.
This implementation currently works around some limitations of the set_memory_xx API, which might be relevant when considering the refactoring proposed here [2]:
* Because the execmem_cache operates outside the linear map, its permissions could theoretically remain untouched (poisoned and RO) until the cache block is fully emptied and freed. However, we currently lack an API to interact exclusively with the vmalloc area (e.g., setting it to RW) without simultaneously setting the linear alias to RW.
* Additionally, set_direct_map_valid() has somewhat confusing semantics. It is used while "cleaning" a cache block (after confirming it is entirely empty). The linear map region should be returned to its default state to restore writability, but set_valid might just set the "valid" attribute (as is the case for ARM64, which I have temporarily addressed here with a workaround).
I would be glad to hear your feedback on these changes.
Best regards,
Adrian
[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://lore.kernel.org/all/[email protected]/
--->8
Adrian Barnaś (6):
arm64: mm: explicitly declare module and ftrace execmem regions
arm64: mm: allow huge vmap permission adjustments with bbml2_no_abort
arm64: mm: fix restoring linear map permissions on execmem cache clean
arm64: mm: add helper to fill execmem with trapping instructions
arm64: execmem: enable EXECMEM_ROX_CACHE on supported CPUs
arm64: mm: support PMD page coalescing in the linear map
arch/arm64/Kconfig | 1 + arch/arm64/include/asm/mmu.h | 1 + arch/arm64/mm/init.c | 54 +++++++++++++++++++- arch/arm64/mm/mmu.c | 95 ++++++++++++++++++++++++++++++++++++ arch/arm64/mm/pageattr.c | 46 +++++++++++++---- 5 files changed, 186 insertions(+), 11 deletions(-)
--
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH 1/6 arm64: mm: explicitly declare module and ftrace execmem regions
Replace the reliance on the EXECMEM_DEFAULT fallback by explicitly defining the execution memory (execmem) regions for MODULE_TEXT, MODULE_DATA, and FTRACE in execmem_arch_setup().
--- arch/arm64/mm/init.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
arch/arm64/mm/init.c +17 -1
@@ -519,7 +519,7 @@ struct execmem_info __init *execmem_arch_setup(void) execmem_info = (struct execmem_info ){ .ranges = { - [EXECMEM_DEFAULT ] = { + [EXECMEM_MODULE_TEXT ] = { .start = start, .end = end, .pgprot =PAGE_KERNEL ,
@@ -533,12 +533,28 @@ struct execmem_info __init *execmem_arch_setup(void) .pgprot = PAGE_KERNEL_ROX, .alignment = 1, }, + [EXECMEM_FTRACE] = { + .start = VMALLOC_START, + .end = VMALLOC_END, + .pgprot = PAGE_KERNEL, + .alignment = 1, + .fallback_start = fallback_start, + .fallback_end = fallback_end, + }, [EXECMEM_BPF] = { .start = VMALLOC_START, .end = VMALLOC_END, .pgprot = PAGE_KERNEL, .alignment = 1, }, + [EXECMEM_MODULE_DATA] = { + .start = start, + .end = end, + .pgprot = PAGE_KERNEL, + .alignment = 1, + .fallback_start = fallback_start, + .fallback_end = fallback_end, + }, }, };
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Re: [RFC PATCH 1/6] arm64: mm: explicitly declare module and ftrace execmem regions
Quoted 3 lines · On Thu Jun 11, 2026 at 1:01 PM UTC, =?UTF-8?q?Adrian=20Barna=C5=9B?= wrote:
Replace the reliance on the
EXECMEM_DEFAULTfallback by explicitly defining the execution memory (execmem) regions forMODULE_TEXT,MODULE_DATA, and FTRACE inexecmem_arch_setup().
Please can you explain in the commit message _why_ you do this. This way reviewers don't have to make a mental note and come back later after reading the rest of the patchset. And once it's commited, it will save readers from having to chase down the contextual commits to figure out what's going on.
RFC PATCH 2/6 arm64: mm: allow huge vmap permission adjustments with bbml2_no_abort
In reply to Adrian BarnaśRemove the protection against huge vmap permission adjustments on systems that support the bbml2_no_abort CPU feature.
Splitting live kernel VA section mappings into page mappings was restricted because it could cause TLB Conflict Aborts. This forced permission adjustments on memory allocated with VM_ALLOW_HUGE_VMAP to be rejected, resulting in performance drops (e.g., when enforcing rodata=on disables huge mappings).
The bbml2_no_abort feature (which mirrors the architectural guarantees of FEAT_BBML3) ensures that changing between table and block sizes without following a break-before-make sequence will not generate a TLB Conflict Abort. This hardware guarantee makes it safe to allow dynamic permission adjustments on huge vmap regions.
--- arch/arm64/mm/pageattr.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
arch/arm64/mm/pageattr.c +14 -8
@@ -157,23 +157,29 @@ static int change_memory_common(unsigned long addr, int numpages, }/* -* Kernel VA mappings are always live, and splitting live section -* mappings into page mappings may cause TLB conflicts. This means -* we have to ensure that changing the permission bits of the range -* we are operating on does not result in such splitting. -* * Let's restrict ourselves to mappings created by vmalloc (or vmap). -* Disallow VM_ALLOW_HUGE_VMAP mappings to guarantee that only page -* mappings are updated and splitting is never needed. * * So check whether the [addr, addr + size) interval is entirely * covered by precisely one VM area that has the VM_ALLOC flag set. */ area =find_vm_area ((void *)addr); +if (!area || ((unsigned long )kasan_reset_tag ((void *)end) > (unsigned long )kasan_reset_tag (area->addr) + area->size) || - ((area->flags & (VM_ALLOC |VM_ALLOW_HUGE_VMAP )) !=VM_ALLOC )) + !(area->flags &VM_ALLOC )) +return -EINVAL ; + +/* +* Kernel VA mappings are always live, and splitting live section +* mappings into page mappings may cause TLB conflicts if bbml2_noabort +* is not present. +* +* While bbml2_noabort is not present disallow VM_ALLOW_HUGE_VMAP mappings +* to guarantee that only page mappings are updated and splitting is not +* needed. +*/ +if (!system_supports_bbml2_noabort () && (area->flags & (VM_ALLOW_HUGE_VMAP )))return -EINVAL ;if (!numpages)
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Re: [RFC PATCH 2/6] arm64: mm: allow huge vmap permission adjustments with bbml2_no_abort
Quoted 9 lines · Adrian Barnaś
Remove the protection against huge vmap permission adjustments on systems that support the
bbml2_no_abortCPU feature.Splitting live kernel VA section mappings into page mappings was restricted because it could cause TLB Conflict Aborts. This forced permission adjustments on memory allocated with
VM_ALLOW_HUGE_VMAPto be rejected, resulting in performance drops (e.g., when enforcing rodata=on disables huge mappings).
Quoted 5 lines
The
bbml2_no_abortfeature (which mirrors the architectural guarantees ofFEAT_BBML3) ensures that changing between table and block sizes without following a break-before-make sequence will not generate a TLB Conflict Abort. This hardware guarantee makes it safe to allow dynamic permission adjustments on huge vmap regions.
FYI Linu Cherian has a series that renames bbml2_no_abort to bbml3. I think he's planning to post at -rc1. Would be good to rebase this on top once merged.
Quoted 35 lines
Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/mm/pageattr.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)arch/arm64/mm/pageattr.c +4
-8@@ -157,23 +157,29 @@ static int change_memory_common(unsigned long addr, int numpages, }/* -* Kernel VA mappings are always live, and splitting live section -* mappings into page mappings may cause TLB conflicts. This means -* we have to ensure that changing the permission bits of the range -* we are operating on does not result in such splitting. -* * Let's restrict ourselves to mappings created by vmalloc (or vmap). -* Disallow VM_ALLOW_HUGE_VMAP mappings to guarantee that only page -* mappings are updated and splitting is never needed. * * So check whether the [addr, addr + size) interval is entirely * covered by precisely one VM area that has the VM_ALLOC flag set. */ area =find_vm_area ((void *)addr); +if (!area || ((unsigned long )kasan_reset_tag ((void *)end) > (unsigned long )kasan_reset_tag (area->addr) + area->size) || - ((area->flags & (VM_ALLOC |VM_ALLOW_HUGE_VMAP )) !=VM_ALLOC )) + !(area->flags &VM_ALLOC )) +return -EINVAL ; +
arch/arm64/mm/pageattr.c static int change_memory_common(unsigned long addr, int numpages, · line 182
Quoted 10 lines
arch/arm64/mm/pageattr.c +10
-0+/* +* Kernel VA mappings are always live, and splitting live section +* mappings into page mappings may cause TLB conflicts if bbml2_noabort +* is not present. +* +* While bbml2_noabort is not present disallow VM_ALLOW_HUGE_VMAP mappings +* to guarantee that only page mappings are updated and splitting is not +* needed. +*/ +if (!system_supports_bbml2_noabort () && (area->flags & (VM_ALLOW_HUGE_VMAP )))
nit: no need for the parentheses around VM_ALLOW_HUGE_VMAP.
With that:
Quoted 3 lines
arch/arm64/mm/pageattr.c +0
-0return -EINVAL ;if (!numpages)
Re: [RFC PATCH 2/6] arm64: mm: allow huge vmap permission adjustments with bbml2_no_abort
Quoted 10 lines · Ryan Roberts
Quoted 9 lines · Adrian Barnaś
Remove the protection against huge vmap permission adjustments on systems that support the
bbml2_no_abortCPU feature.Splitting live kernel VA section mappings into page mappings was restricted because it could cause TLB Conflict Aborts. This forced permission adjustments on memory allocated with
VM_ALLOW_HUGE_VMAPto be rejected, resulting in performance drops (e.g., when enforcing rodata=on disables huge mappings).
Quoted 9 lines
Quoted 5 lines
The
bbml2_no_abortfeature (which mirrors the architectural guarantees ofFEAT_BBML3) ensures that changing between table and block sizes without following a break-before-make sequence will not generate a TLB Conflict Abort. This hardware guarantee makes it safe to allow dynamic permission adjustments on huge vmap regions.FYI Linu Cherian has a series that renames
bbml2_no_abortto bbml3. I think he's planning to post at -rc1. Would be good to rebase this on top once merged.
I will keep an eye on next releases.
Quoted 48 lines
Quoted 35 lines
Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/mm/pageattr.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)arch/arm64/mm/pageattr.c +4
-8@@ -157,23 +157,29 @@ static int change_memory_common(unsigned long addr, int numpages, }/* -* Kernel VA mappings are always live, and splitting live section -* mappings into page mappings may cause TLB conflicts. This means -* we have to ensure that changing the permission bits of the range -* we are operating on does not result in such splitting. -* * Let's restrict ourselves to mappings created by vmalloc (or vmap). -* Disallow VM_ALLOW_HUGE_VMAP mappings to guarantee that only page -* mappings are updated and splitting is never needed. * * So check whether the [addr, addr + size) interval is entirely * covered by precisely one VM area that has the VM_ALLOC flag set. */ area =find_vm_area ((void *)addr); +if (!area || ((unsigned long )kasan_reset_tag ((void *)end) > (unsigned long )kasan_reset_tag (area->addr) + area->size) || - ((area->flags & (VM_ALLOC |VM_ALLOW_HUGE_VMAP )) !=VM_ALLOC )) + !(area->flags &VM_ALLOC )) +return -EINVAL ; +arch/arm64/mm/pageattr.c static int change_memory_common(unsigned long addr, int numpages, · line 182
Quoted 10 lines
arch/arm64/mm/pageattr.c +10
-0+/* +* Kernel VA mappings are always live, and splitting live section +* mappings into page mappings may cause TLB conflicts if bbml2_noabort +* is not present. +* +* While bbml2_noabort is not present disallow VM_ALLOW_HUGE_VMAP mappings +* to guarantee that only page mappings are updated and splitting is not +* needed. +*/ +if (!system_supports_bbml2_noabort () && (area->flags & (VM_ALLOW_HUGE_VMAP )))nit: no need for the parentheses around
VM_ALLOW_HUGE_VMAP.
Quoted 8 lines
With that:
Reviewed-by Ryan Roberts <[email protected]>Quoted 3 lines
arch/arm64/mm/pageattr.c +0
-0return -EINVAL ;if (!numpages)
Thanks,
Adrian
RFC PATCH 3/6 arm64: mm: fix restoring linear map permissions on execmem cache clean
In reply to Adrian BarnaśStrip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.
An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.
Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.
--- arch/arm64/mm/pageattr.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
arch/arm64/mm/pageattr.c +16 -1
@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages) __pgprot(PTE_PXN)); } +static int set_memory_default (unsigned long addr,int numpages) +{ +return __change_memory_common (addr,PAGE_SIZE * numpages, +__pgprot (PTE_VALID ), +__pgprot (PTE_RDONLY )); +} +int set_memory_valid (unsigned long addr,int numpages,int enable) {if (enable)
@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)if (!can_set_direct_map ())return 0 ; -return set_memory_valid (addr, nr, valid); +/* +* Execmem cache uses this function to reset permissions on linear mapping +* when freeing unused cache block. On x86 it makes memory RW which is +* desirable. On ARM64 set_memory_valid() just change valid bit which +* leave direct mapping read-only so use set_memory_default instead. +*/ + +return valid ?set_memory_default (addr, nr) : +set_memory_valid (addr, nr, false); }#ifdef CONFIG_DEBUG_PAGEALLOC
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean
Quoted 35 lines · On Thu Jun 11, 2026 at 1:01 PM UTC, =?UTF-8?q?Adrian=20Barna=C5=9B?= wrote:
Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.
An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.
Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.
Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/mm/pageattr.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)arch/arm64/mm/pageattr.c +7
-0@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages) __pgprot(PTE_PXN)); } +static int set_memory_default (unsigned long addr,int numpages) +{ +return __change_memory_common (addr,PAGE_SIZE * numpages, +__pgprot (PTE_VALID ), +__pgprot (PTE_RDONLY )); +} +int set_memory_valid (unsigned long addr,int numpages,int enable) {if (enable)
arch/arm64/mm/pageattr.c int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) · line 375
Quoted 8 lines
arch/arm64/mm/pageattr.c +4
-1@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)if (!can_set_direct_map ())return 0 ; -return set_memory_valid (addr, nr, valid); +/* +* Execmem cache uses this function to reset permissions on linear mapping +* when freeing unused cache block. On x86 it makes memory RW which is +* desirable.
Hm, maybe desirable for execmem but that doesn't really mean the x86 behaviour is correct. Maybe it makes more sense to change the x86 to align with the arm64 behaviour here?
BTW we should probably document this API a little bit, I never thought abut what "valid" actually means until now. I had thought of it as "I can access this memory" but that's an unclear concept and now I realise "valid" is a technical concept in Arm that's confusing. And it's extra confusing if the kernel API uses "valid" to mean a _different_ thing.
Also, shouldn't execmem be using set_memory_default_noflush() before freeing anyway? I guess that shoudl even be documented as "if you change anything you need to call this before you free the page".
Quoted 9 lines
On ARM64
set_memory_valid()just change valid bit whicharch/arm64/mm/pageattr.c +5
-0+* leave direct mapping read-only so use set_memory_default instead. +*/ + +return valid ?set_memory_default (addr, nr) : +set_memory_valid (addr, nr, false); }#ifdef CONFIG_DEBUG_PAGEALLOC
Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean
Quoted 49 lines · Brendan Jackman
Quoted 35 lines · On Thu Jun 11, 2026 at 1:01 PM UTC, =?UTF-8?q?Adrian=20Barna=C5=9B?= wrote:
Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.
An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.
Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.
Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/mm/pageattr.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)arch/arm64/mm/pageattr.c +7
-0@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages) __pgprot(PTE_PXN)); } +static int set_memory_default (unsigned long addr,int numpages) +{ +return __change_memory_common (addr,PAGE_SIZE * numpages, +__pgprot (PTE_VALID ), +__pgprot (PTE_RDONLY )); +} +int set_memory_valid (unsigned long addr,int numpages,int enable) {if (enable)arch/arm64/mm/pageattr.c int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) · line 375
Quoted 8 lines
arch/arm64/mm/pageattr.c +4
-1@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)if (!can_set_direct_map ())return 0 ; -return set_memory_valid (addr, nr, valid); +/* +* Execmem cache uses this function to reset permissions on linear mapping +* when freeing unused cache block. On x86 it makes memory RW which is +* desirable. Hm, maybe desirable for execmem but that doesn't really mean the x86 behaviour is correct. Maybe it makes more sense to change the x86 to align with the arm64 behaviour here?
Quoted 5 lines
BTW we should probably document this API a little bit, I never thought abut what "valid" actually means until now. I had thought of it as "I can access this memory" but that's an unclear concept and now I realise "valid" is a technical concept in Arm that's confusing. And it's extra confusing if the kernel API uses "valid" to mean a _different_ thing.
I've got confused too and that's how set_direct_map_valid() got into x86 with a different semantics than on arm64.
What execmem really needs is set_direct_map_default() variant that gets nr_pages.
AFAIR, set_direct_map_default() has a single 'page' parameter because it was added to reset permissions for the direct map alias for vmalloc()'ed pages before there was VMALLOC_HUGE and each page had to be reset independently anyway.
Maybe it's time to add nr_pages to set_direct_map_valid().
Quoted 3 lines
Also, shouldn't execmem be using
set_memory_default_noflush()before freeing anyway? I guess that shoudl even be documented as "if you change anything you need to call this before you free the page".
It does on x86 because there set_direct_map_valid() is the same as set_direct_map_default().
Quoted 11 lines
Quoted 9 lines
On ARM64
set_memory_valid()just change valid bit whicharch/arm64/mm/pageattr.c +5
-0+* leave direct mapping read-only so use set_memory_default instead. +*/ + +return valid ?set_memory_default (addr, nr) : +set_memory_valid (addr, nr, false); }#ifdef CONFIG_DEBUG_PAGEALLOC
Signature · 2 lines
Sincerely yours, Mike.
Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean
Quoted 84 lines · Mike Rapoport
Quoted 49 lines · Brendan Jackman
Quoted 35 lines · On Thu Jun 11, 2026 at 1:01 PM UTC, =?UTF-8?q?Adrian=20Barna=C5=9B?= wrote:
Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.
An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.
Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.
Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/mm/pageattr.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)arch/arm64/mm/pageattr.c +7
-0@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages) __pgprot(PTE_PXN)); } +static int set_memory_default (unsigned long addr,int numpages) +{ +return __change_memory_common (addr,PAGE_SIZE * numpages, +__pgprot (PTE_VALID ), +__pgprot (PTE_RDONLY )); +} +int set_memory_valid (unsigned long addr,int numpages,int enable) {if (enable)arch/arm64/mm/pageattr.c int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) · line 375
Quoted 8 lines
arch/arm64/mm/pageattr.c +4
-1@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)if (!can_set_direct_map ())return 0 ; -return set_memory_valid (addr, nr, valid); +/* +* Execmem cache uses this function to reset permissions on linear mapping +* when freeing unused cache block. On x86 it makes memory RW which is +* desirable. Hm, maybe desirable for execmem but that doesn't really mean the x86 behaviour is correct. Maybe it makes more sense to change the x86 to align with the arm64 behaviour here?
Quoted 5 lines
BTW we should probably document this API a little bit, I never thought abut what "valid" actually means until now. I had thought of it as "I can access this memory" but that's an unclear concept and now I realise "valid" is a technical concept in Arm that's confusing. And it's extra confusing if the kernel API uses "valid" to mean a _different_ thing.
I've got confused too and that's how
set_direct_map_valid()got into x86 with a different semantics than on arm64.What execmem really needs is
set_direct_map_default()variant that getsnr_pages.AFAIR,
set_direct_map_default()has a single 'page' parameter because it was added to reset permissions for the direct map alias for vmalloc()'ed pages before there wasVMALLOC_HUGEand each page had to be reset independently anyway.Maybe it's time to add
nr_pagestoset_direct_map_valid().Quoted 3 lines
Also, shouldn't execmem be using
set_memory_default_noflush()before freeing anyway? I guess that shoudl even be documented as "if you change anything you need to call this before you free the page".It does on x86 because there
set_direct_map_valid()is the same asset_direct_map_default().Quoted 8 lines
Quoted 8 lines
On ARM64
set_memory_valid()just change valid bit whicharch/arm64/mm/pageattr.c +5
-0+* leave direct mapping read-only so use set_memory_default instead. +*/ + +return valid ?set_memory_default (addr, nr) : +set_memory_valid (addr, nr, false); }
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
#ifdef
CONFIG_DEBUG_PAGEALLOCSignature · 2 lines
Sincerely yours, Mike.
Hi Mike, Brendan,
Thanks for taking a look at the RFC.
I was also quite confused by this initially. I spent some time debugging until I realized why unloading all the modules was causing the kernel to crash.
The reason I took this approach was that I wanted to send out a working prototype for arm64 that wouldn't interfere with the existing, working implementation on x86.
Following your suggestion, I can put together a preparatory patch series to refactor the set_direct_map_* APIs to accept a nr_pages parameter. This refactoring would also allow us to drop the redundant set_area_direct_map helper. I could then rebase the rox_cache series on top of that.
Does this sound like a good path forward?
Thanks,
Adrian
Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean
Hi Adrian,
Quoted 29 lines · Adrian Barnaś
Quoted 17 lines · Mike Rapoport
Quoted 10 lines
Hm, maybe desirable for execmem but that doesn't really mean the x86 behaviour is correct. Maybe it makes more sense to change the x86 to align with the arm64 behaviour here?
BTW we should probably document this API a little bit, I never thought abut what "valid" actually means until now. I had thought of it as "I can access this memory" but that's an unclear concept and now I realise "valid" is a technical concept in Arm that's confusing. And it's extra confusing if the kernel API uses "valid" to mean a _different_ thing.
I've got confused too and that's how
set_direct_map_valid()got into x86 with a different semantics than on arm64.What execmem really needs is
set_direct_map_default()variant that getsnr_pages.Quoted 6 lines
AFAIR,
set_direct_map_default()has a single 'page' parameter because it was added to reset permissions for the direct map alias for vmalloc()'ed pages before there wasVMALLOC_HUGEand each page had to be reset independently anyway.Maybe it's time to add
nr_pagestoset_direct_map_valid().I was also quite confused by this initially. I spent some time debugging until I realized why unloading all the modules was causing the kernel to crash.
Quoted 6 lines
The reason I took this approach was that I wanted to send out a working prototype for arm64 that wouldn't interfere with the existing, working implementation on x86.
Following your suggestion, I can put together a preparatory patch series to refactor the set_direct_map_* APIs to accept a
nr_pagesparameter. This
There was a patch Nikita sent a while ago that does something similar:
https://lore.kernel.org/all/[email protected]
I believe you can start from there.
Quoted 1 line
refactoring would also allow us to drop the redundant
set_area_direct_map
We can't drop set_area_direct_map() because vmalloc pages might be not physically contiguous.
Quoted 6 lines
helper. I could then rebase the
rox_cacheseries on top of that.Does this sound like a good path forward?
Thanks,
Adrian
Signature · 2 lines
Sincerely yours, Mike.
Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean
Quoted 36 lines · Mike Rapoport
Hi Adrian,
Quoted 33 lines · Adrian Barnaś
Quoted 17 lines · Mike Rapoport
Quoted 10 lines
Hm, maybe desirable for execmem but that doesn't really mean the x86 behaviour is correct. Maybe it makes more sense to change the x86 to align with the arm64 behaviour here?
BTW we should probably document this API a little bit, I never thought abut what "valid" actually means until now. I had thought of it as "I can access this memory" but that's an unclear concept and now I realise "valid" is a technical concept in Arm that's confusing. And it's extra confusing if the kernel API uses "valid" to mean a _different_ thing.
I've got confused too and that's how
set_direct_map_valid()got into x86 with a different semantics than on arm64.What execmem really needs is
set_direct_map_default()variant that getsnr_pages.Quoted 6 lines
AFAIR,
set_direct_map_default()has a single 'page' parameter because it was added to reset permissions for the direct map alias for vmalloc()'ed pages before there wasVMALLOC_HUGEand each page had to be reset independently anyway.Maybe it's time to add
nr_pagestoset_direct_map_valid().I was also quite confused by this initially. I spent some time debugging until I realized why unloading all the modules was causing the kernel to crash.
The reason I took this approach was that I wanted to send out a working prototype for arm64 that wouldn't interfere with the existing, working implementation on x86.
Quoted 9 lines
Quoted 2 lines
Following your suggestion, I can put together a preparatory patch series to refactor the set_direct_map_* APIs to accept a
nr_pagesparameter. ThisThere was a patch Nikita sent a while ago that does something similar:
https://lore.kernel.org/all/[email protected]
I believe you can start from there.
I will pick the 01 patch from there to my series.
Quoted 5 lines
Quoted 1 line
refactoring would also allow us to drop the redundant
set_area_direct_mapWe can't drop
set_area_direct_map()because vmalloc pages might be not physically contiguous.
Agree.
Quoted 10 lines
Quoted 6 lines
helper. I could then rebase the
rox_cacheseries on top of that.Does this sound like a good path forward?
Thanks,
AdrianSignature · 2 lines
Sincerely yours, Mike.
Thanks,
Adrian
Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean
In reply to Adrian BarnaśHi Adrian,
Quoted 5 lines · Adrian Barnaś
Following your suggestion, I can put together a preparatory patch series to refactor the set_direct_map_* APIs to accept a
nr_pagesparameter. This refactoring would also allow us to drop the redundantset_area_direct_maphelper. I could then rebase therox_cacheseries on top of that.
I've been looking at something related and I was already half way through adding numpages to set_direct_map when I remembered this conversation :)
Here's what I have for now:
https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=direct-map/v0
This is only lightly build tested for arm64 and x86.
As this is WIP, things may change there, but I hope it can save you some time with your arm64 ROX cache work.
Quoted 4 lines
Does this sound like a good path forward?
Thanks,
Adrian
Signature · 2 lines
Sincerely yours, Mike.
Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean
Quoted 9 lines · Mike Rapoport
Hi Adrian,
Quoted 5 lines · Adrian Barnaś
Following your suggestion, I can put together a preparatory patch series to refactor the set_direct_map_* APIs to accept a
nr_pagesparameter. This refactoring would also allow us to drop the redundantset_area_direct_maphelper. I could then rebase therox_cacheseries on top of that.
Quoted 7 lines
I've been looking at something related and I was already half way through adding numpages to
set_direct_mapwhen I remembered this conversation :)Here's what I have for now:
https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=direct-map/v0
I did roughly the same on my local tree.
Quoted 3 lines
This is only lightly build tested for arm64 and x86.
As this is WIP, things may change there, but I hope it can save you some time with your arm64 ROX cache work.
I will fetch your commits and I will keep an eye on your tree if anything changes.
Thanks,
Adrian
Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean
In reply to Adrian BarnaśQuoted 21 lines · Adrian Barnaś
Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.
An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.
Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.
Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/mm/pageattr.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)arch/arm64/mm/pageattr.c +0
-0
arch/arm64/mm/pageattr.c int set_memory_x(unsigned long addr, int numpages) · line 246
Quoted 8 lines
arch/arm64/mm/pageattr.c +5
-0@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages) __pgprot(PTE_PXN)); } +static int set_memory_default (unsigned long addr,int numpages) +{ +return __change_memory_common (addr,PAGE_SIZE * numpages, +__pgprot (PTE_VALID ), +__pgprot (PTE_RDONLY ));
This is not sufficient to convert an invalid entry to valid. As well as setting the PTE_VALID bit, you would also need to clear the PTE_PRESENT_INVALID and set PTE_MAYBE_NG.
e.g:
int set_memory_valid(unsigned long addr, int numpages, int enable)
{
if (enable)
return __change_memory_common(addr, PAGE_SIZE * numpages,
__pgprot(PTE_PRESENT_VALID_KERNEL),
__pgprot(PTE_PRESENT_INVALID));Quoted 9 lines
arch/arm64/mm/pageattr.c +2
-0+} +int set_memory_valid (unsigned long addr,int numpages,int enable) {if (enable)@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)if (!can_set_direct_map ())return 0 ;
arch/arm64/mm/pageattr.c int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) · line 380
Quoted 10 lines
arch/arm64/mm/pageattr.c +9
-1-return set_memory_valid (addr, nr, valid); +/* +* Execmem cache uses this function to reset permissions on linear mapping +* when freeing unused cache block. On x86 it makes memory RW which is +* desirable. On ARM64 set_memory_valid() just change valid bit which +* leave direct mapping read-only so use set_memory_default instead. +*/ + +return valid ?set_memory_default (addr, nr) : +set_memory_valid (addr, nr, false);
Surely execmem should just be using set_direct_map_default_noflush() if that's the behaviour it wants?
I think that the current implementation of set_direct_map_default_noflush() doesn't undo the effects of set_memory_nx() / set_memory_x(). That might be worth checking?
Thanks,
Ryan
Quoted 3 lines
arch/arm64/mm/pageattr.c +0
-0}#ifdef CONFIG_DEBUG_PAGEALLOC
Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean
Quoted 68 lines · Ryan Roberts
Quoted 21 lines · Adrian Barnaś
Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.
An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.
Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.
Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/mm/pageattr.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)arch/arm64/mm/pageattr.c +0
-0arch/arm64/mm/pageattr.c int set_memory_x(unsigned long addr, int numpages) · line 246
Quoted 8 lines
arch/arm64/mm/pageattr.c +5
-0@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages) __pgprot(PTE_PXN)); } +static int set_memory_default (unsigned long addr,int numpages) +{ +return __change_memory_common (addr,PAGE_SIZE * numpages, +__pgprot (PTE_VALID ), +__pgprot (PTE_RDONLY ));This is not sufficient to convert an invalid entry to valid. As well as setting the
PTE_VALIDbit, you would also need to clear thePTE_PRESENT_INVALIDand setPTE_MAYBE_NG.e.g:
int
set_memory_valid(unsigned long addr, int numpages, int enable){ if (enable) return __change_memory_common(addr, PAGE_SIZE * numpages, __pgprot(PTE_PRESENT_VALID_KERNEL), __pgprot(PTE_PRESENT_INVALID));Quoted 9 lines
arch/arm64/mm/pageattr.c +2
-0+} +int set_memory_valid (unsigned long addr,int numpages,int enable) {if (enable)@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)if (!can_set_direct_map ())return 0 ;arch/arm64/mm/pageattr.c int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) · line 380
Quoted 10 lines
arch/arm64/mm/pageattr.c +9
-1-return set_memory_valid (addr, nr, valid); +/* +* Execmem cache uses this function to reset permissions on linear mapping +* when freeing unused cache block. On x86 it makes memory RW which is +* desirable. On ARM64 set_memory_valid() just change valid bit which +* leave direct mapping read-only so use set_memory_default instead. +*/ + +return valid ?set_memory_default (addr, nr) : +set_memory_valid (addr, nr, false);Surely execmem should just be using
set_direct_map_default_noflush()if that's the behaviour it wants?
Quoted 3 lines
I think that the current implementation of
set_direct_map_default_noflush()doesn't undo the effects ofset_memory_nx()/set_memory_x(). That might be worth checking?
It's also worth mentioning that set_direct_map_valid_noflush() has "noflush" in the name, implies it doesn't expect/require any TLB flushing to occur. But the implementation will perform tlb flushing for any case that is not just a invalid->valid transition (which for the existing impl is the case when valid=true and for your changes is never the case - see __change_memory_common).
But execmem doesn't do any tlb flushing so it looks to me like it actually requires that set_direct_map_valid_noflush() handles the tlb flushing? All seems a bit fishy and probably warrants a cleanup to make things clearer.
Quoted 9 lines
Thanks,
RyanQuoted 3 lines
arch/arm64/mm/pageattr.c +0
-0}#ifdef CONFIG_DEBUG_PAGEALLOC
Re: [RFC PATCH 3/6] arm64: mm: fix restoring linear map permissions on execmem cache clean
Quoted 38 lines · Ryan Roberts
Quoted 37 lines · Ryan Roberts
Quoted 21 lines · Adrian Barnaś
Strip the read-only attribute from the selected memory range when restoring the linear map after an execmem cache clean.
An execmem cache clean is performed when a cache block becomes empty after unloading a module. When making the memory valid again, the linear memory alias must also have its read-only attribute cleared.
Without this change, the linear memory alias remains read-only even after the execmem cache block itself is freed, which prevents subsequent allocations from writing to that memory.
Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/mm/pageattr.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)arch/arm64/mm/pageattr.c +0
-0arch/arm64/mm/pageattr.c int set_memory_x(unsigned long addr, int numpages) · line 246
Quoted 8 lines
arch/arm64/mm/pageattr.c +5
-0@@ -239,6 +239,13 @@ int set_memory_x(unsigned long addr, int numpages) __pgprot(PTE_PXN)); } +static int set_memory_default (unsigned long addr,int numpages) +{ +return __change_memory_common (addr,PAGE_SIZE * numpages, +__pgprot (PTE_VALID ), +__pgprot (PTE_RDONLY ));This is not sufficient to convert an invalid entry to valid. As well as setting the
PTE_VALIDbit, you would also need to clear thePTE_PRESENT_INVALIDand setPTE_MAYBE_NG.e.g:
Quoted 7 lines
Quoted 7 lines
int
set_memory_valid(unsigned long addr, int numpages, int enable){ if (enable) return __change_memory_common(addr, PAGE_SIZE * numpages, __pgprot(PTE_PRESENT_VALID_KERNEL), __pgprot(PTE_PRESENT_INVALID));
Thanks, I will fix that
Quoted 34 lines
Quoted 24 lines
Quoted 10 lines
arch/arm64/mm/pageattr.c +2
-0+} +int set_memory_valid (unsigned long addr,int numpages,int enable) {if (enable)@@ -362,7 +369,15 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)if (!can_set_direct_map ())return 0 ;arch/arm64/mm/pageattr.c int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) · line 380
Quoted 10 lines
arch/arm64/mm/pageattr.c +9
-1-return set_memory_valid (addr, nr, valid); +/* +* Execmem cache uses this function to reset permissions on linear mapping +* when freeing unused cache block. On x86 it makes memory RW which is +* desirable. On ARM64 set_memory_valid() just change valid bit which +* leave direct mapping read-only so use set_memory_default instead. +*/ + +return valid ?set_memory_default (addr, nr) : +set_memory_valid (addr, nr, false);Surely execmem should just be using
set_direct_map_default_noflush()if that's the behaviour it wants?Quoted 3 lines
I think that the current implementation of
set_direct_map_default_noflush()doesn't undo the effects ofset_memory_nx()/set_memory_x(). That might be worth checking?It's also worth mentioning that
set_direct_map_valid_noflush()has "noflush" in the name, implies it doesn't expect/require any TLB flushing to occur. But the implementation will perform tlb flushing for any case that is not just ainvalid->validtransition (which for the existing impl is the case when valid=true and for your changes is never the case - see__change_memory_common).
Quoted 4 lines
But execmem doesn't do any tlb flushing so it looks to me like it actually requires that
set_direct_map_valid_noflush()handles the tlb flushing? All seems a bit fishy and probably warrants a cleanup to make things clearer.
I think that the clean approach would be to have the set_direct_map_default function (without noflush) that does flushing as needed (like the current one on ARM64). I am not entirely sure but x86 seems to handle permission faults gracefully while on ARM64 those would cause panics. Is that correct?
Quoted 10 lines
Quoted 9 lines
Thanks,
RyanQuoted 3 lines
arch/arm64/mm/pageattr.c +0
-0}#ifdef CONFIG_DEBUG_PAGEALLOC
Thanks,
Adrian
RFC PATCH 4/6 arm64: mm: add helper to fill execmem with trapping instructions
In reply to Adrian BarnaśImplement the architecture-specific execmem_fill_trapping_insns() helper to poison executable memory regions.
When CONFIG_ARCH_HAS_EXECMEM_ROX is enabled, the execmem subsystem requires a way to fill unused or freed executable memory with architecture-specific trapping instructions. This implementation fills the specified region with AARCH64_BREAK_FAULT instructions and flushes the icache to ensure the traps are immediately visible to execution.
--- arch/arm64/mm/init.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
arch/arm64/mm/init.c +14 -0
@@ -408,6 +408,20 @@ void dump_mem_limit(void) }#ifdef CONFIG_EXECMEM + +#ifdef CONFIG_ARCH_HAS_EXECMEM_ROX +void execmem_fill_trapping_insns (void *ptr,size_t size) +{ +int nr_inst = size /AARCH64_INSN_SIZE ; +__le32 *updptr = ptr; + +for (int i =0 ; i < nr_inst; i++) + updptr[i] =cpu_to_le32 (AARCH64_BREAK_FAULT ); + +flush_icache_range ((unsigned long )ptr, (unsigned long )ptr + size); +} +#endif +static u64 module_direct_base__ro_after_init =0 ;static u64 module_plt_base__ro_after_init =0 ;
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Re: [RFC PATCH 4/6] arm64: mm: add helper to fill execmem with trapping instructions
Quoted 19 lines · Adrian Barnaś
Implement the architecture-specific
execmem_fill_trapping_insns()helper to poison executable memory regions.When
CONFIG_ARCH_HAS_EXECMEM_ROXis enabled, the execmem subsystem requires a way to fill unused or freed executable memory with architecture-specific trapping instructions. This implementation fills the specified region withAARCH64_BREAK_FAULTinstructions and flushes the icache to ensure the traps are immediately visible to execution.Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/mm/init.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)arch/arm64/mm/init.c +0
-0
arch/arm64/mm/init.c void dump_mem_limit(void) · line 415
Quoted 8 lines
arch/arm64/mm/init.c +5
-0@@ -408,6 +408,20 @@ void dump_mem_limit(void) } #ifdef CONFIG_EXECMEM + +#ifdef CONFIG_ARCH_HAS_EXECMEM_ROX +void execmem_fill_trapping_insns(void *ptr, size_t size) +{ + int nr_inst = size / AARCH64_INSN_SIZE;
The x86 instruction is 1 byte, so it can exactly fill any provided buffer. For arm64, the instruction is 4 bytes so we can only exactly fill the buffer if it's size is 4 byte aligned.
I'm guessing that in practice, size will always be page aligned so we are good? Perhaps worth a WARN_ON_ONCE() though?
Thanks,
Ryan
Quoted 12 lines
arch/arm64/mm/init.c +9
-0+__le32 *updptr = ptr; + +for (int i =0 ; i < nr_inst; i++) + updptr[i] =cpu_to_le32 (AARCH64_BREAK_FAULT ); + + flush_icache_range((unsigned long)ptr, (unsigned long)ptr + size); +} +#endif +static u64 module_direct_base__ro_after_init =0 ;static u64 module_plt_base__ro_after_init =0 ;
Re: [RFC PATCH 4/6] arm64: mm: add helper to fill execmem with trapping instructions
Quoted 33 lines · Ryan Roberts
Quoted 19 lines · Adrian Barnaś
Implement the architecture-specific
execmem_fill_trapping_insns()helper to poison executable memory regions.When
CONFIG_ARCH_HAS_EXECMEM_ROXis enabled, the execmem subsystem requires a way to fill unused or freed executable memory with architecture-specific trapping instructions. This implementation fills the specified region withAARCH64_BREAK_FAULTinstructions and flushes the icache to ensure the traps are immediately visible to execution.Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/mm/init.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)arch/arm64/mm/init.c +0
-0arch/arm64/mm/init.c void dump_mem_limit(void) · line 415
Quoted 8 lines
arch/arm64/mm/init.c +5
-0@@ -408,6 +408,20 @@ void dump_mem_limit(void) } #ifdef CONFIG_EXECMEM + +#ifdef CONFIG_ARCH_HAS_EXECMEM_ROX +void execmem_fill_trapping_insns(void *ptr, size_t size) +{ + int nr_inst = size / AARCH64_INSN_SIZE;The x86 instruction is 1 byte, so it can exactly fill any provided buffer. For arm64, the instruction is 4 bytes so we can only exactly fill the buffer if it's size is 4 byte aligned.
Quoted 1 line
I'm guessing that in practice, size will always be page aligned so we are good?
The size is always page aligned:
void *execmem_alloc(enum execmem_type type, size_t size)
{
...
size = PAGE_ALIGN(size);Signature · 2 lines
Sincerely yours, Mike.
RFC PATCH 5/6 arm64: execmem: enable EXECMEM_ROX_CACHE on supported CPUs
In reply to Adrian BarnaśEnable EXECMEM_ROX_CACHE support for ARM64 systems that implement the bbml2_no_abort CPU feature.
Using the ROX cache brings a performance boost by reducing linear region fragmentation caused by strict memory permissions (e.g., W^X enforcement). Grouping executable code (which is read-only in the linear region alias) into PMD-sized block mappings reduces TLB pressure and page table size.
This is only enabled on systems with bbml2_no_abort, as splitting these large blocks to make pages writable during module loading would otherwise risk triggering TLB Conflict Aborts.
--- arch/arm64/Kconfig | 1 + arch/arm64/mm/init.c | 22 +++++++++++++++++++++- 2 files changed, 22 insertions(+), 1 deletion(-)
arch/arm64/Kconfig +1 -0
@@ -285,6 +285,7 @@ config ARM64 select USER_STACKTRACE_SUPPORT select VDSO_GETRANDOM select VMAP_STACK + select ARCH_HAS_EXECMEM_ROX help ARM 64-bit (AArch64) Linux support.
arch/arm64/mm/init.c +21 -1
@@ -420,6 +420,12 @@ void execmem_fill_trapping_insns(void *ptr, size_t size) flush_icache_range((unsigned long)ptr, (unsigned long)ptr + size); } + +#define MODULE_TEXT_FLAG EXECMEM_ROX_CACHE +#define MODULE_TEXT_PGPROT PAGE_KERNEL_ROX +#else +#define MODULE_TEXT_FLAG (0) +#define MODULE_TEXT_PGPROT PAGE_KERNEL#endif static u64 module_direct_base__ro_after_init =0 ;
@@ -511,6 +517,8 @@ struct execmem_info __init *execmem_arch_setup(void) {unsigned long fallback_start =0 , fallback_end =0 ;unsigned long start =0 , end =0 ; +enum execmem_range_flags module_text_flags =0 ; +pgprot_t module_text_pgprot =PAGE_KERNEL ;module_init_limits ();
@@ -531,12 +539,24 @@ struct execmem_info __init *execmem_arch_setup(void) end = module_plt_base +SZ_2G ; } +/* +* The ROX Cache requires bbml2_no_abort because it uses large block +* mappings. On systems without this guarantee, splitting these blocks +* to make pages writable for module loading can trigger TLB Conflict +* Aborts. +*/ +if (system_supports_bbml2_noabort ()) { + module_text_flags =MODULE_TEXT_FLAG ; + module_text_pgprot =MODULE_TEXT_PGPROT ; + } + execmem_info = (struct execmem_info ){ .ranges = { [EXECMEM_MODULE_TEXT ] = { .start = start, .end = end, - .pgprot =PAGE_KERNEL , + .flags = module_text_flags, + .pgprot = module_text_pgprot, .alignment =1 , .fallback_start = fallback_start, .fallback_end = fallback_end,
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Re: [RFC PATCH 5/6] arm64: execmem: enable EXECMEM_ROX_CACHE on supported CPUs
Quoted 7 lines · Adrian Barnaś
Enable
EXECMEM_ROX_CACHEsupport for ARM64 systems that implement thebbml2_no_abortCPU feature.Using the ROX cache brings a performance boost by reducing linear region fragmentation caused by strict memory permissions (e.g., W^X enforcement). Grouping executable code (which is read-only in the linear region alias) into PMD-sized block mappings reduces TLB pressure and page table size.
Do you have any data on fragmentation reduction and/or performance improvement in practice due to this change?
Quoted 15 lines
This is only enabled on systems with
bbml2_no_abort, as splitting these large blocks to make pages writable during module loading would otherwise risk triggering TLB Conflict Aborts.Quoted 1 line
Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/Kconfig | 1 + arch/arm64/mm/init.c | 22 +++++++++++++++++++++- 2 files changed, 22 insertions(+), 1 deletion(-)arch/arm64/Kconfig +0
-0
arch/arm64/Kconfig config ARM64 · line 288
Quoted 4 lines
arch/arm64/Kconfig +1
-0@@ -285,6 +285,7 @@ config ARM64 select USER_STACKTRACE_SUPPORT select VDSO_GETRANDOM select VMAP_STACK + select ARCH_HAS_EXECMEM_ROX
nit: This list is sorted in alphabetical order; please maintain that ordering.
Quoted 33 lines
arch/arm64/Kconfig +0
-0help ARM 64-bit (AArch64) Linux support.arch/arm64/mm/init.c +8
-0@@ -420,6 +420,12 @@ void execmem_fill_trapping_insns(void *ptr, size_t size) flush_icache_range((unsigned long)ptr, (unsigned long)ptr + size); } + +#define MODULE_TEXT_FLAG EXECMEM_ROX_CACHE +#define MODULE_TEXT_PGPROT PAGE_KERNEL_ROX +#else +#define MODULE_TEXT_FLAG (0) +#define MODULE_TEXT_PGPROT PAGE_KERNEL#endif static u64 module_direct_base__ro_after_init =0 ;@@ -511,6 +517,8 @@ struct execmem_info __init *execmem_arch_setup(void) {unsigned long fallback_start =0 , fallback_end =0 ;unsigned long start =0 , end =0 ; +enum execmem_range_flags module_text_flags =0 ; +pgprot_t module_text_pgprot =PAGE_KERNEL ;module_init_limits ();@@ -531,12 +539,24 @@ struct execmem_info __init *execmem_arch_setup(void) end = module_plt_base +SZ_2G ; }
arch/arm64/mm/init.c struct execmem_info __init *execmem_arch_setup(void) · line 551
Quoted 10 lines
arch/arm64/mm/init.c +10
-0+/* +* The ROX Cache requires bbml2_no_abort because it uses large block +* mappings. On systems without this guarantee, splitting these blocks +* to make pages writable for module loading can trigger TLB Conflict +* Aborts. +*/ +if (system_supports_bbml2_noabort ()) { + module_text_flags =MODULE_TEXT_FLAG ; + module_text_pgprot =MODULE_TEXT_PGPROT ; + }
Perhaps this is a bit clearer? Then you don't need the MODULE_TEXT_* macros:
if (IS_ENABLED(CONFIG_ARCH_HAS_EXECMEM_ROX) &&
system_supports_bbml2_noabort()) {
module_text_flags = EXECMEM_ROX_CACHE;
module_text_pgprot = PAGE_KERNEL_ROX;
}Thanks,
Ryan
Quoted 12 lines
arch/arm64/mm/init.c +3
-1+ execmem_info = (struct execmem_info ){ .ranges = { [EXECMEM_MODULE_TEXT ] = { .start = start, .end = end, - .pgprot =PAGE_KERNEL , + .flags = module_text_flags, + .pgprot = module_text_pgprot, .alignment =1 , .fallback_start = fallback_start, .fallback_end = fallback_end,
RFC PATCH 6/6 arm64: mm: support PMD page coalescing in the linear map
In reply to Adrian BarnaśImplement PMD block coalescing to merge fragmented linear mapping regions back into huge pages when restoring the read-only attribute.
When memory allocated with VM_ALLOW_HUGE_VMAP (such as for the execmem ROX cache) has its permissions modified, the PMD block mapping is split into individual PTEs. Without this change, when that memory have its RO attribute subsequently cleared and set the mapping remains permanently fragmented into 4K pages.
--- arch/arm64/include/asm/mmu.h | 1 + arch/arm64/mm/mmu.c | 95 ++++++++++++++++++++++++++++++++++++ arch/arm64/mm/pageattr.c | 7 +++ 3 files changed, 103 insertions(+)
arch/arm64/include/asm/mmu.h +1 -0
@@ -80,6 +80,7 @@ extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot);extern void mark_linear_text_alias_ro (void );extern int split_kernel_leaf_mapping (unsigned long start,unsigned long end);extern void linear_map_maybe_split_to_ptes (void ); +void try_collapse_kernel_pmd (unsigned long addr);/* * This check is triggered during the early boot before the cpufeature
arch/arm64/mm/mmu.c +95 -0
101 lines not on this page · raw message
arch/arm64/mm/pageattr.c +7 -0
@@ -200,6 +200,13 @@ static int change_memory_common(unsigned long addr, int numpages,if (ret)return ret; } +/* +* When setting a read-only flag on the linear region, the memory +* may have been backed by a PMD before being split. Try to +* collapse it back into a PMD to restore huge page performance. +*/ +if (pgprot_val (set_mask) ==PTE_RDONLY && area->flags &VM_ALLOW_HUGE_VMAP ) +try_collapse_kernel_pmd ((u64 )page_address (area->pages[0 ])); }/*
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Re: [RFC PATCH 6/6] arm64: mm: support PMD page coalescing in the linear map
Quoted 8 lines · Adrian Barnaś
Implement PMD block coalescing to merge fragmented linear mapping regions back into huge pages when restoring the read-only attribute.
When memory allocated with
VM_ALLOW_HUGE_VMAP(such as for the execmem ROX cache) has its permissions modified, the PMD block mapping is split into individual PTEs. Without this change, when that memory have its RO attribute subsequently cleared and set the mapping remains permanently fragmented into 4K pages.
I'd like to make sure I understand this correctly: So the execmem ROX cache is allocated using vmalloc_huge such that all it's backing memory are PMD-sized pages. It is initially poisoned with a faulting instruction and marked ROX. When a module is loaded, it's text is copied into a portion of the ROX cache and executed from there? To do that, the required number of (4K) pages are allocated from ROX cache, and the permissions are changed on that sub-region to RW, meaning we have to split the mapping from a PMD to (cont)PTEs in both vmalloc space and in the linear map. After the text is copied, the sub-region is switched back to ROX. But without that change, the 2M region is now mapped by (cont)PTEs for all of time?
Quoted 25 lines
Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/include/asm/mmu.h | 1 + arch/arm64/mm/mmu.c | 95 ++++++++++++++++++++++++++++++++++++ arch/arm64/mm/pageattr.c | 7 +++ 3 files changed, 103 insertions(+)arch/arm64/include/asm/mmu.h +1
-0@@ -80,6 +80,7 @@ extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot);extern void mark_linear_text_alias_ro (void );extern int split_kernel_leaf_mapping (unsigned long start,unsigned long end);extern void linear_map_maybe_split_to_ptes (void ); +void try_collapse_kernel_pmd (unsigned long addr);/* * This check is triggered during the early boot before the cpufeature arch/arm64/mm/mmu.c +0
-0
arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 778
Quoted 10 lines
arch/arm64/mm/mmu.c +7
-0@@ -769,6 +769,101 @@ static inline bool force_pte_mapping(void)static DEFINE_MUTEX (pgtable_split_lock); +static inline bool __pte_can_be_collapsed (pte_t pte,unsigned long pfn,pgprot_t prot) +{ +if (!pte_valid (pte)) +return false; +if (pte_pfn (pte) != pfn) +return false; +if ((pgprot_val (pte_pgprot (pte)) & ~PTE_CONT ) !=pgprot_val (prot))
Do we want to permit differing values for access and dirty? We do for user space, but I think for kernel, those bits are always set? In which case, what you're doing is correct.
Quoted 25 lines
arch/arm64/mm/mmu.c +25
-0+return false; + +return true; +} + +static void __try_collapse_pmd (pmd_t *pmdp,pmd_t pmd,unsigned long addr) +{ +pte_t *ptep; +pte_t first_pte; +unsigned long pfn; +pgprot_t prot; +int i; + + ptep = (pte_t *)pmd_page_vaddr (pmd); + first_pte =__ptep_get (ptep); + +if (!pte_valid (first_pte)) +return ; + + prot =pte_pgprot (first_pte); + prot =__pgprot (pgprot_val (prot) & ~PTE_CONT ); + pfn =pte_pfn (first_pte); + +if (!IS_ALIGNED (pfn,PMD_SIZE >>PAGE_SHIFT )) +return ;
arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 813
Quoted 10 lines
arch/arm64/mm/mmu.c +10
-0+ +for (i =1 ; i <PTRS_PER_PTE ; i++) { +if (!__pte_can_be_collapsed (__ptep_get (ptep + i), pfn + i, prot)) +return ; + } + +set_pmd (pmdp,pmd_mkhuge (pfn_pmd (pfn, prot))); + +__flush_tlb_kernel_pgtable (addr); +
nit: suggest adding the same comment that other sites has, since this is not obvious:
/* See comment in pud_free_pmd_page for static key logic */
arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 819
Quoted 6 lines
arch/arm64/mm/mmu.c +6
-0+if (static_branch_unlikely (&arm64_ptdump_lock_key)) { +mmap_read_lock (&init_mm); +mmap_read_unlock (&init_mm); + } + +pte_free_kernel (NULL , ptep);
Ooh, fun. Per my comments below we definitely have opportunity for racy pte table accesses (beyond ptdump). I _think_ if you fix that then this will be safe.
Quoted 7 lines
arch/arm64/mm/mmu.c +7
-0+} + +void try_collapse_kernel_pmd (unsigned long addr) +{ +pgd_t *pgdp; +p4d_t *p4dp; +pud_t *pudp;
arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 836
Quoted 10 lines
arch/arm64/mm/mmu.c +10
-0+pmd_t *pmdp; +pmd_t pmd; + +/* +* collapse_pmd expects exact address of block to be collapsed +*/ +if (WARN_ON (ALIGN_DOWN (addr,PMD_SIZE ) != addr)) +return ; + +mutex_lock (&pgtable_split_lock);
I don't think this is safe in general. Let's say we have a 2M region split into 512 x 4K PTEs. It's possible that the first 1M is one object and the second 1M is another object. Different CPUs could set_memory_*() on those 2 objects concurrently. If one of them then calls this function, we could end up collapsing the whole 2M while the other is trying to modify the PTEs and they will race.
Note that splitting _is_ safe (and protected by this lock) because you'd have 2 objects backed by the same PMD, so they would both have to split before modifying the PTEs.
I think you'd need to ensure mutual exclusion at a higher level if doing this; probably execmem is the place that can ensure that no objects within a 2M region are racily trying to modify their permissions?
arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 839
Quoted 3 lines
arch/arm64/mm/mmu.c +3
-0+ + pgdp =pgd_offset_k (addr); +if (pgd_none (READ_ONCE (*pgdp)))
nit: use the getters (e.g. pXdp_get()) instead of READ_ONCE().
Quoted 35 lines
arch/arm64/mm/mmu.c +27
-0+ goto out; + + p4dp =p4d_offset (pgdp, addr); +if (p4d_none (READ_ONCE (*p4dp))) + goto out; + + pudp =pud_offset (p4dp, addr); +if (pud_none (READ_ONCE (*pudp))) + goto out; + +if (pud_leaf (READ_ONCE (*pudp))) + goto out; + + pmdp =pmd_offset (pudp, addr); + pmd =pmdp_get (pmdp); + +if (!pmd_table (pmd)) + goto out; + +lazy_mmu_mode_enable (); +__try_collapse_pmd (pmdp, pmd, addr); +lazy_mmu_mode_disable (); + +out: +mutex_unlock (&pgtable_split_lock); +} +int split_kernel_leaf_mapping (unsigned long start,unsigned long end) {int ret;arch/arm64/mm/pageattr.c +0
-0
arch/arm64/mm/pageattr.c static int change_memory_common(unsigned long addr, int numpages, · line 202
Quoted 3 lines
arch/arm64/mm/pageattr.c +0
-0@@ -200,6 +200,13 @@ static int change_memory_common(unsigned long addr, int numpages,if (ret)return ret; }
The loop here in the unchanged code, calls __change_memory_common() for each PAGE_SIZE page in the linear alias. Perhaps it should be area->page_order aware? That way, if we are changing the permissions of the whole 2M chunk of vmalloc space and it's backed by a 2M page, then we won't split the mapping to PTEs in the linear map. Similarly, if we are changing permissions on a sub-region of the 2M area, we might be able to split only as far as contpte and still have some performance benefit?
arch/arm64/mm/pageattr.c static int change_memory_common(unsigned long addr, int numpages, · line 209
Quoted 7 lines
arch/arm64/mm/pageattr.c +7
-0+/* +* When setting a read-only flag on the linear region, the memory +* may have been backed by a PMD before being split. Try to +* collapse it back into a PMD to restore huge page performance. +*/ +if (pgprot_val (set_mask) ==PTE_RDONLY && area->flags &VM_ALLOW_HUGE_VMAP ) +try_collapse_kernel_pmd ((u64 )page_address (area->pages[0 ]));
try_collapse_kernel_pmd() requires a PMD-aligned address. VM_ALLOW_HUGE_VMAP doesn't guarrantee that - it only says that it's _allowed_ to be huge (and 64K would still be huge). vmalloc may have failed to allocate a PMD-sized page and reverted to something smaller (64K contpte or 4K). You need to look at area->page_order, I think.
You're only calling this for the linear alias. Don't you want to call it for the vmalloc range too? I assume the module text executes using the vmalloc address so you'd want it mapped by a single PMD for best performance?
But in general, I don't really like the idea of special-casing a collapse to pmd here for just execmem. I think we should either investigate a general purpose collapse mechanism or execmem should have extra hooks added to request specific collapse.
Thanks,
Ryan
Quoted 3 lines
arch/arm64/mm/pageattr.c +0
-0}/*
Re: [RFC PATCH 6/6] arm64: mm: support PMD page coalescing in the linear map
Quoted 10 lines · Ryan Roberts
Quoted 8 lines · Adrian Barnaś
Implement PMD block coalescing to merge fragmented linear mapping regions back into huge pages when restoring the read-only attribute.
When memory allocated with
VM_ALLOW_HUGE_VMAP(such as for the execmem ROX cache) has its permissions modified, the PMD block mapping is split into individual PTEs. Without this change, when that memory have its RO attribute subsequently cleared and set the mapping remains permanently fragmented into 4K pages.
Quoted 10 lines
I'd like to make sure I understand this correctly: So the execmem ROX cache is allocated using
vmalloc_hugesuch that all it's backing memory are PMD-sized pages. It is initially poisoned with a faulting instruction and marked ROX. When a module is loaded, it's text is copied into a portion of the ROX cache and executed from there? To do that, the required number of (4K) pages are allocated from ROX cache, and the permissions are changed on that sub-region to RW, meaning we have to split the mapping from a PMD to (cont)PTEs in both vmalloc space and in the linear map. After the text is copied, the sub-region is switched back to ROX. But without that change, the 2M region is now mapped by (cont)PTEs for all of time?
That is correct.
Quoted 170 lines
Quoted 26 lines
Signed-off-by Adrian Barnaś <[email protected]>--- arch/arm64/include/asm/mmu.h | 1 + arch/arm64/mm/mmu.c | 95 ++++++++++++++++++++++++++++++++++++ arch/arm64/mm/pageattr.c | 7 +++ 3 files changed, 103 insertions(+)arch/arm64/include/asm/mmu.h +1
-0@@ -80,6 +80,7 @@ extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot);extern void mark_linear_text_alias_ro (void );extern int split_kernel_leaf_mapping (unsigned long start,unsigned long end);extern void linear_map_maybe_split_to_ptes (void ); +void try_collapse_kernel_pmd (unsigned long addr);/* * This check is triggered during the early boot before the cpufeature arch/arm64/mm/mmu.c +0
-0arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 778
Quoted 10 lines
arch/arm64/mm/mmu.c +7
-0@@ -769,6 +769,101 @@ static inline bool force_pte_mapping(void)static DEFINE_MUTEX (pgtable_split_lock); +static inline bool __pte_can_be_collapsed (pte_t pte,unsigned long pfn,pgprot_t prot) +{ +if (!pte_valid (pte)) +return false; +if (pte_pfn (pte) != pfn) +return false; +if ((pgprot_val (pte_pgprot (pte)) & ~PTE_CONT ) !=pgprot_val (prot))Do we want to permit differing values for access and dirty? We do for user space, but I think for kernel, those bits are always set? In which case, what you're doing is correct.
Quoted 25 lines
arch/arm64/mm/mmu.c +25
-0+return false; + +return true; +} + +static void __try_collapse_pmd (pmd_t *pmdp,pmd_t pmd,unsigned long addr) +{ +pte_t *ptep; +pte_t first_pte; +unsigned long pfn; +pgprot_t prot; +int i; + + ptep = (pte_t *)pmd_page_vaddr (pmd); + first_pte =__ptep_get (ptep); + +if (!pte_valid (first_pte)) +return ; + + prot =pte_pgprot (first_pte); + prot =__pgprot (pgprot_val (prot) & ~PTE_CONT ); + pfn =pte_pfn (first_pte); + +if (!IS_ALIGNED (pfn,PMD_SIZE >>PAGE_SHIFT )) +return ;arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 813
Quoted 10 lines
arch/arm64/mm/mmu.c +10
-0+ +for (i =1 ; i <PTRS_PER_PTE ; i++) { +if (!__pte_can_be_collapsed (__ptep_get (ptep + i), pfn + i, prot)) +return ; + } + +set_pmd (pmdp,pmd_mkhuge (pfn_pmd (pfn, prot))); + +__flush_tlb_kernel_pgtable (addr); +nit: suggest adding the same comment that other sites has, since this is not obvious:
/* See comment in
pud_free_pmd_pagefor static key logic */arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 819
Quoted 6 lines
arch/arm64/mm/mmu.c +6
-0+if (static_branch_unlikely (&arm64_ptdump_lock_key)) { +mmap_read_lock (&init_mm); +mmap_read_unlock (&init_mm); + } + +pte_free_kernel (NULL , ptep);Ooh, fun. Per my comments below we definitely have opportunity for racy pte table accesses (beyond ptdump). I _think_ if you fix that then this will be safe.
Quoted 7 lines
arch/arm64/mm/mmu.c +7
-0+} + +void try_collapse_kernel_pmd (unsigned long addr) +{ +pgd_t *pgdp; +p4d_t *p4dp; +pud_t *pudp;arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 836
Quoted 10 lines
arch/arm64/mm/mmu.c +10
-0+pmd_t *pmdp; +pmd_t pmd; + +/* +* collapse_pmd expects exact address of block to be collapsed +*/ +if (WARN_ON (ALIGN_DOWN (addr,PMD_SIZE ) != addr)) +return ; + +mutex_lock (&pgtable_split_lock);I don't think this is safe in general. Let's say we have a 2M region split into 512 x 4K PTEs. It's possible that the first 1M is one object and the second 1M is another object. Different CPUs could set_memory_*() on those 2 objects concurrently. If one of them then calls this function, we could end up collapsing the whole 2M while the other is trying to modify the PTEs and they will race.
Note that splitting _is_ safe (and protected by this lock) because you'd have 2 objects backed by the same PMD, so they would both have to split before modifying the PTEs.
I think you'd need to ensure mutual exclusion at a higher level if doing this; probably execmem is the place that can ensure that no objects within a 2M region are racily trying to modify their permissions?
arch/arm64/mm/mmu.c static inline bool force_pte_mapping(void) · line 839
Quoted 3 lines
arch/arm64/mm/mmu.c +3
-0+ + pgdp =pgd_offset_k (addr); +if (pgd_none (READ_ONCE (*pgdp)))nit: use the getters (e.g.
pXdp_get()) instead ofREAD_ONCE().Quoted 35 lines
arch/arm64/mm/mmu.c +27
-0+ goto out; + + p4dp =p4d_offset (pgdp, addr); +if (p4d_none (READ_ONCE (*p4dp))) + goto out; + + pudp =pud_offset (p4dp, addr); +if (pud_none (READ_ONCE (*pudp))) + goto out; + +if (pud_leaf (READ_ONCE (*pudp))) + goto out; + + pmdp =pmd_offset (pudp, addr); + pmd =pmdp_get (pmdp); + +if (!pmd_table (pmd)) + goto out; + +lazy_mmu_mode_enable (); +__try_collapse_pmd (pmdp, pmd, addr); +lazy_mmu_mode_disable (); + +out: +mutex_unlock (&pgtable_split_lock); +} +int split_kernel_leaf_mapping (unsigned long start,unsigned long end) {int ret;arch/arm64/mm/pageattr.c +0
-0arch/arm64/mm/pageattr.c static int change_memory_common(unsigned long addr, int numpages, · line 202
Quoted 3 lines
arch/arm64/mm/pageattr.c +0
-0@@ -200,6 +200,13 @@ static int change_memory_common(unsigned long addr, int numpages,if (ret)return ret; }
Quoted 8 lines
The loop here in the unchanged code, calls
__change_memory_common()for eachPAGE_SIZEpage in the linear alias. Perhaps it should bearea->page_orderaware? That way, if we are changing the permissions of the whole 2M chunk of vmalloc space and it's backed by a 2M page, then we won't split the mapping to PTEs in the linear map. Similarly, if we are changing permissions on a sub-region of the 2M area, we might be able to split only as far as contpte and still have some performance benefit?
This is out of scope of execmem (as you noted we are mostly changing permision for only a part of the block) but your sugestion seems to be valid for other cases.
Quoted 5 lines
Quoted 5 lines
arch/arm64/mm/pageattr.c +5
-0+/* +* When setting a read-only flag on the linear region, the memory +* may have been backed by a PMD before being split. Try to +* collapse it back into a PMD to restore huge page performance. +*/
Quoted 9 lines
arch/arm64/mm/pageattr.c static int change_memory_common(unsigned long addr, int numpages, · line 209
Quoted 2 lines
arch/arm64/mm/pageattr.c +2
-0+if (pgprot_val (set_mask) ==PTE_RDONLY && area->flags &VM_ALLOW_HUGE_VMAP ) +try_collapse_kernel_pmd ((u64 )page_address (area->pages[0 ]));
try_collapse_kernel_pmd()requires a PMD-aligned address.VM_ALLOW_HUGE_VMAPdoesn't guarrantee that - it only says that it's _allowed_ to be huge (and 64K would still be huge). vmalloc may have failed to allocate a PMD-sized page and reverted to something smaller (64K contpte or 4K). You need to look atarea->page_order, I think.
Agree.
Quoted 4 lines
You're only calling this for the linear alias. Don't you want to call it for the vmalloc range too? I assume the module text executes using the vmalloc address so you'd want it mapped by a single PMD for best performance?
I think it is a good idea to have them both (linear alias and vmalloc area) mapped as PMD.
Quoted 5 lines
But in general, I don't really like the idea of special-casing a collapse to pmd here for just execmem. I think we should either investigate a general purpose collapse mechanism or execmem should have extra hooks added to request specific collapse.>
Quoted 8 lines
Thanks,
RyanQuoted 3 lines
arch/arm64/mm/pageattr.c +0
-0}/*
I will rethink this based on your suggestion and try to properly fix the concurrency issues.
I am also wondering if we should avoid splitting linear aliases altogether. I think I understand why it was done originally (to restrict writing through
linear mapping aliases for read-only vmalloc areas). However, it is not necessary to make them writable when the vmalloc area is writable. Maybe we should create a special case for execmem to prevent this. WDYT?
Thank you for the review,
Adrian
Re: [RFC PATCH 6/6] arm64: mm: support PMD page coalescing in the linear map
In reply to Ryan RobertsHi Mike,
Quoted 11 lines · Ryan Roberts
I don't think this is safe in general. Let's say we have a 2M region split into 512 x 4K PTEs. It's possible that the first 1M is one object and the second 1M is another object. Different CPUs could set_memory_*() on those 2 objects concurrently. If one of them then calls this function, we could end up collapsing the whole 2M while the other is trying to modify the PTEs and they will race.
Note that splitting _is_ safe (and protected by this lock) because you'd have 2 objects backed by the same PMD, so they would both have to split before modifying the PTEs.
Quoted 6 lines
I think you'd need to ensure mutual exclusion at a higher level if doing this; probably execmem is the place that can ensure that no objects within a 2M region are racily trying to modify their permissions?
Thanks,
Ryan
I wonder what do you think about enforcing mutual exclusion for permission changes to the ROX_CACHE block as Ryan sugest?
Would it be appropriate to add locks (over &execmem_cache.mutex) to execmem_force_rw and execmem_restore_rox?
I see that x86 has a lock that wraps entire process of updating attributes (including spliting and collapsing pages):
```
spin_lock(&cpa_lock);
ret = __change_page_attr(cpa, primary);
spin_unlock(&cpa_lock);
``` As I understand it makes it safe already. However, on ARM64 there is a split lock (which works fine for spliting) but for collapsing at least on my implementation it is not sufficient as Ryan pointed out.
Best regards
Adrian
Re: [RFC PATCH 6/6] arm64: mm: support PMD page coalescing in the linear map
Quoted 25 lines · Adrian Barnaś
Hi Mike,
Quoted 11 lines · Ryan Roberts
I don't think this is safe in general. Let's say we have a 2M region split into 512 x 4K PTEs. It's possible that the first 1M is one object and the second 1M is another object. Different CPUs could set_memory_*() on those 2 objects concurrently. If one of them then calls this function, we could end up collapsing the whole 2M while the other is trying to modify the PTEs and they will race.
Note that splitting _is_ safe (and protected by this lock) because you'd have 2 objects backed by the same PMD, so they would both have to split before modifying the PTEs.
Quoted 6 lines
I think you'd need to ensure mutual exclusion at a higher level if doing this; probably execmem is the place that can ensure that no objects within a 2M region are racily trying to modify their permissions?
Thanks,
RyanI wonder what do you think about enforcing mutual exclusion for permission changes to the
ROX_CACHEblock as Ryan sugest?
Quoted 13 lines
Would it be appropriate to add locks (over
&execmem_cache.mutex) toexecmem_force_rwandexecmem_restore_rox?
I
see that x86 has a lock that wraps entire process of updating attributes (including spliting and collapsing pages):
```spin_lock(&cpa_lock);
ret =__change_page_attr(cpa, primary);spin_unlock(&cpa_lock); ```
As I understand it makes it safe already.
However, on ARM64 there is a split lock (which works fine for spliting) but for collapsing at least on my implementation it is not sufficient as Ryan pointed out.
I think the lock should be at set_memory level so that set_memory callers shouldn't care about the differences in arch implementations.
I also suspect that the lockless page table walk in update_range_prot() isn't safe today.
Quoted 2 lines
Best regards
Adrian
Signature · 2 lines
Sincerely yours, Mike.