s390/mm: Batch PTE updates in lazy MMU mode
Subsystems (11)
Alexander Gordeev
maintainerv3 cover s390/mm: Batch PTE updates in lazy MMU mode
Hi All!
This is v3 of the batched PTE updates in lazy MMU mode rework.
The prereq patches 4,5 are in the mm tree and scheduled for -next already:git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
The presented implementation sets up per-cpu caches in the s390-specific hotplug callbacks as opposed to CPUHP_BP_PREPARE_DYN hooks. I like this approach better, since it locates the SMP-related lowcore initialization in one place. But that is up for discussion.
Changes since v2:
- lazy_mmu_mode_enable_for_pte_range() renamed to lazy_mmu_mode_enable_with_ptes() (David Hildenbrand)
- patch "mm/pgtable: Fix bogus comment to clear_not_present_full_ptes()" is dropped (David Hildenbrand)
- direct PTE dereferencing KASAN sanitizer added (Heiko Carstens)
- CONFIG_IPTE_BATCH option is dropped (Heiko Carstens)
- PTE_POISON changed from zero to 0x800 (Heiko Carstens)
- allocate per-cpu caches on CPU hot-plug (Heiko Carstens)
- introduced a lowcore field for fast lazy mode checking (Heiko Carstens)
- few minor code changes (Heiko Carstens)
Changes since v1:
- lazy_mmu_mode_enable_pte() renamed to lazy_mmu_mode_enable_for_pte_range()
- lazy_mmu_mode_enable_for_pte_range() semantics clarified
- some sashiko comments addressed [1] including one bug fix
1. https://sashiko.dev/#/patchset/cover.1774420056.git.agordeev%40linux.ibm.com
- patches 2-4 added
This series addresses an s390-specific aspect of how page table entries are modified. In many cases, changing a valid PTE (for example, setting or clearing a hardware bit) requires issuing an Invalidate Page Table Entry (IPTE) instruction beforehand.
A disadvantage of the IPTE instruction is that it may initiate a machine-wide quiesce state. This state acts as an expensive global hardware lock and should be avoided whenever possible.
Currently, IPTE is invoked for each individual PTE update in most code paths. However, the instruction itself supports invalidating multiple PTEs at once, covering up to 256 entries. Using this capability can significantly reduce the number of quiesce events, with a positive impact on overall system performance. At present, this feature is not utilized.
An effort was therefore made to identify kernel code paths that update large numbers of consecutive PTEs. Such updates can be batched and handled by a single IPTE invocation, leveraging the hardware support described above.
A natural candidate for this optimization is page-table walkers that change attributes of memory ranges and thus modify contiguous ranges of PTEs. Many memory-management system calls enter lazy MMU mode while updating such ranges.
This lazy MMU mode can be leveraged to build on the already existing infrastructure and implement a software-level lazy MMU mechanism, allowing expensive PTE invalidations on s390 to be batched.
Alexander Gordeev (7):
mm: Make lazy MMU mode context-aware
s390/mm: Complete ptep_get() conversion
s390/mm: Batch PTE updates in lazy MMU mode
mm/gup: Cleanup pgtable entry accessors mm/page_vma_mapped_walk: Use ptep_get_lockless() for lockless access mm/kasan: Introduce helpers for lazy MMU mode sanitizer s390/mm: Lazy MMU mode sanitizer
arch/s390/Kconfig | 1 + arch/s390/boot/vmem.c | 32 +-- arch/s390/include/asm/hugetlb.h | 2 +- arch/s390/include/asm/lazy_mmu.h | 9 + arch/s390/include/asm/lowcore.h | 2 +- arch/s390/include/asm/pgtable.h | 213 +++++++++++++--- arch/s390/kernel/setup.c | 2 + arch/s390/kernel/smp.c | 7 + arch/s390/mm/Makefile | 2 +- arch/s390/mm/hugetlbpage.c | 12 +- arch/s390/mm/lazy_mmu.c | 401 +++++++++++++++++++++++++++++++ arch/s390/mm/pageattr.c | 45 ++-- arch/s390/mm/pgtable.c | 8 +- arch/s390/mm/vmem.c | 82 ++++--- fs/proc/task_mmu.c | 2 +- include/linux/kasan.h | 16 ++ include/linux/pgtable.h | 46 ++++ mm/gup.c | 8 +- mm/kasan/common.c | 10 + mm/kasan/kasan.h | 2 + mm/madvise.c | 8 +- mm/memory.c | 8 +- mm/mprotect.c | 2 +- mm/mremap.c | 2 +- mm/page_vma_mapped.c | 9 +- mm/vmalloc.c | 6 +- 26 files changed, 805 insertions(+), 132 deletions(-) create mode 100644 arch/s390/include/asm/lazy_mmu.h create mode 100644 arch/s390/mm/lazy_mmu.c
Signature · 1 line
2.53.0
Alexander Gordeev
maintainerPATCH v3 1/7 mm: Make lazy MMU mode context-aware
Lazy MMU mode is assumed to be context-independent, in the sense that it does not need any additional information while operating. However, the s390 architecture benefits from knowing the exact page table entries being modified.
Introduce lazy_mmu_mode_enable_with_ptes(), which is provided with the process address space and the page table being operated on. This information is required to enable s390-specific optimizations.
The function takes parameters that are typically passed to page- table level walkers, which implies that the span of PTE entries never crosses a page table boundary.
Architectures that do not require such information simply do not need to define the lazy_mmu_mode_enable_with_ptes() callback.
--- fs/proc/task_mmu.c | 2 +- include/linux/pgtable.h | 46 +++++++++++++++++++++++++++++++++++++++++ mm/madvise.c | 8 +++---- mm/memory.c | 8 +++---- mm/mprotect.c | 2 +- mm/mremap.c | 2 +- mm/vmalloc.c | 6 +++--- 7 files changed, 60 insertions(+), 14 deletions(-)
fs/proc/task_mmu.c +1 -1
@@ -2752,7 +2752,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,return 0 ; } -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (vma->vm_mm, start, end, start_pte);if ((p->arg.flags &PM_SCAN_WP_MATCHING ) && !p->vec_out) {/* Fast path for performing exclusive WP */
include/linux/pgtable.h +46 -0
58 lines not on this page · raw message
mm/madvise.c +4 -4
@@ -453,7 +453,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (!start_pte)return 0 ;flush_tlb_batched_pending (mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, start_pte);for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { nr =1 ; ptent =ptep_get (pte);
@@ -508,7 +508,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (!start_pte)break ;flush_tlb_batched_pending (mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, start_pte);if (!err) nr =0 ;continue ;
@@ -675,7 +675,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,if (!start_pte)return 0 ;flush_tlb_batched_pending (mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, start_pte);for (; addr != end; pte += nr, addr +=PAGE_SIZE * nr) { nr =1 ; ptent =ptep_get (pte);
@@ -735,7 +735,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,if (!start_pte)break ;flush_tlb_batched_pending (mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, pte);if (!err) nr =0 ;continue ;
mm/memory.c +4 -4
@@ -1272,7 +1272,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,spin_lock_nested (src_ptl,SINGLE_DEPTH_NESTING ); orig_src_pte = src_pte; orig_dst_pte = dst_pte; -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (src_mm, addr, end, src_pte); do { nr = 1;
@@ -1922,7 +1922,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,return addr;flush_tlb_batched_pending (mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, start_pte); do { bool any_skipped = false;
@@ -2919,7 +2919,7 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, mapped_pte = pte =pte_alloc_map_lock (mm, pmd, addr, &ptl);if (!pte)return -ENOMEM ; -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, mapped_pte); do { BUG_ON(!pte_none(ptep_get(pte))); if (!pfn_modify_allowed(pfn, prot)) {
@@ -3330,7 +3330,7 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,return -EINVAL ; } -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, mapped_pte);if (fn) { do {
mm/mprotect.c +1 -1
@@ -337,7 +337,7 @@ static long change_pte_range(struct mmu_gather *tlb, is_private_single_threaded =vma_is_single_threaded_private (vma);flush_tlb_batched_pending (vma->vm_mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (vma->vm_mm, addr, end, pte); do { nr_ptes = 1; oldpte = ptep_get(pte);
mm/mremap.c +1 -1
@@ -260,7 +260,7 @@ static int move_ptes(struct pagetable_move_control *pmc,if (new_ptl != old_ptl)spin_lock_nested (new_ptl,SINGLE_DEPTH_NESTING );flush_tlb_batched_pending (vma->vm_mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, old_addr, old_end, old_ptep);for (; old_addr < old_end; old_ptep += nr_ptes, old_addr += nr_ptes *PAGE_SIZE , new_ptep += nr_ptes, new_addr += nr_ptes *PAGE_SIZE ) {
mm/vmalloc.c +3 -3
@@ -108,7 +108,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,if (!pte)return -ENOMEM ; -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (&init_mm, addr, end, pte); do { if (unlikely(!pte_none(ptep_get(pte)))) {
@@ -371,7 +371,7 @@ static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,unsigned long size =PAGE_SIZE ; pte =pte_offset_kernel (pmd, addr); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (&init_mm, addr, end, pte); do { #ifdef CONFIG_HUGETLB_PAGE
@@ -538,7 +538,7 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr,if (!pte)return -ENOMEM ; -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (&init_mm, addr, end, pte); do { struct page *page = pages[*nr];
Signature · 1 line
2.53.0
Alexander Gordeev
maintainerPATCH v3 2/7 s390/mm: Complete ptep_get() conversion
In reply to Alexander GordeevFinalize commit c33c794828f2 ("mm: ptep_get() conversion") and replace direct page table entry dereferencing with the proper accessors (ptep_get(), pmdp_get(), etc.).
Override the default getter implementations even though they are currently identical: pud_clear(), p4d_clear(), and pgd_clear() require corresponding architecture-specific getters, but these are not yet defined. This avoids a dependency loop.
--- arch/s390/boot/vmem.c | 32 +++++++------ arch/s390/include/asm/hugetlb.h | 2 +- arch/s390/include/asm/pgtable.h | 60 ++++++++++++++++++------ arch/s390/mm/hugetlbpage.c | 12 ++--- arch/s390/mm/pageattr.c | 45 ++++++++++-------- arch/s390/mm/vmem.c | 82 ++++++++++++++++++--------------- 6 files changed, 140 insertions(+), 93 deletions(-)
arch/s390/boot/vmem.c +17 -15
@@ -338,7 +338,7 @@ static void pgtable_pte_populate(pmd_t *pmd, unsigned long addr, unsigned long e pte =pte_offset_kernel (pmd, addr);for (; addr < end; addr +=PAGE_SIZE , pte++) { -if (pte_none (*pte)) { +if (pte_none (ptep_get (pte))) {if (kasan_pte_populate_zero_shadow (pte, mode))continue ; entry =__pte (resolve_pa_may_alloc (addr,PAGE_SIZE , mode));
@@ -355,26 +355,27 @@ static void pgtable_pmd_populate(pud_t *pud, unsigned long addr, unsigned long e enum populate_mode mode) {unsigned long pa, next, pages =0 ; -pmd_t *pmd, entry; +pmd_t *pmd, entry, large_entry;pte_t *pte; pmd =pmd_offset (pud, addr);for (; addr < end; addr = next, pmd++) { next =pmd_addr_end (addr, end); -if (pmd_none (*pmd)) { + entry =pmdp_get (pmd); +if (pmd_none (entry)) {if (kasan_pmd_populate_zero_shadow (pmd, addr, next, mode))continue ; pa =try_get_large_pmd_pa (pmd, addr, next, mode);if (pa !=INVALID_PHYS_ADDR ) { - entry =__pmd (pa); - entry =set_pmd_bit (entry,SEGMENT_KERNEL ); -set_pmd (pmd, entry); + large_entry =__pmd (pa); + large_entry =set_pmd_bit (large_entry,SEGMENT_KERNEL ); +set_pmd (pmd, large_entry); pages++;continue ; } pte =boot_pte_alloc ();pmd_populate (&init_mm, pmd, pte); - }else if (pmd_leaf (*pmd)) { + }else if (pmd_leaf (entry)) {continue ; }pgtable_pte_populate (pmd, addr, next, mode);
@@ -387,26 +388,27 @@ static void pgtable_pud_populate(p4d_t *p4d, unsigned long addr, unsigned long e enum populate_mode mode) {unsigned long pa, next, pages =0 ; -pud_t *pud, entry; +pud_t *pud, entry, large_entry;pmd_t *pmd; pud =pud_offset (p4d, addr);for (; addr < end; addr = next, pud++) { next =pud_addr_end (addr, end); -if (pud_none (*pud)) { + entry =pudp_get (pud); +if (pud_none (entry)) {if (kasan_pud_populate_zero_shadow (pud, addr, next, mode))continue ; pa =try_get_large_pud_pa (pud, addr, next, mode);if (pa !=INVALID_PHYS_ADDR ) { - entry =__pud (pa); - entry =set_pud_bit (entry,REGION3_KERNEL ); -set_pud (pud, entry); + large_entry =__pud (pa); + large_entry =set_pud_bit (large_entry,REGION3_KERNEL ); +set_pud (pud, large_entry); pages++;continue ; } pmd =boot_crst_alloc (_SEGMENT_ENTRY_EMPTY);pud_populate (&init_mm, pud, pmd); - }else if (pud_leaf (*pud)) { + }else if (pud_leaf (entry)) {continue ; }pgtable_pmd_populate (pud, addr, next, mode);
@@ -425,7 +427,7 @@ static void pgtable_p4d_populate(pgd_t *pgd, unsigned long addr, unsigned long e p4d =p4d_offset (pgd, addr);for (; addr < end; addr = next, p4d++) { next =p4d_addr_end (addr, end); -if (p4d_none (*p4d)) { +if (p4d_none (p4dp_get (p4d))) {if (kasan_p4d_populate_zero_shadow (p4d, addr, next, mode))continue ; pud =boot_crst_alloc (_REGION3_ENTRY_EMPTY);
@@ -451,7 +453,7 @@ static void pgtable_populate(unsigned long addr, unsigned long end, enum populat pgd =pgd_offset (&init_mm, addr);for (; addr < end; addr = next, pgd++) { next =pgd_addr_end (addr, end); -if (pgd_none (*pgd)) { +if (pgd_none (pgdp_get (pgd))) {if (kasan_pgd_populate_zero_shadow (pgd, addr, next, mode))continue ; p4d =boot_crst_alloc (_REGION2_ENTRY_EMPTY);
arch/s390/include/asm/hugetlb.h +1 -1
@@ -41,7 +41,7 @@ static inline pte_t huge_ptep_get_and_clear(struct mm_struct *mm,static inline void huge_pte_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep,unsigned long sz) { -if ((pte_val (*ptep) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R3) +if ((pte_val (ptep_get (ptep)) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R3)set_pte (ptep,__pte (_REGION3_ENTRY_EMPTY));else set_pte (ptep,__pte (_SEGMENT_ENTRY_EMPTY));
arch/s390/include/asm/pgtable.h +45 -15
104 lines not on this page · raw message
arch/s390/mm/hugetlbpage.c +6 -6
@@ -143,7 +143,7 @@ void __set_huge_pte_at(struct mm_struct *mm, unsigned long addr, rste =__pte_to_rste (pte);/* Set correct table type for 2G hugepages */ -if ((pte_val (*ptep) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R3) { +if ((pte_val (ptep_get (ptep)) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R3) {if (likely (pte_present (pte))) rste |= _REGION3_ENTRY_LARGE; rste |= _REGION_ENTRY_TYPE_R3;
@@ -161,7 +161,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr,pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { -return __rste_to_pte (pte_val (*ptep)); +return __rste_to_pte (pte_val (ptep_get (ptep))); } pte_t __huge_ptep_get_and_clear(struct mm_struct *mm,
@@ -171,7 +171,7 @@ pte_t __huge_ptep_get_and_clear(struct mm_struct *mm,pmd_t *pmdp = (pmd_t *) ptep;pud_t *pudp = (pud_t *) ptep; -if ((pte_val (*ptep) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R3) +if ((pte_val (ptep_get (ptep)) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R3)pudp_xchg_direct (mm, addr, pudp,__pud (_REGION3_ENTRY_EMPTY));else pmdp_xchg_direct (mm, addr, pmdp,__pmd (_SEGMENT_ENTRY_EMPTY));
@@ -209,13 +209,13 @@ pte_t *huge_pte_offset(struct mm_struct *mm,pmd_t *pmdp =NULL ; pgdp =pgd_offset (mm, addr); -if (pgd_present (*pgdp)) { +if (pgd_present (pgdp_get (pgdp))) { p4dp =p4d_offset (pgdp, addr); -if (p4d_present (*p4dp)) { +if (p4d_present (p4dp_get (p4dp))) { pudp =pud_offset (p4dp, addr);if (sz ==PUD_SIZE )return (pte_t *)pudp; -if (pud_present (*pudp)) +if (pud_present (pudp_get (pudp))) pmdp =pmd_offset (pudp, addr); } }
arch/s390/mm/pageattr.c +25 -20
129 lines not on this page · raw message
arch/s390/mm/vmem.c +46 -36
253 lines not on this page · raw message
Signature · 1 line
2.53.0
Alexander Gordeev
maintainerPATCH v3 3/7 s390/mm: Batch PTE updates in lazy MMU mode
In reply to Alexander GordeevMake use of the IPTE instruction's "Additional Entries" field to invalidate multiple PTEs in one go while in lazy MMU mode. This is the mode in which many memory-management system calls (like mremap(), mprotect(), etc.) update memory attributes.
To achieve that, the set_pte() and ptep_get() primitives use a per-CPU cache to store and retrieve PTE values and apply the cached values to the real page table once lazy MMU mode is left.
The same is done for memory-management platform callbacks that would otherwise cause intense per-PTE IPTE traffic, reducing the number of IPTE instructions from up to PTRS_PER_PTE to a single instruction in the best case. The average reduction is of course smaller.
Since all existing page table iterators called in lazy MMU mode handle one table at a time, the per-CPU cache does not need to be larger than PTRS_PER_PTE entries. That also naturally aligns with the IPTE instruction, which must not cross a page table boundary.
Before this change, the system calls did:
lazy_mmu_mode_enable_pte() ... <update PTEs> // up to PTRS_PER_PTE single-IPTEs ... lazy_mmu_mode_disable()
With this change, the system calls do:
lazy_mmu_mode_enable_pte()
...
<store new PTE values in the per-CPU cache>
...
lazy_mmu_mode_disable() // apply cache with one multi-IPTE
When applied to large memory ranges, some system calls show significant speedups:
mprotect() ~15x
munmap() ~3x
mremap() ~28xAt the same time, fork() shows a measurable slowdown of ~1.5x.
The overall results depend on memory size and access patterns, but the change generally does not degrade performance.
In addition to a process-wide impact, the rework affects the whole Central Electronics Complex (CEC). Each (global) IPTE instruction initiates a quiesce state in a CEC, so reducing the number of IPTE calls relieves CEC-wide quiesce traffic.
In an extreme case of mprotect() contiguously triggering the quiesce state on four LPARs in parallel, measurements show ~25x fewer quiesce events.
--- arch/s390/Kconfig | 1 + arch/s390/include/asm/lazy_mmu.h | 9 + arch/s390/include/asm/lowcore.h | 2 +- arch/s390/include/asm/pgtable.h | 157 +++++++++++-- arch/s390/kernel/setup.c | 2 + arch/s390/kernel/smp.c | 7 + arch/s390/mm/Makefile | 2 +- arch/s390/mm/lazy_mmu.c | 382 +++++++++++++++++++++++++++++++ arch/s390/mm/pgtable.c | 8 +- 9 files changed, 546 insertions(+), 24 deletions(-) create mode 100644 arch/s390/include/asm/lazy_mmu.h create mode 100644 arch/s390/mm/lazy_mmu.c
arch/s390/Kconfig +1 -0
@@ -97,6 +97,7 @@ config S390 select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_GIGANTIC_PAGE select ARCH_HAS_KCOV + select ARCH_HAS_LAZY_MMU_MODE select ARCH_HAS_MEMBARRIER_SYNC_CORE select ARCH_HAS_MEM_ENCRYPT select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
arch/s390/include/asm/lazy_mmu.h new file +9 -0
@@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __LAZY_MMU_H +#define __LAZY_MMU_H + +void lazy_mmu_online_boot_cpu (void ); +int lazy_mmu_online_cpu (gfp_t gfp,unsigned int cpu); +void lazy_mmu_offline_cpu (unsigned int cpu); + +#endif /* __LAZY_MMU_H */
arch/s390/include/asm/lowcore.h +1 -1
@@ -163,7 +163,7 @@ struct lowcore {__s32 preempt_count;/* 0x03a8 */ __u32 spinlock_lockval;/* 0x03ac */ __u32 spinlock_index;/* 0x03b0 */ -__u8 pad_0x03b4[0x03b8 -0x03b4 ];/* 0x03b4 */ +__s32 lazy_mmu_count;/* 0x03b4 */ __u64 percpu_offset;/* 0x03b8 */ __u8 percpu_register;/* 0x03c0 */ __u8 pad_0x03c1[0x0400 -0x03c1 ];/* 0x03c1 */
arch/s390/include/asm/pgtable.h +139 -18
220 lines not on this page · raw message
arch/s390/kernel/setup.c +2 -0
@@ -77,6 +77,7 @@#include <asm/maccess.h> #include <asm/uv.h> #include <asm/asm-offsets.h> +#include <asm/lazy_mmu.h> #include "entry.h" /*
@@ -1012,5 +1013,6 @@ void __init setup_arch(char **cmdline_p)void __init arch_cpu_finalize_init (void ) { +lazy_mmu_online_boot_cpu ();sclp_init (); }
arch/s390/kernel/smp.c +7 -0
@@ -60,6 +60,7 @@#include <asm/topology.h> #include <asm/vdso.h> #include <asm/maccess.h> +#include <asm/lazy_mmu.h> #include "entry.h" enum {
@@ -867,6 +868,11 @@ int __cpu_up(unsigned int cpu, struct task_struct *tidle) rc =pcpu_alloc_lowcore (pcpu, cpu);if (rc)return rc; + rc =lazy_mmu_online_cpu (GFP_KERNEL , cpu); +if (rc) { +pcpu_free_lowcore (pcpu, cpu); +return rc; + }/* * Make sure global control register contents do not change * until new CPU has initialized control registers.
@@ -922,6 +928,7 @@ void __cpu_die(unsigned int cpu) pcpu =per_cpu_ptr (&pcpu_devices, cpu);while (!pcpu_stopped (pcpu))cpu_relax (); +lazy_mmu_offline_cpu (cpu);pcpu_free_lowcore (pcpu, cpu);cpumask_clear_cpu (cpu,mm_cpumask (&init_mm));cpumask_clear_cpu (cpu, &init_mm.context.cpu_attach_mask);
arch/s390/mm/Makefile +1 -1
@@ -3,7 +3,7 @@ # Makefile for the linux s390-specific parts of the memory manager. # -obj-y := init.o fault.o extmem.o mmap.o vmem.o maccess.o +obj-y := init.o fault.o extmem.o mmap.o vmem.o maccess.o lazy_mmu.o obj-y += page-states.o pageattr.o pgtable.o pgalloc.o extable.o obj-$(CONFIG_CMM) += cmm.o
arch/s390/mm/lazy_mmu.c new file +382 -0
382 lines not on this page · raw message
arch/s390/mm/pgtable.c +4 -4
@@ -170,14 +170,14 @@ pte_t ptep_xchg_lazy(struct mm_struct *mm, unsigned long addr, }EXPORT_SYMBOL (ptep_xchg_lazy); -pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr, -pte_t *ptep) +pte_t ___ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep) {return ptep_flush_lazy (vma->vm_mm, addr, ptep,1 ); } -void ptep_modify_prot_commit (struct vm_area_struct *vma,unsigned long addr, -pte_t *ptep,pte_t old_pte,pte_t pte) +void ___ptep_modify_prot_commit (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t old_pte,pte_t pte) {if (pte_present (pte)) pte =clear_pte_bit (pte,__pgprot (_PAGE_UNUSED));
Signature · 1 line
2.53.0
Alexander Gordeev
maintainerPATCH v3 4/7 mm/gup: Cleanup pgtable entry accessors
In reply to Alexander GordeevPMD and PUD entries revalidation has the same semantics as PTE entry revalidation. Convert the remaining direct entry dereferences to the corresponding accessors.
The PTE validation in gup_fast_pte_range() is inconsistent with the prior value acquisition in the sense that it drops the lockless access semantics.
Use the lockless accessor not only for the PTE, but also for the PMD validation, which is likewise inconsistent with the prior value acquisition in gup_fast_pmd_range().
--- mm/gup.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
mm/gup.c +4 -4
@@ -2865,8 +2865,8 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,if (!folio) goto pte_unmap; -if (unlikely (pmd_val (pmd) !=pmd_val (*pmdp)) || -unlikely (pte_val (pte) !=pte_val (ptep_get (ptep)))) { +if (unlikely (pmd_val (pmd) !=pmd_val (pmdp_get_lockless (pmdp))) || +unlikely (pte_val (pte) !=pte_val (ptep_get_lockless (ptep)))) {gup_put_folio (folio,1 , flags); goto pte_unmap; }
@@ -2942,7 +2942,7 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr,if (!folio)return 0 ; -if (unlikely (pmd_val (orig) !=pmd_val (*pmdp))) { +if (unlikely (pmd_val (orig) !=pmd_val (pmdp_get_lockless (pmdp)))) {gup_put_folio (folio, refs, flags);return 0 ; }
@@ -2985,7 +2985,7 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr,if (!folio)return 0 ; -if (unlikely (pud_val (orig) !=pud_val (*pudp))) { +if (unlikely (pud_val (orig) !=pud_val (pudp_get (pudp)))) {gup_put_folio (folio, refs, flags);return 0 ; }
Signature · 1 line
2.53.0
Alexander Gordeev
maintainerPATCH v3 5/7 mm/page_vma_mapped_walk: Use ptep_get_lockless() for lockless access
In reply to Alexander GordeevSwitch from ptep_get() to ptep_get_lockless() accessor for PTE reads when no lock is taken.
--- mm/page_vma_mapped.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
mm/page_vma_mapped.c +7 -2
@@ -41,7 +41,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,if (!pvmw->pte)return false; - ptent =ptep_get (pvmw->pte); + ptent =ptep_get_lockless (pvmw->pte);if (pte_none (ptent)) {return false;
@@ -183,6 +183,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)struct mm_struct *mm = vma->vm_mm;unsigned long end;spinlock_t *ptl; +pte_t pteval;pgd_t *pgd;p4d_t *p4d;pud_t *pud;
@@ -310,7 +311,11 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) goto restart; } pvmw->pte++; - }while (pte_none (ptep_get (pvmw->pte))); +if (!pvmw->ptl) + pteval =ptep_get_lockless (pvmw->pte); +else + pteval =ptep_get (pvmw->pte); + }while (pte_none (pteval));if (!pvmw->ptl) {spin_lock (ptl);
Signature · 1 line
2.53.0
Alexander Gordeev
maintainerPATCH v3 6/7 mm/kasan: Introduce helpers for lazy MMU mode sanitizer
In reply to Alexander GordeevProvide helpers that allow architectures implement illegitimate PTE direct accesses while the lazy MMU mode is enabled, such as:
pte_t pte = *ptep; *ptep = pte;
By contrast, these would have to be:
pte_t pte = ptep_get(ptep); set_pte(ptep, pte);
The direct PTE accesses pose a real issue on s390.
--- include/linux/kasan.h | 16 ++++++++++++++++ mm/kasan/common.c | 10 ++++++++++ mm/kasan/kasan.h | 2 ++ 3 files changed, 28 insertions(+)
include/linux/kasan.h +16 -0
@@ -134,6 +134,20 @@ static __always_inline void kasan_poison_slab(struct slab *slab)__kasan_poison_slab (slab); } +void __kasan_poison_pte (pte_t *pte,int nr); +static __always_inline void kasan_poison_pte(pte_t *pte, int nr) +{ +if (kasan_enabled ()) +__kasan_poison_pte (pte, nr); +} + +void __kasan_unpoison_pte (pte_t *pte,int nr); +static __always_inline void kasan_unpoison_pte(pte_t *pte, int nr) +{ +if (kasan_enabled ()) +__kasan_unpoison_pte (pte, nr); +} +void __kasan_unpoison_new_object (struct kmem_cache *cache,void *object);/** * kasan_unpoison_new_object - Temporarily unpoison a new slab object.
@@ -414,6 +428,8 @@ static inline bool kasan_unpoison_pages(struct page *page, unsigned int order,return false; }static inline void kasan_poison_slab (struct slab *slab) {} +static inline void kasan_poison_pte (pte_t *pte,int nr) {} +static inline void kasan_unpoison_pte (pte_t *pte,int nr) {}static inline void kasan_unpoison_new_object (struct kmem_cache *cache,void *object) {} static inline void kasan_poison_new_object(struct kmem_cache *cache,
mm/kasan/common.c +10 -0
@@ -163,6 +163,16 @@ void __kasan_poison_slab(struct slab *slab) KASAN_SLAB_REDZONE, false); } +void __kasan_poison_pte (pte_t *pte,int nr) +{ +kasan_poison (pte,sizeof (*pte) * nr,KASAN_LAZY_MMU_PTE , false); +} + +void __kasan_unpoison_pte (pte_t *pte,int nr) +{ +kasan_unpoison (pte,sizeof (*pte) * nr, false); +} +void __kasan_unpoison_new_object (struct kmem_cache *cache,void *object) {kasan_unpoison (object, cache->object_size, false);
mm/kasan/kasan.h +2 -0
@@ -144,12 +144,14 @@ static inline bool kasan_requires_meta(void)#define KASAN_PAGE_REDZONE 0xFE/* redzone for kmalloc_large allocation */ #define KASAN_SLAB_REDZONE 0xFC/* redzone for slab object */ #define KASAN_SLAB_FREE 0xFB/* freed slab object */ +#define KASAN_LAZY_MMU_PTE 0xFD#define KASAN_VMALLOC_INVALID 0xF8/* inaccessible space in vmap area */ #else #define KASAN_PAGE_FREE KASAN_TAG_INVALID#define KASAN_PAGE_REDZONE KASAN_TAG_INVALID#define KASAN_SLAB_REDZONE KASAN_TAG_INVALID#define KASAN_SLAB_FREE KASAN_TAG_INVALID +#define KASAN_LAZY_MMU_PTE KASAN_TAG_INVALID#define KASAN_VMALLOC_INVALID KASAN_TAG_INVALID/* only used for SW_TAGS */ #endif
Signature · 1 line
2.53.0
Alexander Gordeev
maintainerPATCH v3 7/7 s390/mm: Lazy MMU mode sanitizer
In reply to Alexander GordeevDetect PTE entries access in lazy MMU mode by means other than set_pte() and ptep_get() primitives, which would be a read hazard.
The access to kasan shadow memory from ptep_get_lockless() mistakenly hits invalid access in case a concurrent lazy MMU access to the same PTE is happening. To avoid that disable instrumentation for ptep_get_lockless() altogether.
--- arch/s390/include/asm/pgtable.h | 6 ++++++ arch/s390/mm/lazy_mmu.c | 27 +++++++++++++++++++++++---- 2 files changed, 29 insertions(+), 4 deletions(-)
arch/s390/include/asm/pgtable.h +6 -0
@@ -1047,6 +1047,12 @@ static inline void set_pte(pte_t *ptep, pte_t pte)__set_pte (ptep, pte); } +#define ptep_get_lockless ptep_get_lockless +static inline __no_sanitize_address pte_t ptep_get_lockless(pte_t *ptep) +{ +return READ_ONCE (*ptep); +} +static inline pte_t __ptep_get (pte_t *ptep) {return READ_ONCE (*ptep);
arch/s390/mm/lazy_mmu.c +23 -4
@@ -63,10 +63,13 @@ static int invalidate_pte_range(struct mm_struct *mm, unsigned long addr, }static void set_pte_range (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t *end,pte_t *cache) +pte_t *start,pte_t *end,pte_t *cache) { -int i, nr_ptes; +int nr_ptes, nr_total = end - start; +pte_t *ptep = start; +int i; +kasan_unpoison_pte (start, nr_total);while (ptep < end) { nr_ptes =invalidate_pte_range (mm, addr, ptep, end);
@@ -77,6 +80,7 @@ static void set_pte_range(struct mm_struct *mm, unsigned long addr, addr += nr_ptes *PAGE_SIZE ; } +kasan_poison_pte (start, nr_total); }static void enter_ipte_norange (void )
@@ -94,6 +98,7 @@ static void enter_ipte_range(struct mm_struct *mm, unsigned long addr, unsigned long end, pte_t *pte) { struct ipte_range *range; +unsigned int nr_ptes;if (!test_facility (13 ))return ;
@@ -105,6 +110,9 @@ static void enter_ipte_range(struct mm_struct *mm, range->base_addr = addr; range->base_end = end; range->base_pte = pte; + + nr_ptes = (range->base_end - range->base_addr) /PAGE_SIZE ; +kasan_poison_pte (range->base_pte, nr_ptes); }static void leave_ipte_range (void )
@@ -112,6 +120,7 @@ static void leave_ipte_range(void)pte_t *ptep, *start, *start_cache, *cache;unsigned long start_addr, addr;struct ipte_range *range; +unsigned int nr_ptes;int start_idx;if (!test_facility (13 ))
@@ -148,6 +157,9 @@ static void leave_ipte_range(void) range->end_pte =NULL ; done: + nr_ptes = (range->base_end - range->base_addr) /PAGE_SIZE ; +kasan_unpoison_pte (range->base_pte, nr_ptes); + range->mm =NULL ; range->base_addr =0 ; range->base_end =0 ;
@@ -227,10 +239,17 @@ static void __ipte_range_set_pte(struct ipte_range *range, pte_t *ptep, pte_t ptstatic pte_t __ipte_range_ptep_get (struct ipte_range *range,pte_t *ptep) {unsigned int idx = ptep - range->base_pte; +pte_t pte;lockdep_assert_preemption_disabled (); -if (pte_val (range->cache[idx]) ==PTE_POISON ) -return __ptep_get (ptep); +if (pte_val (range->cache[idx]) ==PTE_POISON ) { +kasan_unpoison_pte (ptep,1 ); + pte =__ptep_get (ptep); +kasan_poison_pte (ptep,1 ); + +return pte; + } +return range->cache[idx]; }
Signature · 1 line
2.53.0