Skip to contentSkip to conversation

s390/mm: Batch PTE updates in lazy MMU mode

PATCH Superseded Alexander Gordeev 4 participants 28 messages Aug 17 – Oct 2

Subsystems (9)

v7 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Alexander Gordeev

author maintainer
#0 lore raw

v7 cover s390/mm: Batch PTE updates in lazy MMU mode

Hi All!

This is v7 of the batched PTE updates in lazy MMU mode rework.

The presented implementation sets up per-cpu caches in the s390-specific hotplug callbacks as opposed to CPUHP_BP_PREPARE_DYN hooks. I like this approach better, since the boot CPU setup is architecture-specific anyway and the whole SMP-related lowcore initialization is handled in one place.

Heiko Carstens requested some mechanism that would rule out direct dereferencing of PTE pointers, which otherwise would bypass the per- cpu cache on s390 and lead to catastrophic results.

To address this request a sparse rework was suggested:
https://lore.kernel.org/linux-mm/[email protected]/

But an altenative (huge) MM rework looks as a superior solution:
https://lore.kernel.org/linux-mm/[email protected]/
https://lore.kernel.org/linux-mm/[email protected]/

Changes since v6:
- bottom-halves are disabled on entering and leaving the lazy mmu mode

Changes since v5:
- IPTE optimization is not applied to secure guests [4]
- __kasan_(un)poison_pte() are marked as EXPORT_SYMBOL_GPL() [5]
- PTE table poisoning is not applied to architectures with PTE entry sizes
unaligned on KASAN_GRANULE_SIZE [5]
4. https://lore.kernel.org/linux-s390/[email protected]/T/#md98724bd3b66d0a0711deb6089fa541f420566d8
5. https://lore.kernel.org/linux-s390/[email protected]/T/#m9c8b31b4863416732d0cbbc5f5f63290db4522c1

Changes since v4:
- verified that a presumable sashiko performance regression finding [2] although appears valid does not really degrade performance
- applied sashiko suggestion [3] and added "direct-pte-access" kasan bug type
2. https://lore.kernel.org/linux-s390/[email protected]/#r
3. https://lore.kernel.org/linux-s390/[email protected]/

Changes since v3:
- all prerequisite patches are landed in -next and removed from the series

Changes since v2:
- lazy_mmu_mode_enable_for_pte_range() renamed to lazy_mmu_mode_enable_with_ptes() (David Hildenbrand)
- patch "mm/pgtable: Fix bogus comment to clear_not_present_full_ptes()" is dropped (David Hildenbrand)
- direct PTE dereferencing KASAN sanitizer added (Heiko Carstens)
- CONFIG_IPTE_BATCH option is dropped (Heiko Carstens)
- PTE_POISON changed from zero to 0x800 (Heiko Carstens)
- allocate per-cpu caches on CPU hot-plug (Heiko Carstens)
- introduced a lowcore field for fast lazy mode checking (Heiko Carstens)
- few minor code changes (Heiko Carstens)

Changes since v1:
- lazy_mmu_mode_enable_pte() renamed to lazy_mmu_mode_enable_for_pte_range()
- lazy_mmu_mode_enable_for_pte_range() semantics clarified
- some sashiko comments addressed [1] including one bug fix [1]
- patches 2-4 added
1. https://sashiko.dev/#/patchset/cover.1774420056.git.agordeev%40linux.ibm.com

This series addresses an s390-specific aspect of how page table entries are modified. In many cases, changing a valid PTE (for example, setting or clearing a hardware bit) requires issuing an Invalidate Page Table Entry (IPTE) instruction beforehand.

A disadvantage of the IPTE instruction is that it may initiate a machine-wide quiesce state. This state acts as an expensive global hardware lock and should be avoided whenever possible.

Currently, IPTE is invoked for each individual PTE update in most code paths. However, the instruction itself supports invalidating multiple PTEs at once, covering up to 256 entries. Using this capability can significantly reduce the number of quiesce events, with a positive impact on overall system performance. At present, this feature is not utilized.

An effort was therefore made to identify kernel code paths that update large numbers of consecutive PTEs. Such updates can be batched and handled by a single IPTE invocation, leveraging the hardware support described above.

A natural candidate for this optimization is page-table walkers that change attributes of memory ranges and thus modify contiguous ranges of PTEs. Many memory-management system calls enter lazy MMU mode while updating such ranges.

This lazy MMU mode can be leveraged to build on the already existing infrastructure and implement a software-level lazy MMU mechanism, allowing expensive PTE invalidations on s390 to be batched.

Alexander Gordeev (4):
mm: Make lazy MMU mode context-aware
s390/mm: Batch PTE updates in lazy MMU mode mm/kasan: Introduce helpers for lazy MMU mode sanitizer s390/mm: Lazy MMU mode sanitizer

 arch/s390/Kconfig                |   1 +
 arch/s390/include/asm/lazy_mmu.h |   9 +
 arch/s390/include/asm/lowcore.h  |   2 +-
 arch/s390/include/asm/pgtable.h  | 157 ++++++++++--
 arch/s390/kernel/setup.c         |   2 +
 arch/s390/kernel/smp.c           |   7 +
 arch/s390/mm/Makefile            |   1 +
 arch/s390/mm/lazy_mmu.c          | 418 +++++++++++++++++++++++++++++++
 arch/s390/mm/pgtable.c           |   8 +-
 fs/proc/task_mmu.c               |   2 +-
 include/linux/kasan.h            |  19 +-
 include/linux/pgtable.h          |  46 ++++
 mm/kasan/common.c                |  14 ++
 mm/kasan/kasan.h                 |   2 +
 mm/kasan/report_generic.c        |   3 +
 mm/madvise.c                     |   8 +-
 mm/memory.c                      |   8 +-
 mm/mprotect.c                    |   2 +-
 mm/mremap.c                      |   2 +-
 mm/vmalloc.c                     |   6 +-
 20 files changed, 678 insertions(+), 39 deletions(-)
 create mode 100644 arch/s390/include/asm/lazy_mmu.h
 create mode 100644 arch/s390/mm/lazy_mmu.c
Signature · 1 line
2.53.0
author maintainer
#1 lore raw

PATCH v7 1/4 mm: Make lazy MMU mode context-aware

Lazy MMU mode is assumed to be context-independent, in the sense that it does not need any additional information while operating. However, the s390 architecture benefits from knowing the exact page table entries being modified.

Introduce lazy_mmu_mode_enable_with_ptes(), which is provided with the process address space and the page table being operated on. This information is required to enable s390-specific optimizations.

The function takes parameters that are typically passed to page- table level walkers, which implies that the span of PTE entries never crosses a page table boundary.

Architectures that do not require such information simply do not need to define the lazy_mmu_mode_enable_with_ptes() callback.

---
 fs/proc/task_mmu.c      |  2 +-
 include/linux/pgtable.h | 46 +++++++++++++++++++++++++++++++++++++++++
 mm/madvise.c            |  8 +++----
 mm/memory.c             |  8 +++----
 mm/mprotect.c           |  2 +-
 mm/mremap.c             |  2 +-
 mm/vmalloc.c            |  6 +++---
 7 files changed, 60 insertions(+), 14 deletions(-)
fs/proc/task_mmu.c +1 -1
index 229d1fc3d7f1..916785c3b2e0 100644
@@ -2852,7 +2852,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
 		return 0;
 	}
 
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(vma->vm_mm, start, end, start_pte);
 
 	if ((p->arg.flags & PM_SCAN_WP_MATCHING) && !p->vec_out) {
 		/* Fast path for performing exclusive WP */
include/linux/pgtable.h +46 -0

58 lines not on this page · raw message

mm/madvise.c +4 -4
index 77552b03d318..4419d16b39f9 100644
@@ -452,7 +452,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 	if (!start_pte)
 		return 0;
 	flush_tlb_batched_pending(mm);
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(mm, addr, end, start_pte);
 	for (; addr < end; pte += nr, addr += nr * PAGE_SIZE) {
 		nr = 1;
 		ptent = ptep_get(pte);
@@ -507,7 +507,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 				if (!start_pte)
 					break;
 				flush_tlb_batched_pending(mm);
-				lazy_mmu_mode_enable();
+				lazy_mmu_mode_enable_with_ptes(mm, addr, end, start_pte);
 				if (!err)
 					nr = 0;
 				continue;
@@ -674,7 +674,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
 	if (!start_pte)
 		return 0;
 	flush_tlb_batched_pending(mm);
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(mm, addr, end, start_pte);
 	for (; addr != end; pte += nr, addr += PAGE_SIZE * nr) {
 		nr = 1;
 		ptent = ptep_get(pte);
@@ -734,7 +734,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
 				if (!start_pte)
 					break;
 				flush_tlb_batched_pending(mm);
-				lazy_mmu_mode_enable();
+				lazy_mmu_mode_enable_with_ptes(mm, addr, end, pte);
 				if (!err)
 					nr = 0;
 				continue;
mm/memory.c +4 -4
index 6b8280cfc1db..0cc07b043384 100644
@@ -1272,7 +1272,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
 	spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
 	orig_src_pte = src_pte;
 	orig_dst_pte = dst_pte;
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(src_mm, addr, end, src_pte);
 
 	do {
 		nr = 1;
@@ -1922,7 +1922,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 		return addr;
 
 	flush_tlb_batched_pending(mm);
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(mm, addr, end, start_pte);
 	do {
 		bool any_skipped = false;
 
@@ -2919,7 +2919,7 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd,
 	mapped_pte = pte = pte_alloc_map_lock(mm, pmd, addr, &ptl);
 	if (!pte)
 		return -ENOMEM;
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(mm, addr, end, mapped_pte);
 	do {
 		BUG_ON(!pte_none(ptep_get(pte)));
 		if (!pfn_modify_allowed(pfn, prot)) {
@@ -3330,7 +3330,7 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,
 			return -EINVAL;
 	}
 
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(mm, addr, end, mapped_pte);
 
 	if (fn) {
 		do {
mm/mprotect.c +1 -1
index 9cbf932b028c..3fc26418e837 100644
@@ -337,7 +337,7 @@ static long change_pte_range(struct mmu_gather *tlb,
 		is_private_single_threaded = vma_is_single_threaded_private(vma);
 
 	flush_tlb_batched_pending(vma->vm_mm);
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(vma->vm_mm, addr, end, pte);
 	do {
 		nr_ptes = 1;
 		oldpte = ptep_get(pte);
mm/mremap.c +1 -1
index e9c8b1d05832..0dfe3de39ccc 100644
@@ -260,7 +260,7 @@ static int move_ptes(struct pagetable_move_control *pmc,
 	if (new_ptl != old_ptl)
 		spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
 	flush_tlb_batched_pending(vma->vm_mm);
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(mm, old_addr, old_end, old_ptep);
 
 	for (; old_addr < old_end; old_ptep += nr_ptes, old_addr += nr_ptes * PAGE_SIZE,
 		new_ptep += nr_ptes, new_addr += nr_ptes * PAGE_SIZE) {
mm/vmalloc.c +3 -3
index f4fa227a8d7f..1c6ef8fd079e 100644
@@ -109,7 +109,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 	if (!pte)
 		return -ENOMEM;
 
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(&init_mm, addr, end, pte);
 
 	do {
 		if (unlikely(!pte_none(ptep_get(pte)))) {
@@ -393,7 +393,7 @@ static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 	unsigned long size = PAGE_SIZE;
 
 	pte = pte_offset_kernel(pmd, addr);
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(&init_mm, addr, end, pte);
 
 	do {
 #ifdef CONFIG_HUGETLB_PAGE
@@ -560,7 +560,7 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr,
 	if (!pte)
 		return -ENOMEM;
 
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(&init_mm, addr, end, pte);
 
 	do {
 		struct page *page = pages[*nr];
Signature · 1 line
2.53.0
#2 lore raw

Re: [PATCH v7 1/4] mm: Make lazy MMU mode context-aware

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Setting walk->action = ACTION_AGAIN when pte_offset_map_lock() fails creates an infinite loop for non-present PMD migration entries. --

--- Patch [1]: [PATCH v7 1/4] mm: Make lazy MMU mode context-aware --- commit 1d7b684aa5a3ef0f53973a6ad48d5f71e845692f
Author: Alexander Gordeev <[email protected]>

mm: Make lazy MMU mode context-aware

This commit introduces lazy_mmu_mode_enable_with_ptes(), which provides the process address space and page table being operated on. This enables s390 and other architectures to apply specific optimizations when entering lazy MMU mode.

fs/proc/task_mmu.c static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, · line 2886

Quoted 10 lines
fs/proc/task_mmu.c +1 -1
index 5c54aebe21182..b8c8d7dda0950 100644
@@ -2883,7 +2883,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
 		return 0;
 	}
 
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_with_ptes(vma->vm_mm, start, end, start_pte);

[Severity: High]
This is a pre-existing issue, but since you are modifying pagemap_scan_pmd_entry() here, I wanted to ask about the page table lock failure path just above this change.

If the lock fails to map the page table (for instance, because of a non-present PMD migration entry during concurrent transparent hugepage migration), the code sets the walk action to retry:

    if (!pte) {
        walk->action = ACTION_AGAIN;
        return 0;
    }

Because pte_offset_map_lock() will fail non-transiently for migration entries, does setting ACTION_AGAIN here create an infinite loop?

Could a local user trigger this by reading /proc/pid/pagemap concurrently with a page migration, causing the thread to hang indefinitely? It seems that gracefully skipping the failure by returning 0 without setting ACTION_AGAIN might be safer.

This same ACTION_AGAIN retry pattern on lock failure appears in several other callbacks:
- smaps_pte_range(), clear_refs_pte_range(), and pagemap_pmd_range() in fs/proc/task_mmu.c
- mincore_pte_range() in mm/mincore.c
- mlock_pte_range() in mm/mlock.c
- queue_folios_pte_range() in mm/mempolicy.c

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author maintainer
#3 lore raw

PATCH v7 2/4 s390/mm: Batch PTE updates in lazy MMU mode

In reply to Alexander Gordeev

Make use of the IPTE instruction's "Additional Entries" field to invalidate multiple PTEs in one go while in lazy MMU mode. This is the mode in which many memory-management system calls (like mremap(), mprotect(), etc.) update memory attributes.

To achieve that, the set_pte() and ptep_get() primitives use a per-CPU cache to store and retrieve PTE values and apply the cached values to the real page table once lazy MMU mode is left.

The same is done for memory-management platform callbacks that would otherwise cause intense per-PTE IPTE traffic, reducing the number of IPTE instructions from up to PTRS_PER_PTE to a single instruction in the best case. The average reduction is of course smaller.

Since all existing page table iterators called in lazy MMU mode handle one table at a time, the per-CPU cache does not need to be larger than PTRS_PER_PTE entries. That also naturally aligns with the IPTE instruction, which must not cross a page table boundary.

Before this change, the system calls did:

	lazy_mmu_mode_enable_pte()
	...
	<update PTEs>		// up to PTRS_PER_PTE single-IPTEs
	...
	lazy_mmu_mode_disable()

With this change, the system calls do:

lazy_mmu_mode_enable_pte()
...
<store new PTE values in the per-CPU cache>
...
lazy_mmu_mode_disable() // apply cache with one multi-IPTE

When applied to large memory ranges, some system calls show significant speedups:

    mprotect()    ~15x
    munmap()      ~3x
    mremap()      ~28x

The overall results depend on memory size and access patterns, but the change generally does not degrade performance.

In addition to a process-wide impact, the rework affects the whole Central Electronics Complex (CEC). Each (global) IPTE instruction initiates a quiesce state in a CEC, so reducing the number of IPTE calls relieves CEC-wide quiesce traffic.

In an extreme case of mprotect() contiguously triggering the quiesce state on four LPARs in parallel, measurements show ~25x fewer quiesce events.

---
 arch/s390/Kconfig                |   1 +
 arch/s390/include/asm/lazy_mmu.h |   9 +
 arch/s390/include/asm/lowcore.h  |   2 +-
 arch/s390/include/asm/pgtable.h  | 151 ++++++++++--
 arch/s390/kernel/setup.c         |   2 +
 arch/s390/kernel/smp.c           |   7 +
 arch/s390/mm/Makefile            |   1 +
 arch/s390/mm/lazy_mmu.c          | 399 +++++++++++++++++++++++++++++++
 arch/s390/mm/pgtable.c           |   8 +-
 9 files changed, 557 insertions(+), 23 deletions(-)
 create mode 100644 arch/s390/include/asm/lazy_mmu.h
 create mode 100644 arch/s390/mm/lazy_mmu.c
arch/s390/Kconfig +1 -0
index 84404e6778d5..7846332dcd0a 100644
@@ -97,6 +97,7 @@ config S390
 	select ARCH_HAS_GCOV_PROFILE_ALL
 	select ARCH_HAS_GIGANTIC_PAGE
 	select ARCH_HAS_KCOV
+	select ARCH_HAS_LAZY_MMU_MODE
 	select ARCH_HAS_MEMBARRIER_SYNC_CORE
 	select ARCH_HAS_MEM_ENCRYPT
 	select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
arch/s390/include/asm/lazy_mmu.h new file +9 -0
new file mode 100644
index 000000000000..98366e9de9bc
@@ -0,0 +1,9 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __LAZY_MMU_H
+#define __LAZY_MMU_H
+
+void lazy_mmu_online_boot_cpu(void);
+int lazy_mmu_online_cpu(gfp_t gfp, unsigned int cpu);
+void lazy_mmu_offline_cpu(unsigned int cpu);
+
+#endif /* __LAZY_MMU_H */
arch/s390/include/asm/lowcore.h +1 -1
index 3b3ecc647993..dba236664da9 100644
@@ -163,7 +163,7 @@ struct lowcore {
 	__s32	preempt_count;			/* 0x03a8 */
 	__u32	spinlock_lockval;		/* 0x03ac */
 	__u32	spinlock_index;			/* 0x03b0 */
-	__u8	pad_0x03b4[0x03b8-0x03b4];	/* 0x03b4 */
+	__s32	lazy_mmu_count;			/* 0x03b4 */
 	__u64	percpu_offset;			/* 0x03b8 */
 	__u8	percpu_register;		/* 0x03c0 */
 	__u8	pad_0x03c1[0x0400-0x03c1];	/* 0x03c1 */
arch/s390/include/asm/pgtable.h +133 -18

216 lines not on this page · raw message

arch/s390/kernel/setup.c +2 -0
index b60284328fe3..f5a3c9e1b6b8 100644
@@ -77,6 +77,7 @@
 #include <asm/maccess.h>
 #include <asm/uv.h>
 #include <asm/asm-offsets.h>
+#include <asm/lazy_mmu.h>
 #include "entry.h"
 
 /*
@@ -1012,5 +1013,6 @@ void __init setup_arch(char **cmdline_p)
 
 void __init arch_cpu_finalize_init(void)
 {
+	lazy_mmu_online_boot_cpu();
 	sclp_init();
 }
arch/s390/kernel/smp.c +7 -0
index 0ba7f89b8161..0a826bbaf1dd 100644
@@ -59,6 +59,7 @@
 #include <asm/topology.h>
 #include <asm/vdso.h>
 #include <asm/maccess.h>
+#include <asm/lazy_mmu.h>
 #include "entry.h"
 
 enum {
@@ -866,6 +867,11 @@ int __cpu_up(unsigned int cpu, struct task_struct *tidle)
 	rc = pcpu_alloc_lowcore(pcpu, cpu);
 	if (rc)
 		return rc;
+	rc = lazy_mmu_online_cpu(GFP_KERNEL, cpu);
+	if (rc) {
+		pcpu_free_lowcore(pcpu, cpu);
+		return rc;
+	}
 	/*
 	 * Make sure global control register contents do not change
 	 * until new CPU has initialized control registers.
@@ -921,6 +927,7 @@ void __cpu_die(unsigned int cpu)
 	pcpu = per_cpu_ptr(&pcpu_devices, cpu);
 	while (!pcpu_stopped(pcpu))
 		cpu_relax();
+	lazy_mmu_offline_cpu(cpu);
 	pcpu_free_lowcore(pcpu, cpu);
 	cpumask_clear_cpu(cpu, mm_cpumask(&init_mm));
 	cpumask_clear_cpu(cpu, &init_mm.context.cpu_attach_mask);
arch/s390/mm/Makefile +1 -0
index 193899c39ca7..96cac3e99f51 100644
@@ -4,6 +4,7 @@
 #
 
 obj-y		:= init.o fault.o extmem.o mmap.o vmem.o maccess.o
+obj-y		:= init.o fault.o extmem.o mmap.o vmem.o maccess.o lazy_mmu.o
 obj-y		+= page-states.o pageattr.o pgtable.o pgalloc.o extable.o
 
 obj-$(CONFIG_CMM)		+= cmm.o
arch/s390/mm/lazy_mmu.c new file +399 -0

399 lines not on this page · raw message

arch/s390/mm/pgtable.c +4 -4
index 4acd8b140c4b..df36523bcbbb 100644
@@ -166,14 +166,14 @@ pte_t ptep_xchg_lazy(struct mm_struct *mm, unsigned long addr,
 }
 EXPORT_SYMBOL(ptep_xchg_lazy);
 
-pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr,
-			     pte_t *ptep)
+pte_t ___ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr,
+				pte_t *ptep)
 {
 	return ptep_flush_lazy(vma->vm_mm, addr, ptep, 1);
 }
 
-void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr,
-			     pte_t *ptep, pte_t old_pte, pte_t pte)
+void ___ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr,
+				pte_t *ptep, pte_t old_pte, pte_t pte)
 {
 	set_pte(ptep, pte);
 }
Signature · 1 line
2.53.0
#4 lore raw
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines

Make use of the IPTE instruction's "Additional Entries" field to invalidate multiple PTEs in one go while in lazy MMU mode. This is the mode in which many memory-management system calls (like mremap(), mprotect(), etc.) update memory attributes.

To achieve that, the set_pte() and ptep_get() primitives use a per-CPU cache to store and retrieve PTE values and apply the cached values to the real page table once lazy MMU mode is left.

The same is done for memory-management platform callbacks that would otherwise cause intense per-PTE IPTE traffic, reducing the number of IPTE instructions from up to PTRS_PER_PTE to a single instruction in the best case. The average reduction is of course smaller.

Quoted 6 lines

[ ... ]
In an extreme case of mprotect() contiguously triggering the quiesce state on four LPARs in parallel, measurements show ~25x fewer quiesce events.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
maintainer
#5 lore raw
In reply to Alexander Gordeev

arch/s390/include/asm/lowcore.h struct lowcore { · line 166

Quoted 10 lines · Alexander Gordeev
arch/s390/include/asm/lowcore.h +1 -1
index 3b3ecc647993..dba236664da9 100644
@@ -163,7 +163,7 @@ struct lowcore {
 	__s32	preempt_count;			/* 0x03a8 */
 	__u32	spinlock_lockval;		/* 0x03ac */
 	__u32	spinlock_index;			/* 0x03b0 */
-	__u8	pad_0x03b4[0x03b8-0x03b4];	/* 0x03b4 */
+	__s32	lazy_mmu_count;			/* 0x03b4 */

Why is this signed? Can it get negative?

arch/s390/include/asm/lowcore.h struct lowcore {

Quoted 6 lines
arch/s390/include/asm/lowcore.h +6 -0
+static __always_inline bool is_lazy_mmu_active(void)
+{
+	if (__is_defined(__DECOMPRESSOR))
+		return false;
+	if (!get_lowcore()->lazy_mmu_count)
+		return false;

I guess there is opportunity to generate better code here using an alternative and using a flag output constraint too.

Quoted 11 lines
arch/s390/kernel/setup.c +1 -0
@@ -77,6 +77,7 @@
 #include <asm/maccess.h>
 #include <asm/uv.h>
 #include <asm/asm-offsets.h>
+#include <asm/lazy_mmu.h>
 #include "entry.h"
 
 /*

arch/s390/kernel/setup.c void __init setup_arch(char **cmdline_p) · line 1018

Quoted 6 lines
arch/s390/kernel/setup.c +1 -0
@@ -1012,5 +1013,6 @@ void __init setup_arch(char **cmdline_p)
 
 void __init arch_cpu_finalize_init(void)
 {
+	lazy_mmu_online_boot_cpu();
 	sclp_init();
 }

What makes this code so special that an explicit call from arch_cpu_finalize_init() is required? This is really the last resort if everything else fails. To me it looks like the code can be changed to use a new static key, and add a generic early (pre-smp) initcall to allocate memory for cpu 0, and if that succeeds enable the static key.

Quoted 23 lines
arch/s390/kernel/smp.c +6 -0
@@ -59,6 +59,7 @@
 #include <asm/topology.h>
 #include <asm/vdso.h>
 #include <asm/maccess.h>
+#include <asm/lazy_mmu.h>
 #include "entry.h"
 
 enum {
@@ -866,6 +867,11 @@ int __cpu_up(unsigned int cpu, struct task_struct *tidle)
 	rc = pcpu_alloc_lowcore(pcpu, cpu);
 	if (rc)
 		return rc;
+	rc = lazy_mmu_online_cpu(GFP_KERNEL, cpu);
+	if (rc) {
+		pcpu_free_lowcore(pcpu, cpu);
+		return rc;
+	}
 	/*
 	 * Make sure global control register contents do not change
 	 * until new CPU has initialized control registers.

arch/s390/kernel/smp.c void __cpu_die(unsigned int cpu) · line 930

Quoted 4 lines
arch/s390/kernel/smp.c +1 -0
@@ -921,6 +927,7 @@ void __cpu_die(unsigned int cpu)
 	pcpu = per_cpu_ptr(&pcpu_devices, cpu);
 	while (!pcpu_stopped(pcpu))
 		cpu_relax();
+	lazy_mmu_offline_cpu(cpu);

Same here: what makes this code so special that this needs to be open-coded into the low level cpu hotplug code? Everybody who needs to change this code in future will wonder why the mmu code is so special that it needs to be directly handled here, and then needs to understand the mmu code. And the answer is: there is no reason.

Please use a generic cpu hotplug notifier to avoid that maintenance get's more expensive.

Quoted 4 lines
arch/s390/kernel/smp.c +4 -0
+static void leave_ipte_range(void)
+{
+	pte_t *ptep, *start, *start_cache, *cache;
+	unsigned long start_addr, addr;

arch/s390/kernel/smp.c void __cpu_die(unsigned int cpu)

Quoted 10 lines
arch/s390/kernel/smp.c +10 -0
+	struct ipte_range *range;
+	int start_idx;
+
+	if (!test_facility(13))
+		return;
+
+	local_bh_disable();
+
+	lockdep_assert_preemption_disabled();
+	range = this_cpu_read(ipte_range);

Why is it required to disable bottom halves? A comment would be helpful. Or a hint in the commit message - this is not obvious.

Also at least for !PREEMPT_RT (which is always true for s390) lockdep_assert_preemption_disabled() is quite pointless, since local_bh_disable() just one line above disables preemption.

I guess you wanted to add that check above local_bh_disable()? If not you could as well remove it

author maintainer
#6 lore raw
Quoted 13 lines · Heiko Carstens

arch/s390/include/asm/lowcore.h struct lowcore { · line 166

Quoted 10 lines · Alexander Gordeev
arch/s390/include/asm/lowcore.h +1 -1
index 3b3ecc647993..dba236664da9 100644
@@ -163,7 +163,7 @@ struct lowcore {
 	__s32	preempt_count;			/* 0x03a8 */
 	__u32	spinlock_lockval;		/* 0x03ac */
 	__u32	spinlock_index;			/* 0x03b0 */
-	__u8	pad_0x03b4[0x03b8-0x03b4];	/* 0x03b4 */
+	__s32	lazy_mmu_count;			/* 0x03b4 */

Why is this signed? Can it get negative?

For the same reason preempt_count is signed, I guess.

No, it can not get negative and it is very handy to observe a disbalance in a crash (I did hit it indeed while debugging).

Quoted 9 lines

arch/s390/include/asm/lowcore.h struct lowcore {

Quoted 6 lines
arch/s390/include/asm/lowcore.h +6 -0
+static __always_inline bool is_lazy_mmu_active(void)
+{
+	if (__is_defined(__DECOMPRESSOR))
+		return false;
+	if (!get_lowcore()->lazy_mmu_count)
+		return false;

I guess there is opportunity to generate better code here using an alternative and using a flag output constraint too.

Will try.

Quoted 12 lines
Quoted 12 lines
arch/s390/kernel/setup.c +1 -0
@@ -77,6 +77,7 @@
 #include <asm/maccess.h>
 #include <asm/uv.h>
 #include <asm/asm-offsets.h>
+#include <asm/lazy_mmu.h>
 #include "entry.h"
 
 /*
@@ -1012,5 +1013,6 @@ void __init setup_arch(char **cmdline_p)
 
Quoted 11 lines

arch/s390/kernel/setup.c void __init setup_arch(char **cmdline_p) · line 1018

Quoted 5 lines
arch/s390/kernel/setup.c +1 -0
 void __init arch_cpu_finalize_init(void)
 {
+	lazy_mmu_online_boot_cpu();
 	sclp_init();
 }

What makes this code so special that an explicit call from arch_cpu_finalize_init() is required? This is really the last resort if everything else fails. To me it looks like the code can be changed to use a new static key, and add a generic early (pre-smp) initcall to allocate memory for cpu 0, and if that succeeds enable the static key.

I had exactly similar variant, but failed to resolve a race when a secondary CPU callback was called before the CPU0's one. Probably, used a wrong event. Will look into it again.

Quoted 22 lines
Quoted 22 lines
arch/s390/kernel/smp.c +6 -0
@@ -59,6 +59,7 @@
 #include <asm/topology.h>
 #include <asm/vdso.h>
 #include <asm/maccess.h>
+#include <asm/lazy_mmu.h>
 #include "entry.h"
 
 enum {
@@ -866,6 +867,11 @@ int __cpu_up(unsigned int cpu, struct task_struct *tidle)
 	rc = pcpu_alloc_lowcore(pcpu, cpu);
 	if (rc)
 		return rc;
+	rc = lazy_mmu_online_cpu(GFP_KERNEL, cpu);
+	if (rc) {
+		pcpu_free_lowcore(pcpu, cpu);
+		return rc;
+	}
 	/*
 	 * Make sure global control register contents do not change
 	 * until new CPU has initialized control registers.
Quoted 8 lines

arch/s390/kernel/smp.c void __cpu_die(unsigned int cpu) · line 930

Quoted 5 lines
arch/s390/kernel/smp.c +1 -0
@@ -921,6 +927,7 @@ void __cpu_die(unsigned int cpu)
 	pcpu = per_cpu_ptr(&pcpu_devices, cpu);
 	while (!pcpu_stopped(pcpu))
 		cpu_relax();
+	lazy_mmu_offline_cpu(cpu);

Same here: what makes this code so special that this needs to be open-coded into the low level cpu hotplug code? Everybody who needs to change this

This is just a follow-up of the boot CPU initialization above. AKA "Do not use CPU hotplug events".

Quoted 6 lines

code in future will wonder why the mmu code is so special that it needs to be directly handled here, and then needs to understand the mmu code. And the answer is: there is no reason.

Please use a generic cpu hotplug notifier to avoid that maintenance get's more expensive.

Will retry.

Quoted 9 lines
Quoted 9 lines
arch/s390/kernel/smp.c +9 -0
+static void leave_ipte_range(void)
+{
+	pte_t *ptep, *start, *start_cache, *cache;
+	unsigned long start_addr, addr;
+	struct ipte_range *range;
+	int start_idx;
+
+	if (!test_facility(13))
+		return;
Quoted 8 lines

arch/s390/kernel/smp.c

Quoted 5 lines
arch/s390/kernel/smp.c +5 -0
+
+	local_bh_disable();
+
+	lockdep_assert_preemption_disabled();
+	range = this_cpu_read(ipte_range);

Why is it required to disable bottom halves? A comment would be helpful. Or a hint in the commit message - this is not obvious.

When an interrupt arrives in the middle of enter|leave_ipte_range() the chain pcpu_addr_to_page() -> vmalloc_to_page() -> ptep_get() decides ptep_get() is called in lazy mode, while the per-cpu state not yet (de-)initialized (AKA inconsistent). That led to crashes:

    [    6.784258] Call Trace:
    [    7.784260]  [<0013d8935c3fe9dc>] pcpu_free_area+0x11c/0x3f0
    [    6.784265]  [<0013d8935c400bb6>] free_percpu.part.0+0x1b6/0xc70
    [    6.784270]  [<0013d8935bd8931c>] sched_free_group_rcu+0x2c/0x60
    [    6.784274]  [<0013d8935bea9196>] rcu_do_batch+0x2f6/0xdd0
    [    6.784280]  [<0013d8935beba0d0>] rcu_core+0x270/0x4e0
    [    6.784285]  [<0013d8935bd0a4b4>] handle_softirqs+0x294/0x800
    [    6.784289]  [<0013d8935bd0b020>] irq_exit_rcu+0x140/0x200
    [    6.784293]  [<0013d8935e26d994>] do_ext_irq+0xe4/0x330
    [    6.784298]  [<0013d8935e28c25c>] ext_int_handler+0xec/0x118
    [    6.784303]  [<0013d8935bc9764e>] enter_ipte_range+0xfe/0x1a0
    [    6.784308] ([<0013d8935bc975f6>] enter_ipte_range+0xa6/0x1a0)
    [    6.784313]  [<0013d8935c4432ce>] zap_pte_range+0x9ee/0xef0
    [    6.784317]  [<0013d8935c443a44>] zap_p4d_range+0x274/0x710
    [    6.784321]  [<0013d8935c44411c>] __zap_vma_range+0x23c/0x450
    [    6.784325]  [<0013d8935c445078>] unmap_vmas+0x1c8/0x440
    [    6.784329]  [<0013d8935c4a8f46>] unmap_region+0x196/0x320
    [    6.784332]  [<0013d8935c4ac174>] vms_complete_munmap_vmas+0x734/0x990
    [    6.784336]  [<0013d8935c4aec74>] do_vmi_align_munmap+0x2a4/0x3a0
    [    6.784340]  [<0013d8935c46138e>] __do_sys_brk+0x5fe/0x750
    [    6.784344]  [<0013d8935e26d3be>] __do_syscall+0x17e/0x3f0
    [    6.784348]  [<0013d8935e28b882>] system_call+0x72/0x90
    [    6.784352] Last Breaking-Event-Address:
    [    6.784353]  [<0013d8935c3febea>] pcpu_free_area+0x32a/0x3f0
    [    6.784361] Kernel panic - not syncing: Fatal exception in interrupt
Quoted 5 lines

Also at least for !PREEMPT_RT (which is always true for s390) lockdep_assert_preemption_disabled() is quite pointless, since local_bh_disable() just one line above disables preemption.

I guess you wanted to add that check above local_bh_disable()?

Exactly.

Quoted 1 line

If not you could as well remove it

Thanks a lot for the review, Heiko!

maintainer
#7 lore raw
Quoted 15 lines · Alexander Gordeev
Quoted 13 lines · Heiko Carstens

arch/s390/include/asm/lowcore.h struct lowcore { · line 166

Quoted 10 lines · Alexander Gordeev
arch/s390/include/asm/lowcore.h +1 -1
index 3b3ecc647993..dba236664da9 100644
@@ -163,7 +163,7 @@ struct lowcore {
 	__s32	preempt_count;			/* 0x03a8 */
 	__u32	spinlock_lockval;		/* 0x03ac */
 	__u32	spinlock_index;			/* 0x03b0 */
-	__u8	pad_0x03b4[0x03b8-0x03b4];	/* 0x03b4 */
+	__s32	lazy_mmu_count;			/* 0x03b4 */

Why is this signed? Can it get negative?

Quoted 1 line

For the same reason preempt_count is signed, I guess.

Check again, preempt_count is now unsigned and 64 bit ;)

Quoted 13 lines

No, it can not get negative and it is very handy to observe a disbalance in a crash (I did hit it indeed while debugging).

Quoted 9 lines

arch/s390/include/asm/lowcore.h struct lowcore {

Quoted 6 lines
arch/s390/include/asm/lowcore.h +6 -0
+static __always_inline bool is_lazy_mmu_active(void)
+{
+	if (__is_defined(__DECOMPRESSOR))
+		return false;
+	if (!get_lowcore()->lazy_mmu_count)
+		return false;

I guess there is opportunity to generate better code here using an alternative and using a flag output constraint too.

Quoted 1 line

Will try.

Guess that will be quite ugly code. Just let me know if I should provide that code, since I coded up such a mess already several times.

Quoted 12 lines
Quoted 11 lines

arch/s390/include/asm/lowcore.h

Quoted 5 lines

void __init arch_cpu_finalize_init(void)
{

arch/s390/include/asm/lowcore.h +1 -0
+	lazy_mmu_online_boot_cpu();
 	sclp_init();
 }

What makes this code so special that an explicit call from arch_cpu_finalize_init() is required? This is really the last resort if everything else fails. To me it looks like the code can be changed to use a new static key, and add a generic early (pre-smp) initcall to allocate memory for cpu 0, and if that succeeds enable the static key.

Quoted 3 lines

I had exactly similar variant, but failed to resolve a race when a secondary CPU callback was called before the CPU0's one. Probably, used a wrong event. Will look into it again.

early_initcall() should do the trick.

Quoted 8 lines
Quoted 7 lines

arch/s390/include/asm/lowcore.h

Quoted 4 lines
arch/s390/include/asm/lowcore.h +4 -0
+	local_bh_disable();
+
+	lockdep_assert_preemption_disabled();
+	range = this_cpu_read(ipte_range);

Why is it required to disable bottom halves? A comment would be helpful. Or a hint in the commit message - this is not obvious.

Quoted 4 lines

When an interrupt arrives in the middle of enter|leave_ipte_range() the chain pcpu_addr_to_page() -> vmalloc_to_page() -> ptep_get() decides ptep_get() is called in lazy mode, while the per-cpu state not yet (de-)initialized (AKA inconsistent). That led to crashes:

So, I don't know what exactly lead to the crash, but I guess the problem is an only partially initialized ipte_range struct, while the lazy_mmu_count has been incremented already? Isn't it possible to solve that problem without disabling bottom halves by reordering sequences in enter_ipte_range() and leave_ipte_range()?

Would be nice if we could avoid the not so obvious local_bh_disable() and local_bh_enable() pairs.

author maintainer
#8 lore raw
Quoted 16 lines · Heiko Carstens
Quoted 15 lines · Alexander Gordeev
Quoted 13 lines · Heiko Carstens

arch/s390/include/asm/lowcore.h struct lowcore { · line 166

Quoted 10 lines · Alexander Gordeev
arch/s390/include/asm/lowcore.h +1 -1
index 3b3ecc647993..dba236664da9 100644
@@ -163,7 +163,7 @@ struct lowcore {
 	__s32	preempt_count;			/* 0x03a8 */
 	__u32	spinlock_lockval;		/* 0x03ac */
 	__u32	spinlock_index;			/* 0x03b0 */
-	__u8	pad_0x03b4[0x03b8-0x03b4];	/* 0x03b4 */
+	__s32	lazy_mmu_count;			/* 0x03b4 */

Why is this signed? Can it get negative?

Quoted 3 lines
Quoted 1 line

For the same reason preempt_count is signed, I guess.

Check again, preempt_count is now unsigned and 64 bit ;)

D'oh :)

...

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

arch/s390/include/asm/lowcore.h struct lowcore {

Quoted 5 lines
arch/s390/include/asm/lowcore.h +1 -0
 void __init arch_cpu_finalize_init(void)
 {
+	lazy_mmu_online_boot_cpu();
 	sclp_init();
 }

What makes this code so special that an explicit call from arch_cpu_finalize_init() is required? This is really the last resort if everything else fails. To me it looks like the code can be changed to use a new static key, and add a generic early (pre-smp) initcall to allocate memory for cpu 0, and if that succeeds enable the static key.

Quoted 5 lines
Quoted 3 lines

I had exactly similar variant, but failed to resolve a race when a secondary CPU callback was called before the CPU0's one. Probably, used a wrong event. Will look into it again.

early_initcall() should do the trick.

Yep, early_initcall() + CPUHP_BP_PREPARE_DYN seems to be flying ;)

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

arch/s390/include/asm/lowcore.h

Quoted 4 lines
arch/s390/include/asm/lowcore.h +4 -0
+	local_bh_disable();
+
+	lockdep_assert_preemption_disabled();
+	range = this_cpu_read(ipte_range);

Why is it required to disable bottom halves? A comment would be helpful. Or a hint in the commit message - this is not obvious.

Quoted 8 lines
Quoted 4 lines

When an interrupt arrives in the middle of enter|leave_ipte_range() the chain pcpu_addr_to_page() -> vmalloc_to_page() -> ptep_get() decides ptep_get() is called in lazy mode, while the per-cpu state not yet (de-)initialized (AKA inconsistent). That led to crashes:

So, I don't know what exactly lead to the crash, but I guess the problem is an only partially initialized ipte_range struct, while the lazy_mmu_count has been incremented already? Isn't it possible to

Exactly.

Quoted 2 lines

solve that problem without disabling bottom halves by reordering sequences in enter_ipte_range() and leave_ipte_range()?

I guess you mean setting|unsetting the lazy mmu flag after|before the state is initialized? I will think about it, though I keep the door open for a possible lazy_mmu_mode_pause()|resume() implementation, which I am unable to predict how such rework would fit: lazy_mmu_count would have to stay non-zero in such a case.

Quoted 2 lines

Would be nice if we could avoid the not so obvious local_bh_disable() and local_bh_enable() pairs.

Calling ptep_get() from BH context was certainly unexpected, but the way local_bh_enable|disable() pairs are used is actually straightforward. This is a slow path anyway, so I would think the simplicity prevails in this case.

But again, I will try to avoid that.

Thanks!

maintainer
#9 lore raw
Quoted 10 lines · Alexander Gordeev
Quoted 2 lines · Heiko Carstens

Would be nice if we could avoid the not so obvious local_bh_disable() and local_bh_enable() pairs.

Calling ptep_get() from BH context was certainly unexpected, but the way local_bh_enable|disable() pairs are used is actually straightforward. This is a slow path anyway, so I would think the simplicity prevails in this case.

But again, I will try to avoid that.

I do agree that ptep_get() being called from BH context is not what I would have expected too. But then again, nothing prevents people from doing that from irq context too, no matter if that is sane or not.

Imho the "final" version should either be implemented that it can go without disabling bottom halves, or, if that is not worth the effort, even disable interrupts, just to avoid other surprises. Plus a comment why it is needed, please.

author maintainer
#10 lore raw
Quoted 10 lines · Heiko Carstens
Quoted 9 lines · Alexander Gordeev
Quoted 2 lines · Heiko Carstens

Would be nice if we could avoid the not so obvious local_bh_disable() and local_bh_enable() pairs.

Calling ptep_get() from BH context was certainly unexpected, but the way local_bh_enable|disable() pairs are used is actually straightforward. This is a slow path anyway, so I would think the simplicity prevails in this case.

Quoted 5 lines
Quoted 1 line

But again, I will try to avoid that.

I do agree that ptep_get() being called from BH context is not what I would have expected too. But then again, nothing prevents people from doing that from irq context too, no matter if that is sane or not.

I am currently looking if any interrupt handler does such a crazy thing. If not, I hope it would be possible to nail this requirement with the maintainers and e.g. add VM_BUG_ON(in_hardirq()) to ptep_get()/set_pte() implementations.

But even with BHs we have an interesting situation: calling ptep_get() from a BH while in the lazy MMU mode is actually a read hazard. IOW what I hit looks to me as a bug in generic code, and one that looks very difficult to fix. So staying with local_bh_enable|disable() would be still a good tradeoff.

Quoted 3 lines

Imho the "final" version should either be implemented that it can go without disabling bottom halves, or, if that is not worth the effort, even disable interrupts, just to avoid other surprises.

Yes, that would be the last resort.

Quoted 1 line

Plus a comment why it is needed, please.

Sure.

author maintainer
#11 lore raw
Quoted 13 lines · Alexander Gordeev
Quoted 12 lines · Heiko Carstens
Quoted 10 lines · Alexander Gordeev
Quoted 2 lines · Heiko Carstens

Would be nice if we could avoid the not so obvious local_bh_disable() and local_bh_enable() pairs.

Calling ptep_get() from BH context was certainly unexpected, but the way local_bh_enable|disable() pairs are used is actually straightforward. This is a slow path anyway, so I would think the simplicity prevails in this case.

But again, I will try to avoid that.

Quoted 8 lines
Quoted 3 lines

I do agree that ptep_get() being called from BH context is not what I would have expected too. But then again, nothing prevents people from doing that from irq context too, no matter if that is sane or not.

I am currently looking if any interrupt handler does such a crazy thing. If not, I hope it would be possible to nail this requirement with the maintainers and e.g. add VM_BUG_ON(in_hardirq()) to ptep_get()/set_pte() implementations.

I did not manage to find any code path that does something like this.

But few archs call ptep_get_lockless() from the hardirq perf handler, bypassing the per-cpu cache on s390 and (as I read it) violating their hardware lazy mmu mode. If this is an issue, then it is pre-existing.

Quoted 11 lines

But even with BHs we have an interesting situation: calling ptep_get() from a BH while in the lazy MMU mode is actually a read hazard. IOW what I hit looks to me as a bug in generic code, and one that looks very difficult to fix. So staying with local_bh_enable|disable() would be still a good tradeoff.

Quoted 3 lines

Imho the "final" version should either be implemented that it can go without disabling bottom halves, or, if that is not worth the effort, even disable interrupts, just to avoid other surprises.

Yes, that would be the last resort.

So leaving local_bh_enable|disable() while adding VM_BUG_ON(in_hardirq()) could actually work.

Also, I do not see how the code could be rearranged to avoid BH disable.

maintainer
#12 lore raw
Quoted 6 lines · Alexander Gordeev
Quoted 5 lines · Alexander Gordeev
Quoted 3 lines · Heiko Carstens

Imho the "final" version should either be implemented that it can go without disabling bottom halves, or, if that is not worth the effort, even disable interrupts, just to avoid other surprises.

Quoted 6 lines
Quoted 1 line

Yes, that would be the last resort.

So leaving local_bh_enable|disable() while adding VM_BUG_ON(in_hardirq()) could actually work.

Also, I do not see how the code could be rearranged to avoid BH disable.

Ok, then let's move forward with the current implementation. Thanks for looking into this!

author maintainer
#13 lore raw
In reply to Heiko Carstens
Quoted 9 lines · Heiko Carstens

arch/s390/include/asm/pgtable.h enum { · line 55

Quoted 6 lines
arch/s390/include/asm/pgtable.h +6 -0
+static __always_inline bool is_lazy_mmu_active(void)
+{
+	if (__is_defined(__DECOMPRESSOR))
+		return false;
+	if (!get_lowcore()->lazy_mmu_count)
+		return false;

I guess there is opportunity to generate better code here using an alternative and using a flag output constraint too.

It is only CY and LT instructions that make sense for me. Here is what I came up with:

static __always_inline bool is_lazy_mmu_active(void)

 {
	unsigned long lc_lazy_mmu_count;
	int cc;

if (__is_defined(__DECOMPRESSOR))

 		return false;
	lc_lazy_mmu_count = offsetof(struct lowcore, lazy_mmu_count);
	asm_inline(
		ALTERNATIVE("	cy	%[zero],%[offzero](%%r0)\n",
			    "	cy	%[zero],%[offalt](%%r0)\n",
			    ALT_FEATURE(MFEATURE_LOWCORE))
		CC_IPM(cc)
		: CC_OUT(cc, cc)
		: [offzero] "i" (lc_lazy_mmu_count),
		  [offalt] "i" (lc_lazy_mmu_count + LOWCORE_ALT_ADDRESS),
		  [zero] "d" (0), "m" (((struct lowcore *)0)->lazy_mmu_count)
		: CC_CLOBBER);
	return CC_TRANSFORM(cc);

}

As result instead of gcc code:

	lghi	%r8,0
	lt	%r1,1032(%r8)
	je	...

We get more or less the same:

	lhi	%r1,0
	cy	%r1,1032
	jne	...

The only benefit is [zero] "d" (0) could sometimes avoid LHI if a zeroed register is already around. But also the KASAN coverage is not generated.

Does it make sense to go ahead whith the alternative?

maintainer
#14 lore raw
Quoted 13 lines · Alexander Gordeev
Quoted 9 lines · Heiko Carstens

arch/s390/include/asm/pgtable.h enum { · line 55

Quoted 6 lines
arch/s390/include/asm/pgtable.h +6 -0
+static __always_inline bool is_lazy_mmu_active(void)
+{
+	if (__is_defined(__DECOMPRESSOR))
+		return false;
+	if (!get_lowcore()->lazy_mmu_count)
+		return false;

I guess there is opportunity to generate better code here using an alternative and using a flag output constraint too.

It is only CY and LT instructions that make sense for me. Here is what I came up with:

...

Quoted 12 lines

As result instead of gcc code:

	lghi	%r8,0
	lt	%r1,1032(%r8)
	je	...

We get more or less the same:

	lhi	%r1,0
	cy	%r1,1032
	jne	...
Quoted 5 lines

The only benefit is [zero] "d" (0) could sometimes avoid LHI if a zeroed register is already around. But also the KASAN coverage is not generated.

Does it make sense to go ahead whith the alternative?

Not really. But why is lazy_mmu_count 32 bit? Do you expect a nesting level of 2^32? :) If you would reduce that to one byte you could use tmy or cliy.

author maintainer
#15 lore raw
Quoted 33 lines · Heiko Carstens
Quoted 13 lines · Alexander Gordeev
Quoted 9 lines · Heiko Carstens

arch/s390/include/asm/pgtable.h enum { · line 55

Quoted 6 lines
arch/s390/include/asm/pgtable.h +6 -0
+static __always_inline bool is_lazy_mmu_active(void)
+{
+	if (__is_defined(__DECOMPRESSOR))
+		return false;
+	if (!get_lowcore()->lazy_mmu_count)
+		return false;

I guess there is opportunity to generate better code here using an alternative and using a flag output constraint too.

It is only CY and LT instructions that make sense for me. Here is what I came up with:

...

Quoted 16 lines

As result instead of gcc code:

	lghi	%r8,0
	lt	%r1,1032(%r8)
	je	...

We get more or less the same:

	lhi	%r1,0
	cy	%r1,1032
	jne	...

The only benefit is [zero] "d" (0) could sometimes avoid LHI if a zeroed register is already around. But also the KASAN coverage is not generated.

Quoted 5 lines
Quoted 1 line

Does it make sense to go ahead whith the alternative?

Not really. But why is lazy_mmu_count 32 bit? Do you expect a nesting level of 2^32? :) If you would reduce that to one byte you could use tmy or cliy.

Good point. But still, do you think it worth dropping the KASAN check?

maintainer
#16 lore raw
Quoted 8 lines · Alexander Gordeev
Quoted 5 lines · Heiko Carstens
Quoted 1 line · Alexander Gordeev

On Mon, Aug 24, 2026 at 12:40:48PM +0200, Heiko Carstens wrote:

Not really. But why is lazy_mmu_count 32 bit? Do you expect a nesting level of 2^32? :) If you would reduce that to one byte you could use tmy or cliy.

Good point. But still, do you think it worth dropping the KASAN check?

KASAN check for lowcore? I don't think we care - this is initialized to zero anyway, so what could go wrong? Also given how frequently these operations are used, yes, please drop KASAN checks.

maintainer
#17 lore raw
Quoted 8 lines · Heiko Carstens
Quoted 7 lines · Alexander Gordeev
Quoted 5 lines · Heiko Carstens
Quoted 1 line · Alexander Gordeev

On Mon, Aug 24, 2026 at 12:40:48PM +0200, Heiko Carstens wrote:

Not really. But why is lazy_mmu_count 32 bit? Do you expect a nesting level of 2^32? :) If you would reduce that to one byte you could use tmy or cliy.

Quoted 5 lines
Quoted 1 line

Good point. But still, do you think it worth dropping the KASAN check?

KASAN check for lowcore? I don't think we care - this is initialized to zero anyway, so what could go wrong? Also given how frequently these operations are used, yes, please drop KASAN checks.

Of course you could still add an explicit KASAN/KCSAN check by adding an instrument_read() call (see e.g. s390's fpu-insn.h, which has plenty of them).

But why do you think that would be helpful for a lowcore field, where it is known that is initialized to zero? What am I missing?

author maintainer
#18 lore raw
Quoted 11 lines · Heiko Carstens
Quoted 5 lines
Quoted 1 line

Good point. But still, do you think it worth dropping the KASAN check?

KASAN check for lowcore? I don't think we care - this is initialized to zero anyway, so what could go wrong? Also given how frequently these operations are used, yes, please drop KASAN checks.

Of course you could still add an explicit KASAN/KCSAN check by adding an instrument_read() call (see e.g. s390's fpu-insn.h, which has plenty of them).

But why do you think that would be helpful for a lowcore field, where it is known that is initialized to zero? What am I missing?

Nothing. I am going to adopt all your suggestions into the new version.

Thanks!

author maintainer
#19 lore raw

PATCH v7 3/4 mm/kasan: Introduce helpers for lazy MMU mode sanitizer

In reply to Alexander Gordeev

Provide helpers that allow architectures implement illegitimate PTE direct accesses while the lazy MMU mode is enabled, such as:

	pte_t pte = *ptep;
	*ptep = pte;

By contrast, these would have to be:

	pte_t pte = ptep_get(ptep);
	set_pte(ptep, pte);

The direct PTE accesses pose a real issue on s390.

---
 include/linux/kasan.h     | 19 +++++++++++++++++--
 mm/kasan/common.c         | 14 ++++++++++++++
 mm/kasan/kasan.h          |  2 ++
 mm/kasan/report_generic.c |  3 +++
 4 files changed, 36 insertions(+), 2 deletions(-)
include/linux/kasan.h +17 -2
index bf233bde68c7..1aa4d9dd2485 100644
@@ -6,6 +6,7 @@
 #include <linux/kasan-enabled.h>
 #include <linux/kasan-tags.h>
 #include <linux/kernel.h>
+#include <linux/pgtable.h>
 #include <linux/static_key.h>
 #include <linux/types.h>
 
@@ -35,8 +36,6 @@ typedef unsigned int __bitwise kasan_vmalloc_flags_t;
 
 #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
 
-#include <linux/pgtable.h>
-
 /* Software KASAN implementations use shadow memory. */
 
 #ifdef CONFIG_KASAN_SW_TAGS
@@ -134,6 +133,20 @@ static __always_inline void kasan_poison_slab(struct slab *slab)
 		__kasan_poison_slab(slab);
 }
 
+void __kasan_poison_pte(pte_t *pte, int nr);
+static __always_inline void kasan_poison_pte(pte_t *pte, int nr)
+{
+	if (kasan_enabled())
+		__kasan_poison_pte(pte, nr);
+}
+
+void __kasan_unpoison_pte(pte_t *pte, int nr);
+static __always_inline void kasan_unpoison_pte(pte_t *pte, int nr)
+{
+	if (kasan_enabled())
+		__kasan_unpoison_pte(pte, nr);
+}
+
 void __kasan_unpoison_new_object(struct kmem_cache *cache, void *object);
 /**
  * kasan_unpoison_new_object - Temporarily unpoison a new slab object.
@@ -414,6 +427,8 @@ static inline bool kasan_unpoison_pages(struct page *page, unsigned int order,
 	return false;
 }
 static inline void kasan_poison_slab(struct slab *slab) {}
+static inline void kasan_poison_pte(pte_t *pte, int nr) {}
+static inline void kasan_unpoison_pte(pte_t *pte, int nr) {}
 static inline void kasan_unpoison_new_object(struct kmem_cache *cache,
 					void *object) {}
 static inline void kasan_poison_new_object(struct kmem_cache *cache,
mm/kasan/common.c +14 -0
index b7d05c2a6d93..94d106094989 100644
@@ -163,6 +163,20 @@ void __kasan_poison_slab(struct slab *slab)
 		     KASAN_SLAB_REDZONE, false);
 }
 
+void __kasan_poison_pte(pte_t *pte, int nr)
+{
+	if (IS_ALIGNED(sizeof(*pte), KASAN_GRANULE_SIZE))
+		kasan_poison(pte, sizeof(*pte) * nr, KASAN_LAZY_MMU_PTE, false);
+}
+EXPORT_SYMBOL_GPL(__kasan_poison_pte);
+
+void __kasan_unpoison_pte(pte_t *pte, int nr)
+{
+	if (IS_ALIGNED(sizeof(*pte), KASAN_GRANULE_SIZE))
+		kasan_unpoison(pte, sizeof(*pte) * nr, false);
+}
+EXPORT_SYMBOL_GPL(__kasan_unpoison_pte);
+
 void __kasan_unpoison_new_object(struct kmem_cache *cache, void *object)
 {
 	kasan_unpoison(object, cache->object_size, false);
mm/kasan/kasan.h +2 -0
index fc9169a54766..1a2d18cdb21d 100644
@@ -144,12 +144,14 @@ static inline bool kasan_requires_meta(void)
 #define KASAN_PAGE_REDZONE	0xFE  /* redzone for kmalloc_large allocation */
 #define KASAN_SLAB_REDZONE	0xFC  /* redzone for slab object */
 #define KASAN_SLAB_FREE		0xFB  /* freed slab object */
+#define KASAN_LAZY_MMU_PTE	0xFD  /* direct pte access in lazy mmu mode */
 #define KASAN_VMALLOC_INVALID	0xF8  /* inaccessible space in vmap area */
 #else
 #define KASAN_PAGE_FREE		KASAN_TAG_INVALID
 #define KASAN_PAGE_REDZONE	KASAN_TAG_INVALID
 #define KASAN_SLAB_REDZONE	KASAN_TAG_INVALID
 #define KASAN_SLAB_FREE		KASAN_TAG_INVALID
+#define KASAN_LAZY_MMU_PTE	KASAN_TAG_INVALID
 #define KASAN_VMALLOC_INVALID	KASAN_TAG_INVALID /* only used for SW_TAGS */
 #endif
 
mm/kasan/report_generic.c +3 -0
index f5b8e37b3805..489d4a8d6902 100644
@@ -113,6 +113,9 @@ static const char *get_shadow_bug_type(struct kasan_report_info *info)
 	case KASAN_SLAB_FREE_META:
 		bug_type = "slab-use-after-free";
 		break;
+	case KASAN_LAZY_MMU_PTE:
+		bug_type = "lazy-mmu-pte-access";
+		break;
 	case KASAN_ALLOCA_LEFT:
 	case KASAN_ALLOCA_RIGHT:
 		bug_type = "alloca-out-of-bounds";
Signature · 1 line
2.53.0
#20 lore raw

Re: [PATCH v7 3/4] mm/kasan: Introduce helpers for lazy MMU mode sanitizer

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Using KASAN to catch direct PTE accesses during per-CPU lazy MMU mode will cause false positive…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Using KASAN to catch direct PTE accesses during per-CPU lazy MMU mode will cause false positive BUG reports on valid concurrent accesses from other CPUs. --

--- Patch [3]: [PATCH v7 3/4] mm/kasan: Introduce helpers for lazy MMU mode sanitizer --- commit 934c56df440d17dce73b955e4a04d2ffc114ef49
Author: Alexander Gordeev <[email protected]>

mm/kasan: Introduce helpers for lazy MMU mode sanitizer

This commit introduces helpers to allow architectures to use KASAN to catch illegitimate direct PTE accesses while lazy MMU mode is enabled.

Quoted 4 lines
mm/kasan/common.c +0 -0
index b7d05c2a6d93d..94d1060949891 100644

[ ... ]

mm/kasan/common.c void __kasan_poison_slab(struct slab *slab) · line 170

Quoted 9 lines
mm/kasan/common.c +5 -0
@@ -163,6 +163,20 @@ void __kasan_poison_slab(struct slab *slab)
 		     KASAN_SLAB_REDZONE, false);
 }
 
+void __kasan_poison_pte(pte_t *pte, int nr)
+{
+	if (IS_ALIGNED(sizeof(*pte), KASAN_GRANULE_SIZE))
+		kasan_poison(pte, sizeof(*pte) * nr, KASAN_LAZY_MMU_PTE, false);
+}

[Severity: Medium]
If this helper poisons the PTE in the global KASAN shadow memory, will it cause false positive KASAN splats for valid concurrent lockless readers?

For example, if another CPU performs a lockless page table walk like GUP-fast and executes ptep_get(), the instrumented READ_ONCE(*ptep) would trigger a lazy-mmu-pte-access report because the shadow memory was poisoned by the per-CPU lazy MMU mode.

Does this API design conflict with shared page tables?

Quoted 8 lines
mm/kasan/common.c +8 -0
+EXPORT_SYMBOL_GPL(__kasan_poison_pte);
+
+void __kasan_unpoison_pte(pte_t *pte, int nr)
+{
+	if (IS_ALIGNED(sizeof(*pte), KASAN_GRANULE_SIZE))
+		kasan_unpoison(pte, sizeof(*pte) * nr, false);
+}
+EXPORT_SYMBOL_GPL(__kasan_unpoison_pte);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#21 lore raw

Re: [PATCH v7 3/4] mm/kasan: Introduce helpers for lazy MMU mode sanitizer

In reply to Alexander Gordeev
Quoted 136 lines · Alexander Gordeev

Provide helpers that allow architectures implement illegitimate PTE direct accesses while the lazy MMU mode is enabled, such as:

        pte_t pte = *ptep;
        *ptep = pte;

By contrast, these would have to be:

        pte_t pte = ptep_get(ptep);
        set_pte(ptep, pte);

The direct PTE accesses pose a real issue on s390.

---
 include/linux/kasan.h     | 19 +++++++++++++++++--
 mm/kasan/common.c         | 14 ++++++++++++++
 mm/kasan/kasan.h          |  2 ++
 mm/kasan/report_generic.c |  3 +++
 4 files changed, 36 insertions(+), 2 deletions(-)
include/linux/kasan.h +17 -2
index bf233bde68c7..1aa4d9dd2485 100644
@@ -6,6 +6,7 @@
 #include <linux/kasan-enabled.h>
 #include <linux/kasan-tags.h>
 #include <linux/kernel.h>
+#include <linux/pgtable.h>
 #include <linux/static_key.h>
 #include <linux/types.h>
 
@@ -35,8 +36,6 @@ typedef unsigned int __bitwise kasan_vmalloc_flags_t;
 
 #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
 
-#include <linux/pgtable.h>
-
 /* Software KASAN implementations use shadow memory. */
 
 #ifdef CONFIG_KASAN_SW_TAGS
@@ -134,6 +133,20 @@ static __always_inline void kasan_poison_slab(struct slab *slab)
                __kasan_poison_slab(slab);
 }
 
+void __kasan_poison_pte(pte_t *pte, int nr);
+static __always_inline void kasan_poison_pte(pte_t *pte, int nr)
+{
+       if (kasan_enabled())
+               __kasan_poison_pte(pte, nr);
+}
+
+void __kasan_unpoison_pte(pte_t *pte, int nr);
+static __always_inline void kasan_unpoison_pte(pte_t *pte, int nr)
+{
+       if (kasan_enabled())
+               __kasan_unpoison_pte(pte, nr);
+}
+
 void __kasan_unpoison_new_object(struct kmem_cache *cache, void *object);
 /**
  * kasan_unpoison_new_object - Temporarily unpoison a new slab object.
@@ -414,6 +427,8 @@ static inline bool kasan_unpoison_pages(struct page *page, unsigned int order,
        return false;
 }
 static inline void kasan_poison_slab(struct slab *slab) {}
+static inline void kasan_poison_pte(pte_t *pte, int nr) {}
+static inline void kasan_unpoison_pte(pte_t *pte, int nr) {}
 static inline void kasan_unpoison_new_object(struct kmem_cache *cache,
                                        void *object) {}
 static inline void kasan_poison_new_object(struct kmem_cache *cache,
mm/kasan/common.c +14 -0
index b7d05c2a6d93..94d106094989 100644
@@ -163,6 +163,20 @@ void __kasan_poison_slab(struct slab *slab)
                     KASAN_SLAB_REDZONE, false);
 }
 
+void __kasan_poison_pte(pte_t *pte, int nr)
+{
+       if (IS_ALIGNED(sizeof(*pte), KASAN_GRANULE_SIZE))
+               kasan_poison(pte, sizeof(*pte) * nr, KASAN_LAZY_MMU_PTE, false);
+}
+EXPORT_SYMBOL_GPL(__kasan_poison_pte);
+
+void __kasan_unpoison_pte(pte_t *pte, int nr)
+{
+       if (IS_ALIGNED(sizeof(*pte), KASAN_GRANULE_SIZE))
+               kasan_unpoison(pte, sizeof(*pte) * nr, false);
+}
+EXPORT_SYMBOL_GPL(__kasan_unpoison_pte);
+
 void __kasan_unpoison_new_object(struct kmem_cache *cache, void *object)
 {
        kasan_unpoison(object, cache->object_size, false);
mm/kasan/kasan.h +2 -0
index fc9169a54766..1a2d18cdb21d 100644
@@ -144,12 +144,14 @@ static inline bool kasan_requires_meta(void)
 #define KASAN_PAGE_REDZONE     0xFE  /* redzone for kmalloc_large allocation */
 #define KASAN_SLAB_REDZONE     0xFC  /* redzone for slab object */
 #define KASAN_SLAB_FREE                0xFB  /* freed slab object */
+#define KASAN_LAZY_MMU_PTE     0xFD  /* direct pte access in lazy mmu mode */
 #define KASAN_VMALLOC_INVALID  0xF8  /* inaccessible space in vmap area */
 #else
 #define KASAN_PAGE_FREE                KASAN_TAG_INVALID
 #define KASAN_PAGE_REDZONE     KASAN_TAG_INVALID
 #define KASAN_SLAB_REDZONE     KASAN_TAG_INVALID
 #define KASAN_SLAB_FREE                KASAN_TAG_INVALID
+#define KASAN_LAZY_MMU_PTE     KASAN_TAG_INVALID
 #define KASAN_VMALLOC_INVALID  KASAN_TAG_INVALID /* only used for SW_TAGS */
 #endif
 
mm/kasan/report_generic.c +3 -1
index f5b8e37b3805..489d4a8d6902 100644
@@ -113,6 +113,9 @@ static const char *get_shadow_bug_type(struct kasan_report_info *info)
        case KASAN_SLAB_FREE_META:
                bug_type = "slab-use-after-free";
                break;
+       case KASAN_LAZY_MMU_PTE:
+               bug_type = "lazy-mmu-pte-access";
+               break;
        case KASAN_ALLOCA_LEFT:
        case KASAN_ALLOCA_RIGHT:
                bug_type = "alloca-out-of-bounds";
--

2.53.0

Quoted 4 lines

--
You received this message because you are subscribed to the Google Groups "kasan-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion visit https://groups.google.com/d/msgid/kasan-dev/52c9be0cf71a8896a5d76c148c38fb24a2005dd9.1786956464.git.agordeev%40linux.ibm.com.

Hi Alexander,

Should all this be limited to only s390?

Thanks!

author maintainer
#22 lore raw

Re: [PATCH v7 3/4] mm/kasan: Introduce helpers for lazy MMU mode sanitizer

On Tue, Aug 18, 2026 at 12:11:34AM +0200, Andrey Konovalov wrote:
...

Quoted 3 lines

Hi Alexander,

Should all this be limited to only s390?

The idea is to allow any architecture to opt-in. Once [1] is accepted this may look redundant, but I still see the benefit in having it.

I only did not manage to circumvent the ugly __no_sanitize_address for ptep_get_lockless() to make it avoid "bogus" kasan complains when gup_fast walks a page table that entered the lazy mmu mode.

1. https://lore.kernel.org/linux-mm/[email protected]/

Quoted 1 line

Thanks!

maintainer
#23 lore raw

Re: [PATCH v7 3/4] mm/kasan: Introduce helpers for lazy MMU mode sanitizer

Quoted 8 lines · Alexander Gordeev

On Tue, Aug 18, 2026 at 12:11:34AM +0200, Andrey Konovalov wrote:
...

Quoted 3 lines

Hi Alexander,

Should all this be limited to only s390?

The idea is to allow any architecture to opt-in. Once [1] is accepted this may look redundant, but I still see the benefit in having it.

What would be the benefit if [1] is accepted?

author maintainer
#24 lore raw

Re: [PATCH v7 3/4] mm/kasan: Introduce helpers for lazy MMU mode sanitizer

Quoted 11 lines · Heiko Carstens
Quoted 8 lines · Alexander Gordeev

On Tue, Aug 18, 2026 at 12:11:34AM +0200, Andrey Konovalov wrote:
...

Quoted 3 lines

Hi Alexander,

Should all this be limited to only s390?

The idea is to allow any architecture to opt-in. Once [1] is accepted this may look redundant, but I still see the benefit in having it.

What would be the benefit if [1] is accepted?

Just another mechanism for verification, and pretty cheap one. I would dare to say [1] does not rule out an invalid access completely.

maintainer
#25 lore raw

Re: [PATCH v7 3/4] mm/kasan: Introduce helpers for lazy MMU mode sanitizer

Quoted 8 lines · Alexander Gordeev
Quoted 7 lines · Heiko Carstens
Quoted 6 lines · Alexander Gordeev

On Tue, Aug 18, 2026 at 12:11:34AM +0200, Andrey Konovalov wrote:
...

Quoted 3 lines

Hi Alexander,

Should all this be limited to only s390?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

The idea is to allow any architecture to opt-in. Once [1] is accepted this may look redundant, but I still see the benefit in having it.

What would be the benefit if [1] is accepted?

Just another mechanism for verification, and pretty cheap one. I would dare to say [1] does not rule out an invalid access completely.

Ok, maybe we should revisit this when we know how the solution for page table accesses will be. Then hopefully it should be quite obvious if this additional code is helpful or not.

author maintainer
#26 lore raw

PATCH v7 4/4 s390/mm: Lazy MMU mode sanitizer

In reply to Alexander Gordeev

Detect PTE entries access in lazy MMU mode by means other than set_pte() and ptep_get() primitives, which would be a read hazard.

The access to kasan shadow memory from ptep_get_lockless() mistakenly hits invalid access in case a concurrent lazy MMU access to the same PTE is happening. To avoid that disable instrumentation for ptep_get_lockless() altogether.

---
 arch/s390/include/asm/pgtable.h |  6 ++++++
 arch/s390/mm/lazy_mmu.c         | 27 +++++++++++++++++++++++----
 2 files changed, 29 insertions(+), 4 deletions(-)
arch/s390/include/asm/pgtable.h +6 -0
index bf54202a0630..1e8a79989ed6 100644
@@ -1049,6 +1049,12 @@ static inline void set_pte(pte_t *ptep, pte_t pte)
 		__set_pte(ptep, pte);
 }
 
+#define ptep_get_lockless ptep_get_lockless
+static inline __no_sanitize_address pte_t ptep_get_lockless(pte_t *ptep)
+{
+	return READ_ONCE(*ptep);
+}
+
 static inline pte_t __ptep_get(pte_t *ptep)
 {
 	return READ_ONCE(*ptep);
arch/s390/mm/lazy_mmu.c +23 -4
index 002ee766b391..7e3a4115ea04 100644
@@ -63,10 +63,13 @@ static int invalidate_pte_range(struct mm_struct *mm, unsigned long addr,
 }
 
 static void set_pte_range(struct mm_struct *mm, unsigned long addr,
-			  pte_t *ptep, pte_t *end, pte_t *cache)
+			  pte_t *start, pte_t *end, pte_t *cache)
 {
-	int i, nr_ptes;
+	int nr_ptes, nr_total = end - start;
+	pte_t *ptep = start;
+	int i;
 
+	kasan_unpoison_pte(start, nr_total);
 	while (ptep < end) {
 		nr_ptes = invalidate_pte_range(mm, addr, ptep, end);
 
@@ -77,6 +80,7 @@ static void set_pte_range(struct mm_struct *mm, unsigned long addr,
 
 		addr += nr_ptes * PAGE_SIZE;
 	}
+	kasan_poison_pte(start, nr_total);
 }
 
 static void enter_ipte_norange(void)
@@ -98,6 +102,7 @@ static void enter_ipte_range(struct mm_struct *mm,
 			     unsigned long addr, unsigned long end, pte_t *pte)
 {
 	struct ipte_range *range;
+	unsigned int nr_ptes;
 
 	if (!test_facility(13))
 		return;
@@ -117,6 +122,9 @@ static void enter_ipte_range(struct mm_struct *mm,
 	range->base_end = end;
 	range->base_pte = pte;
 
+	nr_ptes = (range->base_end - range->base_addr) / PAGE_SIZE;
+	kasan_poison_pte(range->base_pte, nr_ptes);
+
 	local_bh_enable();
 }
 
@@ -125,6 +133,7 @@ static void leave_ipte_range(void)
 	pte_t *ptep, *start, *start_cache, *cache;
 	unsigned long start_addr, addr;
 	struct ipte_range *range;
+	unsigned int nr_ptes;
 	int start_idx;
 
 	if (!test_facility(13))
@@ -163,6 +172,9 @@ static void leave_ipte_range(void)
 	range->end_pte = NULL;
 
 done:
+	nr_ptes = (range->base_end - range->base_addr) / PAGE_SIZE;
+	kasan_unpoison_pte(range->base_pte, nr_ptes);
+
 	range->mm = NULL;
 	range->base_addr = 0;
 	range->base_end = 0;
@@ -244,10 +256,17 @@ static void __ipte_range_set_pte(struct ipte_range *range, pte_t *ptep, pte_t pt
 static pte_t __ipte_range_ptep_get(struct ipte_range *range, pte_t *ptep)
 {
 	unsigned int idx = ptep - range->base_pte;
+	pte_t pte;
 
 	lockdep_assert_preemption_disabled();
-	if (pte_val(range->cache[idx]) == PTE_POISON)
-		return __ptep_get(ptep);
+	if (pte_val(range->cache[idx]) == PTE_POISON) {
+		kasan_unpoison_pte(ptep, 1);
+		pte = __ptep_get(ptep);
+		kasan_poison_pte(ptep, 1);
+
+		return pte;
+	}
+
 	return range->cache[idx];
 }
 
Signature · 1 line
2.53.0
#27 lore raw

Re: [PATCH v7 4/4] s390/mm: Lazy MMU mode sanitizer

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines

Detect PTE entries access in lazy MMU mode by means other than set_pte() and ptep_get() primitives, which would be a read hazard.

The access to kasan shadow memory from ptep_get_lockless() mistakenly hits invalid access in case a concurrent lazy MMU access to the same PTE is happening. To avoid that disable instrumentation for ptep_get_lockless() altogether.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4