s390/mm: Batch PTE updates in lazy MMU mode

Lazy MMU mode is assumed to be context-independent, in the sense that it does not need any additional information while operating. However, the s390 architecture benefits from knowing the exact page table entries being modified.

Introduce lazy_mmu_mode_enable_pte(), which is provided with the process address space and the page table being operated on. This information is required to enable s390-specific optimizations.

The function takes parameters that are typically passed to page- table level walkers, which implies that the span of PTE entries never crosses a page table boundary.

Architectures that do not require such information simply do not need to define the arch_enter_lazy_mmu_mode_pte() callback.

Signed-off-by Alexander Gordeev <[email protected]>

---
 fs/proc/task_mmu.c      |  2 +-
 include/linux/pgtable.h | 42 +++++++++++++++++++++++++++++++++++++++++
 mm/madvise.c            |  8 ++++----
 mm/memory.c             |  8 ++++----
 mm/mprotect.c           |  2 +-
 mm/mremap.c             |  2 +-
 mm/vmalloc.c            |  6 +++---
 7 files changed, 56 insertions(+), 14 deletions(-)

fs/proc/task_mmu.c +1 -1

index e091931d7ca1..4e3b1987874a 100644
@@ -2752,7 +2752,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
 		return 0;
 	}
 
-	lazy_mmu_mode_enable();
+	lazy_mmu_mode_enable_pte(vma->vm_mm, start, end, start_pte);
 
 	if ((p->arg.flags & PM_SCAN_WP_MATCHING) && !p->vec_out) {
 		/* Fast path for performing exclusive WP */

include/linux/pgtable.h +0 -0

index a50df42a893f..481b45954800 100644

include/linux/pgtable.h +4 -0

+ *
+ * Enters a new lazy MMU mode section; if the mode was not already enabled,
+ * enables it and calls arch_enter_lazy_mmu_mode_pte().
+ *

Quoted 2 lines

+/**
+ * lazy_mmu_mode_enable_pte() - Enable the lazy MMU mode with parameters

You have to be a lot clearer about implications. For example, what happens if we would bail out and not process all ptes? What are the exact semantics.

include/linux/pgtable.h static inline void lazy_mmu_mode_enable(void) · line 299

Quoted 13 lines

+ * Enters a new lazy MMU mode section; if the mode was not already enabled,
+ * enables it and calls arch_enter_lazy_mmu_mode_pte().
+ *
+ * Must be paired with a call to lazy_mmu_mode_disable().
+ *
+ * Has no effect if called:
+ * - While paused - see lazy_mmu_mode_pause()
+ * - In interrupt context
+ */

include/linux/pgtable.h +4 -0

+static inline void lazy_mmu_mode_enable_pte(struct mm_struct *mm,
+					    unsigned long addr,
+					    unsigned long end,
+					    pte_t *ptep)

It can be multiple ptes, so should this be some kind of "pte_range"/

lazy_mmu_mode_enable_for_pte_range()

A bit mouthful but clearer.

include/linux/pgtable.h static inline void lazy_mmu_mode_enable(void) · line 309

Quoted 10 lines

include/linux/pgtable.h +10 -0

+{
+	struct lazy_mmu_state *state = &current->lazy_mmu_state;
+
+	if (in_interrupt() || state->pause_count > 0)
+		return;
+
+	VM_WARN_ON_ONCE(state->enable_count == U8_MAX);
+
+	if (state->enable_count++ == 0)
+		arch_enter_lazy_mmu_mode_pte(mm, addr, end, ptep);

Quoted 1 line

+}

I'm wondering whether that could instead be some optional interface that we trigger after the lazy_mmu_mode_enable. But looking at

lazy_mmu_mode_enable() users, there don't seem to be cases where we would process multiple different ranges under a single enable() call, right?

Signature · 3 lines

Cheers,

David

Multiple different ranges still could be processed, but then one should continue using arch_enter_lazy_mmu_mode().

On Wed, Mar 25, 2026 at 10:55:23AM +0100, David Hildenbrand (Arm) wrote:

Hi David,

Quoted 5 lines

Quoted 2 lines

+/**
+ * lazy_mmu_mode_enable_pte() - Enable the lazy MMU mode with parameters

You have to be a lot clearer about implications. For example, what happens if we would bail out and not process all ptes? What are the exact semantics.

The only implication is "only this address/PTE range could be updated and that range may span one page table at most".

Whether all or portion of PTEs were actually updated is not defined, just like in case of lazy_mmu_mode_enable_pte().

Makes sense?

Quoted 19 lines

include/linux/pgtable.h static inline void lazy_mmu_mode_enable(void) · line 299

Quoted 13 lines

+ * Enters a new lazy MMU mode section; if the mode was not already enabled,
+ * enables it and calls arch_enter_lazy_mmu_mode_pte().
+ *
+ * Must be paired with a call to lazy_mmu_mode_disable().
+ *
+ * Has no effect if called:
+ * - While paused - see lazy_mmu_mode_pause()
+ * - In interrupt context
+ */

include/linux/pgtable.h +4 -0

+static inline void lazy_mmu_mode_enable_pte(struct mm_struct *mm,
+					    unsigned long addr,
+					    unsigned long end,
+					    pte_t *ptep)

It can be multiple ptes, so should this be some kind of "pte_range"/

lazy_mmu_mode_enable_for_pte_range()

A bit mouthful but clearer.

Quoted 10 lines

include/linux/pgtable.h static inline void lazy_mmu_mode_enable(void) · line 309

Quoted 10 lines

include/linux/pgtable.h +10 -0

+{
+	struct lazy_mmu_state *state = &current->lazy_mmu_state;
+
+	if (in_interrupt() || state->pause_count > 0)
+		return;
+
+	VM_WARN_ON_ONCE(state->enable_count == U8_MAX);
+
+	if (state->enable_count++ == 0)
+		arch_enter_lazy_mmu_mode_pte(mm, addr, end, ptep);

I will also change arch_enter_lazy_mmu_mode_pte() to arch_enter_lazy_mmu_mode_for_pte_range() then.

Quoted 12 lines · Alexander Gordeev

On Wed, Mar 25, 2026 at 10:55:23AM +0100, David Hildenbrand (Arm) wrote:

Hi David,

Quoted 5 lines

Quoted 2 lines

+/**
+ * lazy_mmu_mode_enable_pte() - Enable the lazy MMU mode with parameters

You have to be a lot clearer about implications. For example, what happens if we would bail out and not process all ptes? What are the exact semantics.

The only implication is "only this address/PTE range could be updated and that range may span one page table at most".

Quoted 4 lines

Whether all or portion of PTEs were actually updated is not defined, just like in case of lazy_mmu_mode_enable_pte().

Makes sense?

I also feel that the comment needs to be much more specific. From a brief glance at patch 2, it seems that __ipte_batch_set_pte() assumes that all PTEs processed after this function is called are contiguous.

This should be documented.

Quoted 2 lines

I will also change arch_enter_lazy_mmu_mode_pte() to arch_enter_lazy_mmu_mode_for_pte_range() then.

Makes sense. The interface looks reasonable to me with this new name.

One more comment though: in previous discussions you mentioned the need for arch_{pause,resume} hooks, is that no longer necessary simply because {pause,resume} are not used on the paths where you make use of the new enable function?

- Kevin

Quoted 16 lines · Kevin Brodsky

Quoted 15 lines · Alexander Gordeev

On Wed, Mar 25, 2026 at 10:55:23AM +0100, David Hildenbrand (Arm) wrote:

Hi David,

Quoted 5 lines

Quoted 2 lines

+/**
+ * lazy_mmu_mode_enable_pte() - Enable the lazy MMU mode with parameters

You have to be a lot clearer about implications. For example, what happens if we would bail out and not process all ptes? What are the exact semantics.

The only implication is "only this address/PTE range could be updated and that range may span one page table at most".

Whether all or portion of PTEs were actually updated is not defined, just like in case of lazy_mmu_mode_enable_pte().

Quoted 4 lines

Quoted 1 line

Makes sense?

I also feel that the comment needs to be much more specific. From a brief glance at patch 2, it seems that __ipte_batch_set_pte() assumes that all PTEs processed after this function is called are contiguous.

No, this is actually not the case. __ipte_batch_set_pte() just sets ceilings for later processing. The PTEs within the range could be updated in any order and not necessarily all of them.

Quoted 1 line

This should be documented.

Will do.

Quoted 4 lines

Quoted 2 lines

I will also change arch_enter_lazy_mmu_mode_pte() to arch_enter_lazy_mmu_mode_for_pte_range() then.

Makes sense. The interface looks reasonable to me with this new name.

Quoted 4 lines

One more comment though: in previous discussions you mentioned the need for arch_{pause,resume} hooks, is that no longer necessary simply because {pause,resume} are not used on the paths where you make use of the new enable function?

Yes. I did implement arch_pause|resume_lazy_mmu_mode() for a custom KASAN sanitizer to catch direct PTE dereferences - ones that bypass ptep_get()/set_pte() in lazy mode.

But that code is not upstreamed and therefore there is no need to introduce the hooks just right now.

Quoted 1 line

- Kevin

Thanks for the review, Kevin!

On Wed, Mar 25, 2026 at 10:55:23AM +0100, David Hildenbrand (Arm) wrote:

Hi David,

Quoted 3 lines

Quoted 2 lines

+/**
+ * lazy_mmu_mode_enable_pte() - Enable the lazy MMU mode with parameters

Quoted 3 lines

You have to be a lot clearer about implications. For example, what happens if we would bail out and not process all ptes? What are the exact semantics.

The only implication is "only this address/PTE range could be updated and that range may span one page table at most".

Whether all or portion of PTEs were actually updated is not defined, just like in case of lazy_mmu_mode_enable_pte().

Makes sense?

Quoted 20 lines

include/linux/pgtable.h static inline void lazy_mmu_mode_enable(void) · line 299

Quoted 13 lines

+ * Enters a new lazy MMU mode section; if the mode was not already enabled,
+ * enables it and calls arch_enter_lazy_mmu_mode_pte().
+ *
+ * Must be paired with a call to lazy_mmu_mode_disable().
+ *
+ * Has no effect if called:
+ * - While paused - see lazy_mmu_mode_pause()
+ * - In interrupt context
+ */

include/linux/pgtable.h +4 -0

+static inline void lazy_mmu_mode_enable_pte(struct mm_struct *mm,
+					    unsigned long addr,
+					    unsigned long end,
+					    pte_t *ptep)

It can be multiple ptes, so should this be some kind of "pte_range"/

lazy_mmu_mode_enable_for_pte_range()

A bit mouthful but clearer.

Quoted 10 lines

include/linux/pgtable.h static inline void lazy_mmu_mode_enable(void) · line 309

Quoted 10 lines

include/linux/pgtable.h +10 -0

+{
+	struct lazy_mmu_state *state = &current->lazy_mmu_state;
+
+	if (in_interrupt() || state->pause_count > 0)
+		return;
+
+	VM_WARN_ON_ONCE(state->enable_count == U8_MAX);
+
+	if (state->enable_count++ == 0)
+		arch_enter_lazy_mmu_mode_pte(mm, addr, end, ptep);

I will also change arch_enter_lazy_mmu_mode_pte() to arch_enter_lazy_mmu_mode_for_pte_range() then.

Quoted 4 lines

Quoted 1 line

+}

I'm wondering whether that could instead be some optional interface that we trigger after the lazy_mmu_mode_enable. But looking at

To me just two separate and (as you put it) mouthful names appeal better than an optional follow-up interface.

Quoted 7 lines

Quoted 6 lines

Quoted 2 lines

+ * lazy_mmu_mode_enable_pte() - Enable the lazy MMU mode with parameters

You have to be a lot clearer about implications. For example, what happens if we would bail out and not process all ptes? What are the exact semantics.

Quoted 2 lines

The only implication is "only this address/PTE range could be updated and that range may span one page table at most".

Probably phrase it stronger. "No ptes outside of this range must be updated" etc.

Quoted 6 lines · David Hildenbrand (Arm)

Quoted 3 lines

The only implication is "only this address/PTE range could be updated and that range may span one page table at most".

Probably phrase it stronger. "No ptes outside of this range must be updated" etc.

That turns out to be bit more complicated. The below cases do not fit such a strong requirement:

1. copy_pte_range() operates on two ranges: source and destination. Though lazy_mmu_mode_enable_for_pte_range() applies to the source one, updates to the destination are still happen while in tha lazy mode. (Although the lazy mode is not actually needed for the destination unattached MM).

2. move_ptes() also operates on a source and destination ranges, but unlike copy_pte_range() the destination range is also attached to the currently active task.

3. Though theoretical, nesting sections with interleaving calls to lazy_mmu_mode_enable() and lazy_mmu_mode_enable_for_pte_range() make it difficult to define (let alone to implement) which range is currently active, if any.

All of these goes away if we switch from for_pte_range() to fast_pte_range() semantics:

Quoted 5 lines

1. copy_pte_range() operates on two ranges: source and destination. Though lazy_mmu_mode_enable_for_pte_range() applies to the source one, updates to the destination are still happen while in tha lazy mode. (Although the lazy mode is not actually needed for the destination unattached MM).

So, here a

"No ptes outside of this range in the provided @mm must be updated."

could be used.

Quoted 4 lines

2. move_ptes() also operates on a source and destination ranges, but unlike copy_pte_range() the destination range is also attached to the currently active task.

But not here.

Quoted 4 lines

3. Though theoretical, nesting sections with interleaving calls to lazy_mmu_mode_enable() and lazy_mmu_mode_enable_for_pte_range() make it difficult to define (let alone to implement) which range is currently active, if any.

Right. I assume you would specify the source here as well, or which one would it be in your case to speed it up?

Quoted 2 lines

All of these goes away if we switch from for_pte_range() to fast_pte_range() semantics:

I don't quite like the "fast" in there. I think you can keep the old name, but clarifying that it is merely a hint, and only ptes that fall into the hint might observe a speedup.

Could performance benefit from multiple ranges? (like in mremap, for example)?

In that case, an explicit hint interface could be reconsidered.

Signature · 3 lines

Cheers,

David

Quoted 13 lines · David Hildenbrand (Arm)

Quoted 5 lines

1. copy_pte_range() operates on two ranges: source and destination. Though lazy_mmu_mode_enable_for_pte_range() applies to the source one, updates to the destination are still happen while in tha lazy mode. (Although the lazy mode is not actually needed for the destination unattached MM).

So, here a

"No ptes outside of this range in the provided @mm must be updated."

could be used.

Quoted 1 line

Quoted 5 lines

Quoted 3 lines

2. move_ptes() also operates on a source and destination ranges, but unlike copy_pte_range() the destination range is also attached to the currently active task.

But not here.

I did not quite understand these two comments ;), but I think I address them further below.

Quoted 8 lines

Quoted 5 lines

3. Though theoretical, nesting sections with interleaving calls to lazy_mmu_mode_enable() and lazy_mmu_mode_enable_for_pte_range() make it difficult to define (let alone to implement) which range is currently active, if any.

Right. I assume you would specify the source here as well, or which one would it be in your case to speed it up?

It is in all cases the source/old/existing one.

Quoted 7 lines

Quoted 3 lines

All of these goes away if we switch from for_pte_range() to fast_pte_range() semantics:

I don't quite like the "fast" in there. I think you can keep the old name, but clarifying that it is merely a hint, and only ptes that fall into the hint might observe a speedup.

Okay, that simplify things.

Quoted 2 lines

Could performance benefit from multiple ranges? (like in mremap, for example)?

No.

Quoted 1 line

In that case, an explicit hint interface could be reconsidered.

So all things considered, how does it look?

/**
* lazy_mmu_mode_enable_for_pte_range() - Enable the lazy MMU mode with a speedup hint.
* @mm: Address space the ptes represent.
* @addr: Address of the first pte.
* @end: End address of the range.
* @ptep: Page table pointer for the first entry.
*
* Enters a new lazy MMU mode section; if the mode was not already enabled,
* enables it and calls arch_enter_lazy_mmu_mode_for_pte_range().
*
* PTEs that fall within the specified range might observe update speedups.
* The PTE range must belong to the specified memory space and do not cross
* a page table boundary.
*
* There are no requirements on the order or range completeness of PTE

* updates for the specified range.
*
* Must be paired with a call to lazy_mmu_mode_disable(). *
* Has no effect if called:
* - While paused - see lazy_mmu_mode_pause()
* - In interrupt context
*/

/**
* lazy_mmu_mode_enable_for_pte_range() - Enable the lazy MMU mode with a speedup hint.
* @mm: Address space the ptes represent.

* @addr: Address of the first pte.

* @end: End address of the range.
* @ptep: Page table pointer for the first entry.
*
* Enters a new lazy MMU mode section; if the mode was not already enabled,
* enables it and calls arch_enter_lazy_mmu_mode_for_pte_range(). *
* PTEs that fall within the specified range might observe update speedups.
* The PTE range must belong to the specified memory space and do not cross

* a page table boundary.
*
* There are no requirements on the order or range completeness of PTE
* updates for the specified range.
*
* Must be paired with a call to lazy_mmu_mode_disable().
*
* Has no effect if called:
* - While paused - see lazy_mmu_mode_pause()
* - In interrupt context
*/

Quoted 1 line

* @mm: Address space the ptes represent.

Not sure that makes sense, maybe something like "Address space the PTEs belong to"?

Quoted 1 line

* @addr: Address of the first pte.

Isn't it the address of the underlying memory rather?

Quoted 2 lines

* PTEs that fall within the specified range might observe update speedups.
* The PTE range must belong to the specified memory space and do not cross

s/do not/not/

Looks reasonable to me otherwise.

- Kevin

Quoted 6 lines · Kevin Brodsky

Quoted 1 line

* @mm: Address space the ptes represent.

Not sure that makes sense, maybe something like "Address space the PTEs belong to"?

Quoted 1 line

* @addr: Address of the first pte.

Isn't it the address of the underlying memory rather?

Will update both and also the clear_not_present_full_ptes() comment, where I copied it from.