Optimize anonymous large folio unmapping

PATCH Superseded Dev Jain 4 participants 40 messages May 6 – May 12

Subsystems (7)

Memory management MM core MGLRU Memory mapping +3 more subsystems

lore

Share & tools

lore

Dev Jain

author

2026-05-06 09:45 UTC #0 lore raw

v3 cover Optimize anonymous large folio unmapping

Speed up unmapping of anonymous large folios by clearing the ptes, and setting swap ptes, in one go.

The following benchmark (stolen from Barry at [1]) is used to measure the time taken to swapout 256M worth of memory backed by 64K large folios:

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <errno.h>

#define SIZE_MB 256
#define SIZE_BYTES (SIZE_MB * 1024 * 1024)

 int main() {
     void *addr = mmap(NULL, SIZE_BYTES, PROT_READ | PROT_WRITE,
                       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
     if (addr == MAP_FAILED) {
         perror("mmap failed");
         return 1;
     }

     memset(addr, 0, SIZE_BYTES);

     struct timespec start, end;
     clock_gettime(CLOCK_MONOTONIC, &start);

     if (madvise(addr, SIZE_BYTES, MADV_PAGEOUT) != 0) {
         perror("madvise(MADV_PAGEOUT) failed");
         munmap(addr, SIZE_BYTES);
         return 1;
     }

     clock_gettime(CLOCK_MONOTONIC, &end);

     long duration_ns = (end.tv_sec - start.tv_sec) * 1e9 +
                        (end.tv_nsec - start.tv_nsec);
     printf("madvise(MADV_PAGEOUT) took %ld ns (%.3f ms)\n",
            duration_ns, duration_ns / 1e6);

     munmap(addr, SIZE_BYTES);
     return 0;

}

Performance as measured on a Linux VM on Apple M3 (arm64):

Vanilla - Mean: 37401913 ns, std dev: 12%
Patched - Mean: 17420282 ns, std dev: 11%

No regression observed on 4K folios.

Performance as measured on bare metal x86:

Vanilla - mean: 54986286 ns, std dev: 1.5%
Patched - mean: 51930795 ns, std dev: 3%

Interestingly, no obvious improvement is observed on x86, hinting that the benefit lies mainly in the reduction of ptep_get() calls and the reduction of TLB flushes during contpte-unfolding, on arm64.

No regression is observed on 4K folios on x86 too.

---
Applies on mm-unstable (2d565cbaafd4).

v2->v3:
Mostly a resend after merge window. Some minor changes:

- Match kerneldoc parameter with function parameter (pte -> ptep)
- Mention change BUG->WARN in patch description
- Rename walk_done -> exit_walk in patch 2
-
v1->v2:
- Keep nr_pages as unsigned long
- Add patch 2
- Rename some functions, make return type bool for functions returning 0/1
- Drop page_vma_mapped_walk_jump - this is implicitly handled
- Drop likely()
- Add folio_dup/put_swap_pages, do subpage -> page
- Shorten the kerneldoc to remove unnecessary information - keep it aligned with analogous functions
- Put clear_pages_anon_exclusive to mm.h
- Some more refactoring in last patch with finish_folio_unmap

Dev Jain (9):
mm/rmap: initialize nr_pages to 1 at loop start in try_to_unmap_one mm/rmap: refactor hugetlb pte clearing in try_to_unmap_one mm/rmap: refactor some code around lazyfree folio unmapping mm/memory: Batch set uffd-wp markers during zapping mm/rmap: batch unmap folios belonging to uffd-wp VMAs mm/swapfile: Add batched version of folio_dup_swap mm/swapfile: Add batched version of folio_put_swap mm/rmap: Add batched version of folio_try_share_anon_rmap_pte mm/rmap: enable batch unmapping of anonymous folios

 include/linux/mm.h        |  11 ++
 include/linux/mm_inline.h |  34 ++--
 include/linux/rmap.h      |  27 ++-
 mm/internal.h             |  26 +++
 mm/memory.c               |  26 +--
 mm/mprotect.c             |  17 --
 mm/rmap.c                 | 405 ++++++++++++++++++++++++--------------
 mm/shmem.c                |   8 +-
 mm/swap.h                 |  23 ++-
 mm/swapfile.c             |  42 ++--
 10 files changed, 383 insertions(+), 236 deletions(-)

Signature · 1 line

2.34.1

Dev Jain

author

2026-05-06 09:45 UTC #1 lore raw

PATCH v3 1/9 mm/rmap: initialize nr_pages to 1 at loop start in try_to_unmap_one

Initialize nr_pages to 1 at the start of each loop iteration, like folio_referenced_one() does.

Without this, nr_pages computed by a previous folio_unmap_pte_batch() call can be reused on a later iteration that does not run folio_unmap_pte_batch() again.

I don’t think this is causing a bug today, but it is fragile.

A real bug would require this sequence within the same try_to_unmap_one() call:

1. Hit the pte_present(pteval) branch and set nr_pages > 1.
2. Later hit the else branch and do pte_clear() for device-exclusive PTE, and execute rest of the code with nr_pages > 1.

Executing the above would imply a lazyfree folio is mapped by a mix of present PTEs and device-exclusive PTEs.

In practice, device-exclusive PTEs imply a GUP pin on the folio, and lazyfree unmapping aborts try_to_unmap_one() when it detects that condition. So today this likely does not manifest, but initializing nr_pages per-iteration is still the correct and safer behavior.

Signed-off-by Dev Jain <[email protected]>

Acked-by Barry Song <[email protected]>

---
 mm/rmap.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

mm/rmap.c +3 -1

index fb3c351f8c458..a5f067a09de0f 100644
@@ -1991,7 +1991,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 	struct page *subpage;
 	struct mmu_notifier_range range;
 	enum ttu_flags flags = (enum ttu_flags)(long)arg;
-	unsigned long nr_pages = 1, end_addr;
+	unsigned long nr_pages;
+	unsigned long end_addr;
 	unsigned long pfn;
 	unsigned long hsz = 0;
 	int ptes = 0;

@@ -2030,6 +2031,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 	mmu_notifier_invalidate_range_start(&range);
 
 	while (page_vma_mapped_walk(&pvmw)) {
+		nr_pages = 1;
 		/*
 		 * If the folio is in an mlock()d vma, we must not swap it out.
 		 */

Signature · 1 line

2.34.1

David Hildenbrand (Arm)

maintainer

2026-05-11 06:48 UTC #2 lore raw

Re: [PATCH v3 1/9] mm/rmap: initialize nr_pages to 1 at loop start in try_to_unmap_one

Quoted 7 lines · Dev Jain

Initialize nr_pages to 1 at the start of each loop iteration, like folio_referenced_one() does.
Without this, nr_pages computed by a previous folio_unmap_pte_batch() call can be reused on a later iteration that does not run folio_unmap_pte_batch() again.

Quoted 8 lines

I don’t think this is causing a bug today, but it is fragile.
A real bug would require this sequence within the same try_to_unmap_one() call:
1. Hit the pte_present(pteval) branch and set nr_pages > 1.
2. Later hit the else branch and do pte_clear() for device-exclusive PTE, and execute rest of the code with nr_pages > 1.

Right, for hugetlb folios it should always stay at 1.

Quoted 3 lines

Executing the above would imply a lazyfree folio is mapped by a mix of present PTEs and device-exclusive PTEs.

Why lazyfree? We use nr_pages also for

	folio_remove_rmap_ptes(folio, subpage, nr_pages, vma);

and

	folio_put_refs(folio, nr_pages);

Given that make_device_exclusive() operates on individual PTEs, wouldn't it be possible to trigger that?

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-11 08:18 UTC #3 lore raw

Re: [PATCH v3 1/9] mm/rmap: initialize nr_pages to 1 at loop start in try_to_unmap_one

Quoted 25 lines · David Hildenbrand (Arm)

Quoted 7 lines · Dev Jain
Initialize nr_pages to 1 at the start of each loop iteration, like folio_referenced_one() does.
Without this, nr_pages computed by a previous folio_unmap_pte_batch() call can be reused on a later iteration that does not run folio_unmap_pte_batch() again.
Quoted 8 lines
I don’t think this is causing a bug today, but it is fragile.
A real bug would require this sequence within the same try_to_unmap_one() call:
1. Hit the pte_present(pteval) branch and set nr_pages > 1.
2. Later hit the else branch and do pte_clear() for device-exclusive PTE, and execute rest of the code with nr_pages > 1.
Right, for hugetlb folios it should always stay at 1.
Quoted 3 lines
Executing the above would imply a lazyfree folio is mapped by a mix of present PTEs and device-exclusive PTEs.
Why lazyfree? We use nr_pages also for

Quoted 8 lines

	folio_remove_rmap_ptes(folio, subpage, nr_pages, vma);
and
	folio_put_refs(folio, nr_pages);
Given that make_device_exclusive() operates on individual PTEs, wouldn't it be possible to trigger that?

At the point of this patch, batching is supported for lazyfree and file folios. make_device_exclusive does not operate on file folios.

Quoted 2 lines

David Hildenbrand (Arm)

maintainer

2026-05-11 08:32 UTC #4 lore raw

Re: [PATCH v3 1/9] mm/rmap: initialize nr_pages to 1 at loop start in try_to_unmap_one

Quoted 32 lines · Dev Jain

Quoted 29 lines · David Hildenbrand (Arm)
Quoted 7 lines · Dev Jain
Initialize nr_pages to 1 at the start of each loop iteration, like folio_referenced_one() does.
Without this, nr_pages computed by a previous folio_unmap_pte_batch() call can be reused on a later iteration that does not run folio_unmap_pte_batch() again.
Quoted 8 lines
I don’t think this is causing a bug today, but it is fragile.
A real bug would require this sequence within the same try_to_unmap_one() call:
1. Hit the pte_present(pteval) branch and set nr_pages > 1.
2. Later hit the else branch and do pte_clear() for device-exclusive PTE, and execute rest of the code with nr_pages > 1.
Right, for hugetlb folios it should always stay at 1.
Quoted 3 lines
Executing the above would imply a lazyfree folio is mapped by a mix of present PTEs and device-exclusive PTEs.
Why lazyfree? We use nr_pages also for
	folio_remove_rmap_ptes(folio, subpage, nr_pages, vma);
and

Quoted 7 lines

Quoted 4 lines
	folio_put_refs(folio, nr_pages);
Given that make_device_exclusive() operates on individual PTEs, wouldn't it be possible to trigger that?
At the point of this patch, batching is supported for lazyfree and file folios. make_device_exclusive does not operate on file folios.

That makes sense.

You write "In practice, device-exclusive PTEs imply a GUP pin on the folio, and lazyfree unmapping aborts try_to_unmap_one() when it detects that condition. ".

But I don't think the get_user_page_vma_remote() will set the pte/folio dirty?

And the pin is only temporary. The caller of make_device_exclusive() will essentially immediately drop that reference.

So can't we just hit that?

1) Mark PTE-mapped folio lazyfree. Folio+ptes are clean. Can still be writable.

2) Convert last PTE to device-exclusive. get_user_page_vma_remote() only need writable ptes, not dirty ptes. Caller drops the reference.

3) try_to_unmap_one()

Note that make_device_exclusive() documents: "device-exclusive entries are considered "clean" and "old" by core-mm. Device drivers must update the folio state when informed by MMU notifiers."

But if it wasn't dirtied, there should be nothing guaranteeing that MMU notifiers will set the folio dirty when MMU notifiers are triggered.

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-12 08:14 UTC #5 lore raw

Re: [PATCH v3 1/9] mm/rmap: initialize nr_pages to 1 at loop start in try_to_unmap_one

Quoted 62 lines · David Hildenbrand (Arm)

Quoted 32 lines · Dev Jain
Quoted 29 lines · David Hildenbrand (Arm)
Quoted 7 lines · Dev Jain
Initialize nr_pages to 1 at the start of each loop iteration, like folio_referenced_one() does.
Without this, nr_pages computed by a previous folio_unmap_pte_batch() call can be reused on a later iteration that does not run folio_unmap_pte_batch() again.
Quoted 8 lines
I don’t think this is causing a bug today, but it is fragile.
A real bug would require this sequence within the same try_to_unmap_one() call:
1. Hit the pte_present(pteval) branch and set nr_pages > 1.
2. Later hit the else branch and do pte_clear() for device-exclusive PTE, and execute rest of the code with nr_pages > 1.
Right, for hugetlb folios it should always stay at 1.
Quoted 3 lines
Executing the above would imply a lazyfree folio is mapped by a mix of present PTEs and device-exclusive PTEs.
Why lazyfree? We use nr_pages also for
	folio_remove_rmap_ptes(folio, subpage, nr_pages, vma);
and
Quoted 7 lines
Quoted 4 lines
	folio_put_refs(folio, nr_pages);
Given that make_device_exclusive() operates on individual PTEs, wouldn't it be possible to trigger that?
At the point of this patch, batching is supported for lazyfree and file folios. make_device_exclusive does not operate on file folios.
That makes sense.
You write "In practice, device-exclusive PTEs imply a GUP pin on the folio, and lazyfree unmapping aborts try_to_unmap_one() when it detects that condition. ".
But I don't think the get_user_page_vma_remote() will set the pte/folio dirty?
And the pin is only temporary. The caller of make_device_exclusive() will essentially immediately drop that reference.
So can't we just hit that?
1) Mark PTE-mapped folio lazyfree. Folio+ptes are clean. Can still be writable.
2) Convert last PTE to device-exclusive. get_user_page_vma_remote() only need writable ptes, not dirty ptes. Caller drops the reference.
3) try_to_unmap_one()

Quoted 6 lines

Note that make_device_exclusive() documents: "device-exclusive entries are considered "clean" and "old" by core-mm. Device drivers must update the folio state when informed by MMU notifiers."
But if it wasn't dirtied, there should be nothing guaranteeing that MMU notifiers will set the folio dirty when MMU notifiers are triggered.

You are correct.

I did some changes in hmm-tests.c, to mmap and fault in 64K folios, MADV_FREE them, then trigger make_device_exclusive() via hmm_dmirror_cmd() on the last 4K part of the mapping, then trigger reclaim. I get:

[   96.896674] added new 256 MB chunk (total 1 chunks, 256 MB) PFNs [0x800030000 0x800040000)
[   96.897857] added new 256 MB chunk (total 1 chunks, 256 MB) PFNs [0x800020000 0x800030000)
[   96.898181] HMM test module loaded. This is only for testing HMM.
[   97.136132] page: refcount:17 mapcount:1 mapping:0000000000000000 index:0xfffff7bf0 pfn:0xc1a00
[   97.136160] head: order:4 mapcount:16 entire_mapcount:0 nr_pages_mapped:16 pincount:0
[   97.136211] memcg:ffff00019d433040
[   97.136219] anon flags: 0x1ffff000000085d(locked|referenced|uptodate|dirty|owner_2|head|node=0|zone=0|lastcpupid=0x1ffff|kasantag=0x0)
[   97.136264] raw: 01ffff000000085d dead000000000100 dead000000000122 ffff0000030f8781
[   97.136391] raw: 0000000fffff7bf0 0000000000000000 0000001100000000 ffff00019d433040
[   97.136587] head: 01ffff000000085d dead000000000100 dead000000000122 ffff0000030f8781
[   97.136828] head: 0000000fffff7bf0 0000000000000000 0000001100000000 ffff00019d433040
[   97.137083] head: 01ffff0000000a04 fffffdffc2068001 000000100000000f 00000000ffffffff
[   97.137090] head: ffffffff0000000f 0000000000000021 0000000000000000 0000000000000010
[   97.137096] page dumped because: VM_WARN_ON_FOLIO(!((!!(((pte).pte) & (((pteval_t)(1)) << 0))) || ((((pte).pte) & ((((pteval_t)(1)) << 0) |
((((pteval_t)(1)) << 11)))) == ((((pteval_t)(1)) << 11)))))
[   97.137122] ------------[ cut here ]------------
[   97.137125] WARNING: mm/internal.h:346 at folio_pte_batch+0x54/0x360, CPU#4: hmm-tests/2283
[   97.137206] Modules linked in: test_hmm
[   97.137234] CPU: 4 UID: 0 PID: 2283 Comm: hmm-tests Not tainted 7.1.0-rc1+ #17 PREEMPT
[   97.137237] Hardware name: linux,dummy-virt (DT)
[   97.137238] pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[   97.137247] pc : folio_pte_batch+0x54/0x360
[   97.137253] lr : folio_pte_batch+0x54/0x360
[   97.137254] sp : ffff80008e7a3490
[   97.137263] x29: ffff80008e7a3490 x28: 0000000000000001 x27: 0000fffff7dff000
[   97.137266] x26: ffff0000451ceff0 x25: ffff000040fcaf00 x24: 00000000c1a0f780
[   97.137269] x23: 0000000000001000 x22: fffffdffc2068000 x21: fffffdffc2068000
[   97.137272] x20: ffff0000451ceff8 x19: 0000000000000001 x18: 0000000000000010
[   97.137274] x17: 3030303030303020 x16: 3030303030303030 x15: 5f6c617665747028
[   97.137276] x14: 282828207c202930 x13: 29312829745f6c61 x12: 7665747028282828
[   97.137277] x11: 2929292929313120 x10: ffff8000838feb80 x9 : ffff800080287cb8
[   97.137280] x8 : 3fffffffffffefff x7 : ffff8000838feb80 x6 : 0000000000000000
[   97.137281] x5 : ffff0002fe74a0c8 x4 : 0000000000000000 x3 : 0000000000000000
[   97.137282] x2 : 0000000000000000 x1 : ffff00014e120000 x0 : 00000000000000bb
[   97.137284] Call trace:
[   97.137285]  folio_pte_batch+0x54/0x360 (P)
[   97.137288]  folio_referenced_one+0x398/0x638
[   97.137295]  rmap_walk_anon+0x100/0x250
[   97.137296]  folio_referenced+0x17c/0x248
[   97.137297]  shrink_folio_list+0xf38/0x1968
[   97.137307]  shrink_lruvec+0x610/0xae8
[   97.137311]  shrink_node+0x218/0x888
[   97.137314]  __node_reclaim.constprop.0+0x98/0x328
[   97.137318]  user_proactive_reclaim+0x2b0/0x350
[   97.137320]  reclaim_store+0x3c/0x60
[   97.137321]  dev_attr_store+0x20/0x40
[   97.137338]  sysfs_kf_write+0x84/0xa8
[   97.137351]  kernfs_fop_write_iter+0x130/0x1c8
[   97.137352]  vfs_write+0x2c0/0x370
[   97.137360]  ksys_write+0x74/0x118
[   97.137362]  __arm64_sys_write+0x24/0x38
[   97.137363]  invoke_syscall+0x5c/0x120
[   97.137374]  el0_svc_common.constprop.0+0x48/0xf8
[   97.137376]  do_el0_svc+0x28/0x40
[   97.137377]  el0_svc+0x38/0x168
[   97.137396]  el0t_64_sync_handler+0xa0/0xe8
[   97.137398]  el0t_64_sync+0x1a4/0x1a8
[   97.137400] ---[ end trace 0000000000000000 ]---

the warning happens in folio_referenced_one -> folio_pte_batch -> !pte_present(). Not sure why it happens in folio_referenced_one instead of try_to_unmap_one.

I set nr_pages = 1 at the start of the pvmw walk in try_to_unmap_one and this goes away.

Will send this as a separate fix patch.

David Hildenbrand (Arm)

maintainer

2026-05-12 08:17 UTC #6 lore raw

Re: [PATCH v3 1/9] mm/rmap: initialize nr_pages to 1 at loop start in try_to_unmap_one

Quoted 104 lines · Dev Jain

Quoted 28 lines · David Hildenbrand (Arm)

Quoted 5 lines · Dev Jain
At the point of this patch, batching is supported for lazyfree and file folios. make_device_exclusive does not operate on file folios.
That makes sense.
You write "In practice, device-exclusive PTEs imply a GUP pin on the folio, and lazyfree unmapping aborts try_to_unmap_one() when it detects that condition. ".
But I don't think the get_user_page_vma_remote() will set the pte/folio dirty?
And the pin is only temporary. The caller of make_device_exclusive() will essentially immediately drop that reference.
So can't we just hit that?
1) Mark PTE-mapped folio lazyfree. Folio+ptes are clean. Can still be writable.
2) Convert last PTE to device-exclusive. get_user_page_vma_remote() only need writable ptes, not dirty ptes. Caller drops the reference.
3) try_to_unmap_one()

Quoted 6 lines

Note that make_device_exclusive() documents: "device-exclusive entries are considered "clean" and "old" by core-mm. Device drivers must update the folio state when informed by MMU notifiers."
But if it wasn't dirtied, there should be nothing guaranteeing that MMU notifiers will set the folio dirty when MMU notifiers are triggered.

You are correct.

[   96.896674] added new 256 MB chunk (total 1 chunks, 256 MB) PFNs [0x800030000 0x800040000)
[   96.897857] added new 256 MB chunk (total 1 chunks, 256 MB) PFNs [0x800020000 0x800030000)
[   96.898181] HMM test module loaded. This is only for testing HMM.
[   97.136132] page: refcount:17 mapcount:1 mapping:0000000000000000 index:0xfffff7bf0 pfn:0xc1a00
[   97.136160] head: order:4 mapcount:16 entire_mapcount:0 nr_pages_mapped:16 pincount:0
[   97.136211] memcg:ffff00019d433040
[   97.136219] anon flags: 0x1ffff000000085d(locked|referenced|uptodate|dirty|owner_2|head|node=0|zone=0|lastcpupid=0x1ffff|kasantag=0x0)
[   97.136264] raw: 01ffff000000085d dead000000000100 dead000000000122 ffff0000030f8781
[   97.136391] raw: 0000000fffff7bf0 0000000000000000 0000001100000000 ffff00019d433040
[   97.136587] head: 01ffff000000085d dead000000000100 dead000000000122 ffff0000030f8781
[   97.136828] head: 0000000fffff7bf0 0000000000000000 0000001100000000 ffff00019d433040
[   97.137083] head: 01ffff0000000a04 fffffdffc2068001 000000100000000f 00000000ffffffff
[   97.137090] head: ffffffff0000000f 0000000000000021 0000000000000000 0000000000000010
[   97.137096] page dumped because: VM_WARN_ON_FOLIO(!((!!(((pte).pte) & (((pteval_t)(1)) << 0))) || ((((pte).pte) & ((((pteval_t)(1)) << 0) |
((((pteval_t)(1)) << 11)))) == ((((pteval_t)(1)) << 11)))))
[   97.137122] ------------[ cut here ]------------
[   97.137125] WARNING: mm/internal.h:346 at folio_pte_batch+0x54/0x360, CPU#4: hmm-tests/2283
[   97.137206] Modules linked in: test_hmm
[   97.137234] CPU: 4 UID: 0 PID: 2283 Comm: hmm-tests Not tainted 7.1.0-rc1+ #17 PREEMPT
[   97.137237] Hardware name: linux,dummy-virt (DT)
[   97.137238] pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[   97.137247] pc : folio_pte_batch+0x54/0x360
[   97.137253] lr : folio_pte_batch+0x54/0x360
[   97.137254] sp : ffff80008e7a3490
[   97.137263] x29: ffff80008e7a3490 x28: 0000000000000001 x27: 0000fffff7dff000
[   97.137266] x26: ffff0000451ceff0 x25: ffff000040fcaf00 x24: 00000000c1a0f780
[   97.137269] x23: 0000000000001000 x22: fffffdffc2068000 x21: fffffdffc2068000
[   97.137272] x20: ffff0000451ceff8 x19: 0000000000000001 x18: 0000000000000010
[   97.137274] x17: 3030303030303020 x16: 3030303030303030 x15: 5f6c617665747028
[   97.137276] x14: 282828207c202930 x13: 29312829745f6c61 x12: 7665747028282828
[   97.137277] x11: 2929292929313120 x10: ffff8000838feb80 x9 : ffff800080287cb8
[   97.137280] x8 : 3fffffffffffefff x7 : ffff8000838feb80 x6 : 0000000000000000
[   97.137281] x5 : ffff0002fe74a0c8 x4 : 0000000000000000 x3 : 0000000000000000
[   97.137282] x2 : 0000000000000000 x1 : ffff00014e120000 x0 : 00000000000000bb
[   97.137284] Call trace:
[   97.137285]  folio_pte_batch+0x54/0x360 (P)
[   97.137288]  folio_referenced_one+0x398/0x638
[   97.137295]  rmap_walk_anon+0x100/0x250
[   97.137296]  folio_referenced+0x17c/0x248
[   97.137297]  shrink_folio_list+0xf38/0x1968
[   97.137307]  shrink_lruvec+0x610/0xae8
[   97.137311]  shrink_node+0x218/0x888
[   97.137314]  __node_reclaim.constprop.0+0x98/0x328
[   97.137318]  user_proactive_reclaim+0x2b0/0x350
[   97.137320]  reclaim_store+0x3c/0x60
[   97.137321]  dev_attr_store+0x20/0x40
[   97.137338]  sysfs_kf_write+0x84/0xa8
[   97.137351]  kernfs_fop_write_iter+0x130/0x1c8
[   97.137352]  vfs_write+0x2c0/0x370
[   97.137360]  ksys_write+0x74/0x118
[   97.137362]  __arm64_sys_write+0x24/0x38
[   97.137363]  invoke_syscall+0x5c/0x120
[   97.137374]  el0_svc_common.constprop.0+0x48/0xf8
[   97.137376]  do_el0_svc+0x28/0x40
[   97.137377]  el0_svc+0x38/0x168
[   97.137396]  el0t_64_sync_handler+0xa0/0xe8
[   97.137398]  el0t_64_sync+0x1a4/0x1a8
[   97.137400] ---[ end trace 0000000000000000 ]---

Quoted 7 lines

the warning happens in folio_referenced_one -> folio_pte_batch -> !pte_present(). Not sure why it happens in folio_referenced_one instead of try_to_unmap_one.
I set nr_pages = 1 at the start of the pvmw walk in try_to_unmap_one and this goes away.
Will send this as a separate fix patch.

Awesome, thanks! (CC stable)

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-12 10:49 UTC #7 lore raw

Re: [PATCH v3 1/9] mm/rmap: initialize nr_pages to 1 at loop start in try_to_unmap_one

Quoted 108 lines · David Hildenbrand (Arm)

Quoted 107 lines · Dev Jain

Quoted 28 lines · David Hildenbrand (Arm)

Quoted 5 lines · Dev Jain
At the point of this patch, batching is supported for lazyfree and file folios. make_device_exclusive does not operate on file folios.
That makes sense.
You write "In practice, device-exclusive PTEs imply a GUP pin on the folio, and lazyfree unmapping aborts try_to_unmap_one() when it detects that condition. ".
But I don't think the get_user_page_vma_remote() will set the pte/folio dirty?
And the pin is only temporary. The caller of make_device_exclusive() will essentially immediately drop that reference.
So can't we just hit that?
1) Mark PTE-mapped folio lazyfree. Folio+ptes are clean. Can still be writable.
2) Convert last PTE to device-exclusive. get_user_page_vma_remote() only need writable ptes, not dirty ptes. Caller drops the reference.
3) try_to_unmap_one()

Quoted 6 lines

Note that make_device_exclusive() documents: "device-exclusive entries are considered "clean" and "old" by core-mm. Device drivers must update the folio state when informed by MMU notifiers."
But if it wasn't dirtied, there should be nothing guaranteeing that MMU notifiers will set the folio dirty when MMU notifiers are triggered.

You are correct.

[   96.896674] added new 256 MB chunk (total 1 chunks, 256 MB) PFNs [0x800030000 0x800040000)
[   96.897857] added new 256 MB chunk (total 1 chunks, 256 MB) PFNs [0x800020000 0x800030000)
[   96.898181] HMM test module loaded. This is only for testing HMM.
[   97.136132] page: refcount:17 mapcount:1 mapping:0000000000000000 index:0xfffff7bf0 pfn:0xc1a00
[   97.136160] head: order:4 mapcount:16 entire_mapcount:0 nr_pages_mapped:16 pincount:0
[   97.136211] memcg:ffff00019d433040
[   97.136219] anon flags: 0x1ffff000000085d(locked|referenced|uptodate|dirty|owner_2|head|node=0|zone=0|lastcpupid=0x1ffff|kasantag=0x0)
[   97.136264] raw: 01ffff000000085d dead000000000100 dead000000000122 ffff0000030f8781
[   97.136391] raw: 0000000fffff7bf0 0000000000000000 0000001100000000 ffff00019d433040
[   97.136587] head: 01ffff000000085d dead000000000100 dead000000000122 ffff0000030f8781
[   97.136828] head: 0000000fffff7bf0 0000000000000000 0000001100000000 ffff00019d433040
[   97.137083] head: 01ffff0000000a04 fffffdffc2068001 000000100000000f 00000000ffffffff
[   97.137090] head: ffffffff0000000f 0000000000000021 0000000000000000 0000000000000010
[   97.137096] page dumped because: VM_WARN_ON_FOLIO(!((!!(((pte).pte) & (((pteval_t)(1)) << 0))) || ((((pte).pte) & ((((pteval_t)(1)) << 0) |
((((pteval_t)(1)) << 11)))) == ((((pteval_t)(1)) << 11)))))
[   97.137122] ------------[ cut here ]------------
[   97.137125] WARNING: mm/internal.h:346 at folio_pte_batch+0x54/0x360, CPU#4: hmm-tests/2283
[   97.137206] Modules linked in: test_hmm
[   97.137234] CPU: 4 UID: 0 PID: 2283 Comm: hmm-tests Not tainted 7.1.0-rc1+ #17 PREEMPT
[   97.137237] Hardware name: linux,dummy-virt (DT)
[   97.137238] pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[   97.137247] pc : folio_pte_batch+0x54/0x360
[   97.137253] lr : folio_pte_batch+0x54/0x360
[   97.137254] sp : ffff80008e7a3490
[   97.137263] x29: ffff80008e7a3490 x28: 0000000000000001 x27: 0000fffff7dff000
[   97.137266] x26: ffff0000451ceff0 x25: ffff000040fcaf00 x24: 00000000c1a0f780
[   97.137269] x23: 0000000000001000 x22: fffffdffc2068000 x21: fffffdffc2068000
[   97.137272] x20: ffff0000451ceff8 x19: 0000000000000001 x18: 0000000000000010
[   97.137274] x17: 3030303030303020 x16: 3030303030303030 x15: 5f6c617665747028
[   97.137276] x14: 282828207c202930 x13: 29312829745f6c61 x12: 7665747028282828
[   97.137277] x11: 2929292929313120 x10: ffff8000838feb80 x9 : ffff800080287cb8
[   97.137280] x8 : 3fffffffffffefff x7 : ffff8000838feb80 x6 : 0000000000000000
[   97.137281] x5 : ffff0002fe74a0c8 x4 : 0000000000000000 x3 : 0000000000000000
[   97.137282] x2 : 0000000000000000 x1 : ffff00014e120000 x0 : 00000000000000bb
[   97.137284] Call trace:
[   97.137285]  folio_pte_batch+0x54/0x360 (P)
[   97.137288]  folio_referenced_one+0x398/0x638
[   97.137295]  rmap_walk_anon+0x100/0x250
[   97.137296]  folio_referenced+0x17c/0x248
[   97.137297]  shrink_folio_list+0xf38/0x1968
[   97.137307]  shrink_lruvec+0x610/0xae8
[   97.137311]  shrink_node+0x218/0x888
[   97.137314]  __node_reclaim.constprop.0+0x98/0x328
[   97.137318]  user_proactive_reclaim+0x2b0/0x350
[   97.137320]  reclaim_store+0x3c/0x60
[   97.137321]  dev_attr_store+0x20/0x40
[   97.137338]  sysfs_kf_write+0x84/0xa8
[   97.137351]  kernfs_fop_write_iter+0x130/0x1c8
[   97.137352]  vfs_write+0x2c0/0x370
[   97.137360]  ksys_write+0x74/0x118
[   97.137362]  __arm64_sys_write+0x24/0x38
[   97.137363]  invoke_syscall+0x5c/0x120
[   97.137374]  el0_svc_common.constprop.0+0x48/0xf8
[   97.137376]  do_el0_svc+0x28/0x40
[   97.137377]  el0_svc+0x38/0x168
[   97.137396]  el0t_64_sync_handler+0xa0/0xe8
[   97.137398]  el0t_64_sync+0x1a4/0x1a8
[   97.137400] ---[ end trace 0000000000000000 ]---

the warning happens in folio_referenced_one -> folio_pte_batch -> !pte_present(). Not sure why it happens in folio_referenced_one instead of try_to_unmap_one.

Quoted 6 lines

Quoted 4 lines
I set nr_pages = 1 at the start of the pvmw walk in try_to_unmap_one and this goes away.
Will send this as a separate fix patch.
Awesome, thanks! (CC stable)

Okay I think there is another bug. In folio_referenced_one,

	if (folio_test_large(folio)) {
		unsigned long end_addr = pmd_addr_end(address, vma->vm_end);
		unsigned int max_nr = (end_addr - address) >> PAGE_SHIFT;
		pte_t pteval = ptep_get(pvmw.pte);

		nr = folio_pte_batch(folio, pvmw.pte,
				     pteval, max_nr);
	}

There is no pte_present(pteval) check here. We will encounter a non-present entry in folio_pte_batch(), giving the trace above.

Quoted 1 line

David Hildenbrand (Arm)

maintainer

2026-05-12 11:01 UTC #8 lore raw

Re: [PATCH v3 1/9] mm/rmap: initialize nr_pages to 1 at loop start in try_to_unmap_one

Quoted 94 lines · Dev Jain

Quoted 73 lines · David Hildenbrand (Arm)

Quoted 72 lines · Dev Jain

You are correct.

[   96.896674] added new 256 MB chunk (total 1 chunks, 256 MB) PFNs [0x800030000 0x800040000)
[   96.897857] added new 256 MB chunk (total 1 chunks, 256 MB) PFNs [0x800020000 0x800030000)
[   96.898181] HMM test module loaded. This is only for testing HMM.
[   97.136132] page: refcount:17 mapcount:1 mapping:0000000000000000 index:0xfffff7bf0 pfn:0xc1a00
[   97.136160] head: order:4 mapcount:16 entire_mapcount:0 nr_pages_mapped:16 pincount:0
[   97.136211] memcg:ffff00019d433040
[   97.136219] anon flags: 0x1ffff000000085d(locked|referenced|uptodate|dirty|owner_2|head|node=0|zone=0|lastcpupid=0x1ffff|kasantag=0x0)
[   97.136264] raw: 01ffff000000085d dead000000000100 dead000000000122 ffff0000030f8781
[   97.136391] raw: 0000000fffff7bf0 0000000000000000 0000001100000000 ffff00019d433040
[   97.136587] head: 01ffff000000085d dead000000000100 dead000000000122 ffff0000030f8781
[   97.136828] head: 0000000fffff7bf0 0000000000000000 0000001100000000 ffff00019d433040
[   97.137083] head: 01ffff0000000a04 fffffdffc2068001 000000100000000f 00000000ffffffff
[   97.137090] head: ffffffff0000000f 0000000000000021 0000000000000000 0000000000000010
[   97.137096] page dumped because: VM_WARN_ON_FOLIO(!((!!(((pte).pte) & (((pteval_t)(1)) << 0))) || ((((pte).pte) & ((((pteval_t)(1)) << 0) |
((((pteval_t)(1)) << 11)))) == ((((pteval_t)(1)) << 11)))))
[   97.137122] ------------[ cut here ]------------
[   97.137125] WARNING: mm/internal.h:346 at folio_pte_batch+0x54/0x360, CPU#4: hmm-tests/2283
[   97.137206] Modules linked in: test_hmm
[   97.137234] CPU: 4 UID: 0 PID: 2283 Comm: hmm-tests Not tainted 7.1.0-rc1+ #17 PREEMPT
[   97.137237] Hardware name: linux,dummy-virt (DT)
[   97.137238] pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[   97.137247] pc : folio_pte_batch+0x54/0x360
[   97.137253] lr : folio_pte_batch+0x54/0x360
[   97.137254] sp : ffff80008e7a3490
[   97.137263] x29: ffff80008e7a3490 x28: 0000000000000001 x27: 0000fffff7dff000
[   97.137266] x26: ffff0000451ceff0 x25: ffff000040fcaf00 x24: 00000000c1a0f780
[   97.137269] x23: 0000000000001000 x22: fffffdffc2068000 x21: fffffdffc2068000
[   97.137272] x20: ffff0000451ceff8 x19: 0000000000000001 x18: 0000000000000010
[   97.137274] x17: 3030303030303020 x16: 3030303030303030 x15: 5f6c617665747028
[   97.137276] x14: 282828207c202930 x13: 29312829745f6c61 x12: 7665747028282828
[   97.137277] x11: 2929292929313120 x10: ffff8000838feb80 x9 : ffff800080287cb8
[   97.137280] x8 : 3fffffffffffefff x7 : ffff8000838feb80 x6 : 0000000000000000
[   97.137281] x5 : ffff0002fe74a0c8 x4 : 0000000000000000 x3 : 0000000000000000
[   97.137282] x2 : 0000000000000000 x1 : ffff00014e120000 x0 : 00000000000000bb
[   97.137284] Call trace:
[   97.137285]  folio_pte_batch+0x54/0x360 (P)
[   97.137288]  folio_referenced_one+0x398/0x638
[   97.137295]  rmap_walk_anon+0x100/0x250
[   97.137296]  folio_referenced+0x17c/0x248
[   97.137297]  shrink_folio_list+0xf38/0x1968
[   97.137307]  shrink_lruvec+0x610/0xae8
[   97.137311]  shrink_node+0x218/0x888
[   97.137314]  __node_reclaim.constprop.0+0x98/0x328
[   97.137318]  user_proactive_reclaim+0x2b0/0x350
[   97.137320]  reclaim_store+0x3c/0x60
[   97.137321]  dev_attr_store+0x20/0x40
[   97.137338]  sysfs_kf_write+0x84/0xa8
[   97.137351]  kernfs_fop_write_iter+0x130/0x1c8
[   97.137352]  vfs_write+0x2c0/0x370
[   97.137360]  ksys_write+0x74/0x118
[   97.137362]  __arm64_sys_write+0x24/0x38
[   97.137363]  invoke_syscall+0x5c/0x120
[   97.137374]  el0_svc_common.constprop.0+0x48/0xf8
[   97.137376]  do_el0_svc+0x28/0x40
[   97.137377]  el0_svc+0x38/0x168
[   97.137396]  el0t_64_sync_handler+0xa0/0xe8
[   97.137398]  el0t_64_sync+0x1a4/0x1a8
[   97.137400] ---[ end trace 0000000000000000 ]---

the warning happens in folio_referenced_one -> folio_pte_batch -> !pte_present(). Not sure why it happens in folio_referenced_one instead of try_to_unmap_one.

Quoted 6 lines

Quoted 4 lines
I set nr_pages = 1 at the start of the pvmw walk in try_to_unmap_one and this goes away.
Will send this as a separate fix patch.
Awesome, thanks! (CC stable)

Okay I think there is another bug. In folio_referenced_one,

	if (folio_test_large(folio)) {
		unsigned long end_addr = pmd_addr_end(address, vma->vm_end);
		unsigned int max_nr = (end_addr - address) >> PAGE_SHIFT;
		pte_t pteval = ptep_get(pvmw.pte);

		nr = folio_pte_batch(folio, pvmw.pte,
				     pteval, max_nr);
	}

Quoted 2 lines

There is no pte_present(pteval) check here. We will encounter a non-present entry in folio_pte_batch(), giving the trace above.

clear_flush_young_ptes_notify() should also only get called for present PTEs.

See damon_ptep_mkold(), where we trigger mmu notifiers separately to handle exactly that.

I recall that I looked at that code in context of

	https://lore.kernel.org/all/[email protected]/T/#mf98677cb5a9419a5d695b2ed5427fdd75ed08dcb

And assumed that it would not be required in folio_referenced_one().

If only I could remember why I thought it would be ok ...

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-12 11:16 UTC #9 lore raw

Re: [PATCH v3 1/9] mm/rmap: initialize nr_pages to 1 at loop start in try_to_unmap_one

Quoted 103 lines · David Hildenbrand (Arm)

Quoted 94 lines · Dev Jain

Quoted 73 lines · David Hildenbrand (Arm)

Quoted 72 lines · Dev Jain

You are correct.

[   96.896674] added new 256 MB chunk (total 1 chunks, 256 MB) PFNs [0x800030000 0x800040000)
[   96.897857] added new 256 MB chunk (total 1 chunks, 256 MB) PFNs [0x800020000 0x800030000)
[   96.898181] HMM test module loaded. This is only for testing HMM.
[   97.136132] page: refcount:17 mapcount:1 mapping:0000000000000000 index:0xfffff7bf0 pfn:0xc1a00
[   97.136160] head: order:4 mapcount:16 entire_mapcount:0 nr_pages_mapped:16 pincount:0
[   97.136211] memcg:ffff00019d433040
[   97.136219] anon flags: 0x1ffff000000085d(locked|referenced|uptodate|dirty|owner_2|head|node=0|zone=0|lastcpupid=0x1ffff|kasantag=0x0)
[   97.136264] raw: 01ffff000000085d dead000000000100 dead000000000122 ffff0000030f8781
[   97.136391] raw: 0000000fffff7bf0 0000000000000000 0000001100000000 ffff00019d433040
[   97.136587] head: 01ffff000000085d dead000000000100 dead000000000122 ffff0000030f8781
[   97.136828] head: 0000000fffff7bf0 0000000000000000 0000001100000000 ffff00019d433040
[   97.137083] head: 01ffff0000000a04 fffffdffc2068001 000000100000000f 00000000ffffffff
[   97.137090] head: ffffffff0000000f 0000000000000021 0000000000000000 0000000000000010
[   97.137096] page dumped because: VM_WARN_ON_FOLIO(!((!!(((pte).pte) & (((pteval_t)(1)) << 0))) || ((((pte).pte) & ((((pteval_t)(1)) << 0) |
((((pteval_t)(1)) << 11)))) == ((((pteval_t)(1)) << 11)))))
[   97.137122] ------------[ cut here ]------------
[   97.137125] WARNING: mm/internal.h:346 at folio_pte_batch+0x54/0x360, CPU#4: hmm-tests/2283
[   97.137206] Modules linked in: test_hmm
[   97.137234] CPU: 4 UID: 0 PID: 2283 Comm: hmm-tests Not tainted 7.1.0-rc1+ #17 PREEMPT
[   97.137237] Hardware name: linux,dummy-virt (DT)
[   97.137238] pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[   97.137247] pc : folio_pte_batch+0x54/0x360
[   97.137253] lr : folio_pte_batch+0x54/0x360
[   97.137254] sp : ffff80008e7a3490
[   97.137263] x29: ffff80008e7a3490 x28: 0000000000000001 x27: 0000fffff7dff000
[   97.137266] x26: ffff0000451ceff0 x25: ffff000040fcaf00 x24: 00000000c1a0f780
[   97.137269] x23: 0000000000001000 x22: fffffdffc2068000 x21: fffffdffc2068000
[   97.137272] x20: ffff0000451ceff8 x19: 0000000000000001 x18: 0000000000000010
[   97.137274] x17: 3030303030303020 x16: 3030303030303030 x15: 5f6c617665747028
[   97.137276] x14: 282828207c202930 x13: 29312829745f6c61 x12: 7665747028282828
[   97.137277] x11: 2929292929313120 x10: ffff8000838feb80 x9 : ffff800080287cb8
[   97.137280] x8 : 3fffffffffffefff x7 : ffff8000838feb80 x6 : 0000000000000000
[   97.137281] x5 : ffff0002fe74a0c8 x4 : 0000000000000000 x3 : 0000000000000000
[   97.137282] x2 : 0000000000000000 x1 : ffff00014e120000 x0 : 00000000000000bb
[   97.137284] Call trace:
[   97.137285]  folio_pte_batch+0x54/0x360 (P)
[   97.137288]  folio_referenced_one+0x398/0x638
[   97.137295]  rmap_walk_anon+0x100/0x250
[   97.137296]  folio_referenced+0x17c/0x248
[   97.137297]  shrink_folio_list+0xf38/0x1968
[   97.137307]  shrink_lruvec+0x610/0xae8
[   97.137311]  shrink_node+0x218/0x888
[   97.137314]  __node_reclaim.constprop.0+0x98/0x328
[   97.137318]  user_proactive_reclaim+0x2b0/0x350
[   97.137320]  reclaim_store+0x3c/0x60
[   97.137321]  dev_attr_store+0x20/0x40
[   97.137338]  sysfs_kf_write+0x84/0xa8
[   97.137351]  kernfs_fop_write_iter+0x130/0x1c8
[   97.137352]  vfs_write+0x2c0/0x370
[   97.137360]  ksys_write+0x74/0x118
[   97.137362]  __arm64_sys_write+0x24/0x38
[   97.137363]  invoke_syscall+0x5c/0x120
[   97.137374]  el0_svc_common.constprop.0+0x48/0xf8
[   97.137376]  do_el0_svc+0x28/0x40
[   97.137377]  el0_svc+0x38/0x168
[   97.137396]  el0t_64_sync_handler+0xa0/0xe8
[   97.137398]  el0t_64_sync+0x1a4/0x1a8
[   97.137400] ---[ end trace 0000000000000000 ]---

the warning happens in folio_referenced_one -> folio_pte_batch -> !pte_present(). Not sure why it happens in folio_referenced_one instead of try_to_unmap_one.

Quoted 6 lines

Quoted 4 lines
I set nr_pages = 1 at the start of the pvmw walk in try_to_unmap_one and this goes away.
Will send this as a separate fix patch.
Awesome, thanks! (CC stable)

Okay I think there is another bug. In folio_referenced_one,

	if (folio_test_large(folio)) {
		unsigned long end_addr = pmd_addr_end(address, vma->vm_end);
		unsigned int max_nr = (end_addr - address) >> PAGE_SHIFT;
		pte_t pteval = ptep_get(pvmw.pte);

		nr = folio_pte_batch(folio, pvmw.pte,
				     pteval, max_nr);
	}

Quoted 2 lines

There is no pte_present(pteval) check here. We will encounter a non-present entry in folio_pte_batch(), giving the trace above.

clear_flush_young_ptes_notify() should also only get called for present PTEs.

See damon_ptep_mkold(), where we trigger mmu notifiers separately to handle exactly that.

Quoted 7 lines

I recall that I looked at that code in context of
	https://lore.kernel.org/all/[email protected]/T/#mf98677cb5a9419a5d695b2ed5427fdd75ed08dcb
And assumed that it would not be required in folio_referenced_one().
If only I could remember why I thought it would be ok ...

For your benefit, here is the reproducer. Replace the current TEST_F(hmm, exclusive) segment with the following:

void write_to_reclaim() {
    const char *path = "/sys/devices/system/node/node0/reclaim";
    const char *value = "409600000000";
    int fd = open(path, O_WRONLY);
    if (fd == -1) {
        perror("open");
        exit(EXIT_FAILURE);
    }

    if (write(fd, value, sizeof("409600000000") - 1) == -1) {
        perror("write");
        close(fd);
        exit(EXIT_FAILURE);
    }

    printf("Successfully wrote %s to %s\n", value, path);
    close(fd);
}

/*
* Basic check of exclusive faulting.
*/
TEST_F(hmm, exclusive)

{
	struct hmm_buffer buffer = {};
	unsigned long huge_size;
	unsigned long npages = 1;
	unsigned long i;
	unsigned char *mapping;
	void *raw_mapping;
	unsigned char *ptr;
	int ret;

	huge_size = 2 * 1024 * 1024;
	ASSERT_GE(huge_size, self->page_size * 2);

	buffer.fd = -1;
	buffer.size = self->page_size;
	buffer.mirror = malloc(buffer.size);
	ASSERT_NE(buffer.mirror, NULL);

	raw_mapping = mmap(NULL, 2 * huge_size, PROT_READ | PROT_WRITE,
			   MAP_PRIVATE | MAP_ANONYMOUS, buffer.fd, 0);
	ASSERT_NE(raw_mapping, MAP_FAILED);
	mapping = (unsigned char *)ALIGN((uintptr_t)raw_mapping, huge_size);

	memset(mapping, 0xab, huge_size);


	ret = madvise(mapping, huge_size, MADV_FREE);
	ASSERT_EQ(ret, 0);

	/*
	 * Exercise device-exclusive conversion on a single 4K page inside a
	 * lazyfree PMD-sized mapping, not on the whole mapping.
	 */
	buffer.ptr = mapping + huge_size - self->page_size;


	ret = hmm_dmirror_cmd(self->fd, HMM_DMIRROR_EXCLUSIVE, &buffer, npages);
	ASSERT_EQ(ret, 0);
	ASSERT_EQ(buffer.cpages, npages);

	write_to_reclaim();

	/* Give the lazyfree folio a chance to be reclaimed after exclusive conversion. */
	sleep(100);

	/* Check what the device read. */
	for (i = 0, ptr = buffer.mirror; i < buffer.size; ++i)
		ASSERT_EQ(ptr[i], 0xab);

	/* Fault the exclusive page back to system memory. */
	ptr = buffer.ptr;
	for (i = 0; i < buffer.size; ++i)
		ASSERT_EQ(ptr[i], 0xab);
	ptr[0] = 0xcd;

	/* Check atomic access revoked */
	ret = hmm_dmirror_cmd(self->fd, HMM_DMIRROR_CHECK_EXCLUSIVE, &buffer,
			      npages);
	ASSERT_EQ(ret, 0);

	ASSERT_EQ(munmap(raw_mapping, 2 * huge_size), 0);
	free(buffer.mirror);
}

Then, patch test_hmm.sh with

-       $(dirname "${BASH_SOURCE[0]}")/hmm-tests
+       $(dirname "${BASH_SOURCE[0]}")/hmm-tests \
+       -r hmm.hmm_device_private.exclusive

Set 2M thp to never and 64K thp to always.

Quoted 1 line

Dev Jain

author

2026-05-06 09:45 UTC #10 lore raw

PATCH v3 2/9 mm/rmap: refactor hugetlb pte clearing in try_to_unmap_one

In reply to Dev Jain

Simplify the code by refactoring the folio_test_hugetlb() branch into a new function.

While at it, convert BUG helpers to WARN helpers.

Signed-off-by Dev Jain <[email protected]>

---
 mm/rmap.c | 117 ++++++++++++++++++++++++++++++++----------------------
 1 file changed, 69 insertions(+), 48 deletions(-)

mm/rmap.c +69 ~~-48~~

index a5f067a09de0f..a98acdea0530a 100644
@@ -1978,6 +1978,68 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,
 				     FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY);
 }
 
+/* Returns false if unmap needs to be aborted */
+static inline bool unmap_hugetlb_folio(struct vm_area_struct *vma,
+		struct folio *folio, struct page_vma_mapped_walk *pvmw,
+		struct page *page, enum ttu_flags flags, pte_t *pteval,
+		struct mmu_notifier_range *range, bool *exit_walk)
+{
+	/*
+	 * The try_to_unmap() is only passed a hugetlb page
+	 * in the case where the hugetlb page is poisoned.
+	 */
+	VM_WARN_ON_PAGE(!PageHWPoison(page), page);
+	/*
+	 * huge_pmd_unshare may unmap an entire PMD page.
+	 * There is no way of knowing exactly which PMDs may
+	 * be cached for this mm, so we must flush them all.
+	 * start/end were already adjusted above to cover this
+	 * range.
+	 */
+	flush_cache_range(vma, range->start, range->end);
+
+	/*
+	 * To call huge_pmd_unshare, i_mmap_rwsem must be
+	 * held in write mode.  Caller needs to explicitly
+	 * do this outside rmap routines.
+	 *
+	 * We also must hold hugetlb vma_lock in write mode.
+	 * Lock order dictates acquiring vma_lock BEFORE
+	 * i_mmap_rwsem.  We can only try lock here and fail
+	 * if unsuccessful.
+	 */
+	if (!folio_test_anon(folio)) {
+		struct mmu_gather tlb;
+
+		VM_WARN_ON(!(flags & TTU_RMAP_LOCKED));
+		if (!hugetlb_vma_trylock_write(vma)) {
+			*exit_walk = true;
+			return false;
+		}
+
+		tlb_gather_mmu_vma(&tlb, vma);
+		if (huge_pmd_unshare(&tlb, vma, pvmw->address, pvmw->pte)) {
+			hugetlb_vma_unlock_write(vma);
+			huge_pmd_unshare_flush(&tlb, vma);
+			tlb_finish_mmu(&tlb);
+			/*
+			 * The PMD table was unmapped,
+			 * consequently unmapping the folio.
+			 */
+			*exit_walk = true;
+			return true;
+		}
+		hugetlb_vma_unlock_write(vma);
+		tlb_finish_mmu(&tlb);
+	}
+	*pteval = huge_ptep_clear_flush(vma, pvmw->address, pvmw->pte);
+	if (pte_dirty(*pteval))
+		folio_mark_dirty(folio);
+
+	*exit_walk = false;
+	return true;
+}
+
 /*
  * @arg: enum ttu_flags will be passed to this argument
  */

@@ -2115,56 +2177,15 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 				 PageAnonExclusive(subpage);
 
 		if (folio_test_hugetlb(folio)) {
-			bool anon = folio_test_anon(folio);
-
-			/*
-			 * The try_to_unmap() is only passed a hugetlb page
-			 * in the case where the hugetlb page is poisoned.
-			 */
-			VM_BUG_ON_PAGE(!PageHWPoison(subpage), subpage);
-			/*
-			 * huge_pmd_unshare may unmap an entire PMD page.
-			 * There is no way of knowing exactly which PMDs may
-			 * be cached for this mm, so we must flush them all.
-			 * start/end were already adjusted above to cover this
-			 * range.
-			 */
-			flush_cache_range(vma, range.start, range.end);
+			bool exit_walk;
 
-			/*
-			 * To call huge_pmd_unshare, i_mmap_rwsem must be
-			 * held in write mode.  Caller needs to explicitly
-			 * do this outside rmap routines.
-			 *
-			 * We also must hold hugetlb vma_lock in write mode.
-			 * Lock order dictates acquiring vma_lock BEFORE
-			 * i_mmap_rwsem.  We can only try lock here and fail
-			 * if unsuccessful.
-			 */
-			if (!anon) {
-				struct mmu_gather tlb;
-
-				VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
-				if (!hugetlb_vma_trylock_write(vma))
-					goto walk_abort;
-
-				tlb_gather_mmu_vma(&tlb, vma);
-				if (huge_pmd_unshare(&tlb, vma, address, pvmw.pte)) {
-					hugetlb_vma_unlock_write(vma);
-					huge_pmd_unshare_flush(&tlb, vma);
-					tlb_finish_mmu(&tlb);
-					/*
-					 * The PMD table was unmapped,
-					 * consequently unmapping the folio.
-					 */
-					goto walk_done;
-				}
-				hugetlb_vma_unlock_write(vma);
-				tlb_finish_mmu(&tlb);
+			ret = unmap_hugetlb_folio(vma, folio, &pvmw, subpage,
+						  flags, &pteval, &range,
+						  &exit_walk);
+			if (exit_walk) {
+				page_vma_mapped_walk_done(&pvmw);
+				break;
 			}
-			pteval = huge_ptep_clear_flush(vma, address, pvmw.pte);
-			if (pte_dirty(pteval))
-				folio_mark_dirty(folio);
 		} else if (likely(pte_present(pteval))) {
 			nr_pages = folio_unmap_pte_batch(folio, &pvmw, flags, pteval);
 			end_addr = address + nr_pages * PAGE_SIZE;

Signature · 1 line

2.34.1

David Hildenbrand (Arm)

maintainer

2026-05-11 07:10 UTC #11 lore raw

Re: [PATCH v3 2/9] mm/rmap: refactor hugetlb pte clearing in try_to_unmap_one

Quoted 15 lines · Dev Jain

Simplify the code by refactoring the folio_test_hugetlb() branch into a new function.
While at it, convert BUG helpers to WARN helpers.
Signed-off-by Dev Jain <[email protected]>
---
 mm/rmap.c | 117 ++++++++++++++++++++++++++++++++----------------------
 1 file changed, 69 insertions(+), 48 deletions(-)
mm/rmap.c +0 -0
index a5f067a09de0f..a98acdea0530a 100644

mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 1982

Quoted 5 lines

mm/rmap.c +2 -0

@@ -1978,6 +1978,68 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,
 				     FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY);
 }
 
+/* Returns false if unmap needs to be aborted */
+static inline bool unmap_hugetlb_folio(struct vm_area_struct *vma,

I'm wondering whether we should make it clearer that this belongs to the try_to_unmap family by calling it

ttu_hugetlb_folio

mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 1991

Quoted 9 lines

mm/rmap.c +9 -0

+		struct folio *folio, struct page_vma_mapped_walk *pvmw,
+		struct page *page, enum ttu_flags flags, pte_t *pteval,
+		struct mmu_notifier_range *range, bool *exit_walk)
+{
+	/*
+	 * The try_to_unmap() is only passed a hugetlb page
+	 * in the case where the hugetlb page is poisoned.
+	 */
+	VM_WARN_ON_PAGE(!PageHWPoison(page), page);

IIRC, we will never actually get a tail page here.

Can we avoid passing a page by checking instead whether the hugetlb folios is marked as having a poisoned page?

See the folio_test_set_hwpoison() in hugetlb_update_hwpoison().

So you can simply use folio_test_hwpoison here instead.

Quoted 39 lines

mm/rmap.c +39 -0

+	/*
+	 * huge_pmd_unshare may unmap an entire PMD page.
+	 * There is no way of knowing exactly which PMDs may
+	 * be cached for this mm, so we must flush them all.
+	 * start/end were already adjusted above to cover this
+	 * range.
+	 */
+	flush_cache_range(vma, range->start, range->end);
+
+	/*
+	 * To call huge_pmd_unshare, i_mmap_rwsem must be
+	 * held in write mode.  Caller needs to explicitly
+	 * do this outside rmap routines.
+	 *
+	 * We also must hold hugetlb vma_lock in write mode.
+	 * Lock order dictates acquiring vma_lock BEFORE
+	 * i_mmap_rwsem.  We can only try lock here and fail
+	 * if unsuccessful.
+	 */
+	if (!folio_test_anon(folio)) {
+		struct mmu_gather tlb;
+
+		VM_WARN_ON(!(flags & TTU_RMAP_LOCKED));
+		if (!hugetlb_vma_trylock_write(vma)) {
+			*exit_walk = true;
+			return false;
+		}
+
+		tlb_gather_mmu_vma(&tlb, vma);
+		if (huge_pmd_unshare(&tlb, vma, pvmw->address, pvmw->pte)) {
+			hugetlb_vma_unlock_write(vma);
+			huge_pmd_unshare_flush(&tlb, vma);
+			tlb_finish_mmu(&tlb);
+			/*
+			 * The PMD table was unmapped,
+			 * consequently unmapping the folio.
+			 */
+			*exit_walk = true;
+			return true;

mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 2040

Quoted 10 lines

mm/rmap.c +10 -0

+		}
+		hugetlb_vma_unlock_write(vma);
+		tlb_finish_mmu(&tlb);
+	}
+	*pteval = huge_ptep_clear_flush(vma, pvmw->address, pvmw->pte);
+	if (pte_dirty(*pteval))
+		folio_mark_dirty(folio);
+
+	*exit_walk = false;
+	return true;

Can we instead introduce some enum that tells the caller how to proceed?

I assume we have

(a) Abort walk (ret = false + page_vma_mapped_walk_done())

(b) Walk done (ret = true + page_vma_mapped_walk_done())

enum ttu_walk_result {
TTU_WALK_CONTINUE,
TTU_WALK_ABORT,
TTU_WALK_DONE
}

Quoted 52 lines

mm/rmap.c +3 ~~-41~~

+}
+
 /*
  * @arg: enum ttu_flags will be passed to this argument
  */

@@ -2115,56 +2177,15 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 				 PageAnonExclusive(subpage);
 
 		if (folio_test_hugetlb(folio)) {
-			bool anon = folio_test_anon(folio);
-
-			/*
-			 * The try_to_unmap() is only passed a hugetlb page
-			 * in the case where the hugetlb page is poisoned.
-			 */
-			VM_BUG_ON_PAGE(!PageHWPoison(subpage), subpage);
-			/*
-			 * huge_pmd_unshare may unmap an entire PMD page.
-			 * There is no way of knowing exactly which PMDs may
-			 * be cached for this mm, so we must flush them all.
-			 * start/end were already adjusted above to cover this
-			 * range.
-			 */
-			flush_cache_range(vma, range.start, range.end);
+			bool exit_walk;
 
-			/*
-			 * To call huge_pmd_unshare, i_mmap_rwsem must be
-			 * held in write mode.  Caller needs to explicitly
-			 * do this outside rmap routines.
-			 *
-			 * We also must hold hugetlb vma_lock in write mode.
-			 * Lock order dictates acquiring vma_lock BEFORE
-			 * i_mmap_rwsem.  We can only try lock here and fail
-			 * if unsuccessful.
-			 */
-			if (!anon) {
-				struct mmu_gather tlb;
-
-				VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
-				if (!hugetlb_vma_trylock_write(vma))
-					goto walk_abort;
-
-				tlb_gather_mmu_vma(&tlb, vma);
-				if (huge_pmd_unshare(&tlb, vma, address, pvmw.pte)) {
-					hugetlb_vma_unlock_write(vma);
-					huge_pmd_unshare_flush(&tlb, vma);
-					tlb_finish_mmu(&tlb);
-					/*
-					 * The PMD table was unmapped,
-					 * consequently unmapping the folio.
-					 */

mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 2187

Quoted 10 lines

mm/rmap.c +6 -4

-					goto walk_done;
-				}
-				hugetlb_vma_unlock_write(vma);
-				tlb_finish_mmu(&tlb);
+			ret = unmap_hugetlb_folio(vma, folio, &pvmw, subpage,
+						  flags, &pteval, &range,
+						  &exit_walk);
+			if (exit_walk) {
+				page_vma_mapped_walk_done(&pvmw);
+				break;

In the old walk_abort case you wouldn't set ret = false?

When returning the enum you could simply do something like

switch (ret) {
case TTU_WALK_ABORT:

	goto walk_abort;
case TTU_WALK_DONE:
	goto walk_done;
default:
	break;
}

While I like this patch, can we please just move all the hugetlb shite into this helper function?

Essentially, get rid of hugetlb special casing in the remainder of the function.

That also makes the function name clearer (right now it's only doing a part of hugetlb folio unmapping).

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-11 08:53 UTC #12 lore raw

Re: [PATCH v3 2/9] mm/rmap: refactor hugetlb pte clearing in try_to_unmap_one

Quoted 19 lines · David Hildenbrand (Arm)

Quoted 18 lines · Dev Jain
Simplify the code by refactoring the folio_test_hugetlb() branch into a new function.
While at it, convert BUG helpers to WARN helpers.
Signed-off-by Dev Jain <[email protected]>
---
 mm/rmap.c | 117 ++++++++++++++++++++++++++++++++----------------------
 1 file changed, 69 insertions(+), 48 deletions(-)
mm/rmap.c +0 -0
index a5f067a09de0f..a98acdea0530a 100644
@@ -1978,6 +1978,68 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,
 				     FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY);
 }
 

Quoted 7 lines

mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 1982
Quoted 2 lines
mm/rmap.c +2 -0
+/* Returns false if unmap needs to be aborted */
+static inline bool unmap_hugetlb_folio(struct vm_area_struct *vma,
I'm wondering whether we should make it clearer that this belongs to the try_to_unmap family by calling it
ttu_hugetlb_folio

Yes I had suggested a ttu_ prefix somewhere else in the first version, Lorenzo didn't like it (or probably he didn't like that specific use of ttu):

https://lore.kernel.org/all/[email protected]/

Don't know about a better name other than "commit_ttu_lazyfree_folio" in that case, but for the hugetlb case, I like ttu_hugetlb_folio.

Quoted 13 lines

mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 1991
Quoted 10 lines
mm/rmap.c +9 -0
+		struct folio *folio, struct page_vma_mapped_walk *pvmw,
+		struct page *page, enum ttu_flags flags, pte_t *pteval,
+		struct mmu_notifier_range *range, bool *exit_walk)
+{
+	/*
+	 * The try_to_unmap() is only passed a hugetlb page
+	 * in the case where the hugetlb page is poisoned.
+	 */
+	VM_WARN_ON_PAGE(!PageHWPoison(page), page);
IIRC, we will never actually get a tail page here.

Quoted 6 lines

Can we avoid passing a page by checking instead whether the hugetlb folios is marked as having a poisoned page?
See the folio_test_set_hwpoison() in hugetlb_update_hwpoison().
So you can simply use folio_test_hwpoison here instead.

Okay I will confirm and do this.

Quoted 61 lines

Quoted 41 lines

mm/rmap.c +39 -0

+	/*
+	 * huge_pmd_unshare may unmap an entire PMD page.
+	 * There is no way of knowing exactly which PMDs may
+	 * be cached for this mm, so we must flush them all.
+	 * start/end were already adjusted above to cover this
+	 * range.
+	 */
+	flush_cache_range(vma, range->start, range->end);
+
+	/*
+	 * To call huge_pmd_unshare, i_mmap_rwsem must be
+	 * held in write mode.  Caller needs to explicitly
+	 * do this outside rmap routines.
+	 *
+	 * We also must hold hugetlb vma_lock in write mode.
+	 * Lock order dictates acquiring vma_lock BEFORE
+	 * i_mmap_rwsem.  We can only try lock here and fail
+	 * if unsuccessful.
+	 */
+	if (!folio_test_anon(folio)) {
+		struct mmu_gather tlb;
+
+		VM_WARN_ON(!(flags & TTU_RMAP_LOCKED));
+		if (!hugetlb_vma_trylock_write(vma)) {
+			*exit_walk = true;
+			return false;
+		}
+
+		tlb_gather_mmu_vma(&tlb, vma);
+		if (huge_pmd_unshare(&tlb, vma, pvmw->address, pvmw->pte)) {
+			hugetlb_vma_unlock_write(vma);
+			huge_pmd_unshare_flush(&tlb, vma);
+			tlb_finish_mmu(&tlb);
+			/*
+			 * The PMD table was unmapped,
+			 * consequently unmapping the folio.
+			 */
+			*exit_walk = true;
+			return true;

mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 2040

Quoted 10 lines

mm/rmap.c +10 -0

+		}
+		hugetlb_vma_unlock_write(vma);
+		tlb_finish_mmu(&tlb);
+	}
+	*pteval = huge_ptep_clear_flush(vma, pvmw->address, pvmw->pte);
+	if (pte_dirty(*pteval))
+		folio_mark_dirty(folio);
+
+	*exit_walk = false;
+	return true;

Can we instead introduce some enum that tells the caller how to proceed?

I assume we have

(a) Abort walk (ret = false + page_vma_mapped_walk_done())

(b) Walk done (ret = true + page_vma_mapped_walk_done())

Quoted 7 lines

(c) Continue walk (call page_vma_mapped_walk())
enum ttu_walk_result {
TTU_WALK_CONTINUE,
TTU_WALK_ABORT,
TTU_WALK_DONE
}

I had replied to such a suggestion here:

https://lore.kernel.org/all/[email protected]/

Probably we don't have any other solution : )

Quoted 57 lines

mm/rmap.c +3 ~~-45~~

+}
+
 /*
  * @arg: enum ttu_flags will be passed to this argument
  */

@@ -2115,56 +2177,15 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 				 PageAnonExclusive(subpage);
 
 		if (folio_test_hugetlb(folio)) {
-			bool anon = folio_test_anon(folio);
-
-			/*
-			 * The try_to_unmap() is only passed a hugetlb page
-			 * in the case where the hugetlb page is poisoned.
-			 */
-			VM_BUG_ON_PAGE(!PageHWPoison(subpage), subpage);
-			/*
-			 * huge_pmd_unshare may unmap an entire PMD page.
-			 * There is no way of knowing exactly which PMDs may
-			 * be cached for this mm, so we must flush them all.
-			 * start/end were already adjusted above to cover this
-			 * range.
-			 */
-			flush_cache_range(vma, range.start, range.end);
+			bool exit_walk;
 
-			/*
-			 * To call huge_pmd_unshare, i_mmap_rwsem must be
-			 * held in write mode.  Caller needs to explicitly
-			 * do this outside rmap routines.
-			 *
-			 * We also must hold hugetlb vma_lock in write mode.
-			 * Lock order dictates acquiring vma_lock BEFORE
-			 * i_mmap_rwsem.  We can only try lock here and fail
-			 * if unsuccessful.
-			 */
-			if (!anon) {
-				struct mmu_gather tlb;
-
-				VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
-				if (!hugetlb_vma_trylock_write(vma))
-					goto walk_abort;
-
-				tlb_gather_mmu_vma(&tlb, vma);
-				if (huge_pmd_unshare(&tlb, vma, address, pvmw.pte)) {
-					hugetlb_vma_unlock_write(vma);
-					huge_pmd_unshare_flush(&tlb, vma);
-					tlb_finish_mmu(&tlb);
-					/*
-					 * The PMD table was unmapped,
-					 * consequently unmapping the folio.
-					 */
-					goto walk_done;
-				}
-				hugetlb_vma_unlock_write(vma);
-				tlb_finish_mmu(&tlb);

Quoted 8 lines

mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 2187
Quoted 6 lines
mm/rmap.c +6 -0
+			ret = unmap_hugetlb_folio(vma, folio, &pvmw, subpage,
+						  flags, &pteval, &range,
+						  &exit_walk);
+			if (exit_walk) {
+				page_vma_mapped_walk_done(&pvmw);
+				break;
In the old walk_abort case you wouldn't set ret = false?

ret will be set appropriately in unmap_hugetlb_folio.

Quoted 13 lines

When returning the enum you could simply do something like
switch (ret) {
case TTU_WALK_ABORT:
	goto walk_abort;
case TTU_WALK_DONE:
	goto walk_done;
default:
	break;
}

Quoted 7 lines

While I like this patch, can we please just move all the hugetlb shite into this helper function?
Essentially, get rid of hugetlb special casing in the remainder of the function.
That also makes the function name clearer (right now it's only doing a part of hugetlb folio unmapping).

Okay I can try that. That would mean splitting the pvmw walk for hugetlb and non-hugetlb, but I suspect it would be very less code duplication.

Quoted 1 line

David Hildenbrand (Arm)

maintainer

2026-05-11 08:59 UTC #13 lore raw

Re: [PATCH v3 2/9] mm/rmap: refactor hugetlb pte clearing in try_to_unmap_one

Quoted 30 lines · Dev Jain

Quoted 19 lines · David Hildenbrand (Arm)
Quoted 18 lines · Dev Jain
Simplify the code by refactoring the folio_test_hugetlb() branch into a new function.
While at it, convert BUG helpers to WARN helpers.
Signed-off-by Dev Jain <[email protected]>
---
 mm/rmap.c | 117 ++++++++++++++++++++++++++++++++----------------------
 1 file changed, 69 insertions(+), 48 deletions(-)
mm/rmap.c +0 -0
index a5f067a09de0f..a98acdea0530a 100644
@@ -1978,6 +1978,68 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,
 				     FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY);
 }
 
Quoted 7 lines
mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 1982
Quoted 2 lines
mm/rmap.c +2 -0
+/* Returns false if unmap needs to be aborted */
+static inline bool unmap_hugetlb_folio(struct vm_area_struct *vma,
I'm wondering whether we should make it clearer that this belongs to the try_to_unmap family by calling it
ttu_hugetlb_folio

Quoted 8 lines

Yes I had suggested a ttu_ prefix somewhere else in the first version, Lorenzo didn't like it (or probably he didn't like that specific use of ttu):
https://lore.kernel.org/all/[email protected]/
Don't know about a better name other than "commit_ttu_lazyfree_folio" in that case, but for the hugetlb case, I like ttu_hugetlb_folio.

Yes, in particular, once we just process the whole hugetlb oddity in there.

I don't really care about the exact name as long as it's clear that this is not something fairly generic.

[...]

Quoted 69 lines

Quoted 61 lines

Quoted 41 lines

mm/rmap.c +39 -0

+	/*
+	 * huge_pmd_unshare may unmap an entire PMD page.
+	 * There is no way of knowing exactly which PMDs may
+	 * be cached for this mm, so we must flush them all.
+	 * start/end were already adjusted above to cover this
+	 * range.
+	 */
+	flush_cache_range(vma, range->start, range->end);
+
+	/*
+	 * To call huge_pmd_unshare, i_mmap_rwsem must be
+	 * held in write mode.  Caller needs to explicitly
+	 * do this outside rmap routines.
+	 *
+	 * We also must hold hugetlb vma_lock in write mode.
+	 * Lock order dictates acquiring vma_lock BEFORE
+	 * i_mmap_rwsem.  We can only try lock here and fail
+	 * if unsuccessful.
+	 */
+	if (!folio_test_anon(folio)) {
+		struct mmu_gather tlb;
+
+		VM_WARN_ON(!(flags & TTU_RMAP_LOCKED));
+		if (!hugetlb_vma_trylock_write(vma)) {
+			*exit_walk = true;
+			return false;
+		}
+
+		tlb_gather_mmu_vma(&tlb, vma);
+		if (huge_pmd_unshare(&tlb, vma, pvmw->address, pvmw->pte)) {
+			hugetlb_vma_unlock_write(vma);
+			huge_pmd_unshare_flush(&tlb, vma);
+			tlb_finish_mmu(&tlb);
+			/*
+			 * The PMD table was unmapped,
+			 * consequently unmapping the folio.
+			 */
+			*exit_walk = true;
+			return true;

mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 2040

Quoted 10 lines

mm/rmap.c +10 -0

+		}
+		hugetlb_vma_unlock_write(vma);
+		tlb_finish_mmu(&tlb);
+	}
+	*pteval = huge_ptep_clear_flush(vma, pvmw->address, pvmw->pte);
+	if (pte_dirty(*pteval))
+		folio_mark_dirty(folio);
+
+	*exit_walk = false;
+	return true;

Can we instead introduce some enum that tells the caller how to proceed?

I assume we have

(a) Abort walk (ret = false + page_vma_mapped_walk_done())

(b) Walk done (ret = true + page_vma_mapped_walk_done())

Quoted 7 lines

(c) Continue walk (call page_vma_mapped_walk())
enum ttu_walk_result {
TTU_WALK_CONTINUE,
TTU_WALK_ABORT,
TTU_WALK_DONE
}

Quoted 5 lines

I had replied to such a suggestion here:
https://lore.kernel.org/all/[email protected]/
Probably we don't have any other solution : )

That looks like the right way to. The boolean return is just nasty.

Quoted 14 lines

Quoted 5 lines
Quoted 5 lines
mm/rmap.c +0 -5
-					 */
-					goto walk_done;
-				}
-				hugetlb_vma_unlock_write(vma);
-				tlb_finish_mmu(&tlb);
Quoted 8 lines
mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 2187
Quoted 6 lines
mm/rmap.c +6 -0
+			ret = unmap_hugetlb_folio(vma, folio, &pvmw, subpage,
+						  flags, &pteval, &range,
+						  &exit_walk);
+			if (exit_walk) {
+				page_vma_mapped_walk_done(&pvmw);
+				break;
In the old walk_abort case you wouldn't set ret = false?

Quoted 1 line

ret will be set appropriately in unmap_hugetlb_folio.

Ah, right. Confusing ;)

Quoted 16 lines

Quoted 16 lines
When returning the enum you could simply do something like
switch (ret) {
case TTU_WALK_ABORT:
	goto walk_abort;
case TTU_WALK_DONE:
	goto walk_done;
default:
	break;
}
While I like this patch, can we please just move all the hugetlb shite into this helper function?

Quoted 7 lines

Quoted 4 lines
Essentially, get rid of hugetlb special casing in the remainder of the function.
That also makes the function name clearer (right now it's only doing a part of hugetlb folio unmapping).
Okay I can try that. That would mean splitting the pvmw walk for hugetlb and non-hugetlb, but I suspect it would be very less code duplication.

Right, and it would also be clearer that hugetlb really only is called for hwpoison handling.

Signature · 3 lines

Cheers,

David

Barry Song

reviewer

2026-05-11 22:20 UTC #14 lore raw

Re: [PATCH v3 2/9] mm/rmap: refactor hugetlb pte clearing in try_to_unmap_one

In reply to David Hildenbrand (Arm)

On Mon, May 11, 2026 at 3:10 PM David Hildenbrand (Arm) <[email protected]> wrote:
[...]

Quoted 24 lines

mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 2040
Quoted 6 lines
mm/rmap.c +6 -0
+     *pteval = huge_ptep_clear_flush(vma, pvmw->address, pvmw->pte);
+     if (pte_dirty(*pteval))
+             folio_mark_dirty(folio);
+
+     *exit_walk = false;
+     return true;
Can we instead introduce some enum that tells the caller how to proceed?
I assume we have
(a) Abort walk (ret = false + page_vma_mapped_walk_done())
(b) Walk done (ret = true + page_vma_mapped_walk_done())
(c) Continue walk (call page_vma_mapped_walk())
enum ttu_walk_result {
TTU_WALK_CONTINUE,
TTU_WALK_ABORT,
TTU_WALK_DONE
}

[...]

Quoted 5 lines

In the old walk_abort case you wouldn't set ret = false?
When returning the enum you could simply do something like

Quoted 9 lines

switch (ret) {
case TTU_WALK_ABORT:

        goto walk_abort;
case TTU_WALK_DONE:
        goto walk_done;
default:
        break;
}

I believe I suggested the exact same approach in v2[1], but Dev didn’t agree :-)

[1] https://lore.kernel.org/linux-mm/CAGsJ_4zK1W71iSP14gi=8yWSg80EjgYiBXqLChB2i+X+RSTWcw@mail.gmail.com/

Dev Jain

author

2026-05-12 05:17 UTC #15 lore raw

Re: [PATCH v3 2/9] mm/rmap: refactor hugetlb pte clearing in try_to_unmap_one

Quoted 43 lines · Barry Song

On Mon, May 11, 2026 at 3:10 PM David Hildenbrand (Arm) <[email protected]> wrote:
[...]
Quoted 24 lines
mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 2040
Quoted 6 lines
mm/rmap.c +6 -0
+     *pteval = huge_ptep_clear_flush(vma, pvmw->address, pvmw->pte);
+     if (pte_dirty(*pteval))
+             folio_mark_dirty(folio);
+
+     *exit_walk = false;
+     return true;
Can we instead introduce some enum that tells the caller how to proceed?
I assume we have
(a) Abort walk (ret = false + page_vma_mapped_walk_done())
(b) Walk done (ret = true + page_vma_mapped_walk_done())
(c) Continue walk (call page_vma_mapped_walk())
enum ttu_walk_result {
TTU_WALK_CONTINUE,
TTU_WALK_ABORT,
TTU_WALK_DONE
}
[...]
Quoted 5 lines
In the old walk_abort case you wouldn't set ret = false?
When returning the enum you could simply do something like
Quoted 9 lines
switch (ret) {
case TTU_WALK_ABORT:
        goto walk_abort;
case TTU_WALK_DONE:
        goto walk_done;
default:
        break;
}

Quoted 3 lines

I believe I suggested the exact same approach in v2[1], but Dev didn’t agree :-)
[1] https://lore.kernel.org/linux-mm/CAGsJ_4zK1W71iSP14gi=8yWSg80EjgYiBXqLChB2i+X+RSTWcw@mail.gmail.com/

I did mention your suggestion in my reply to David haha :)

I'll see if I can do even more refactoring with the tristate - it doesn't feel nice to do the tristate at one place and then leave all others as it is.

Dev Jain

author

2026-05-06 09:45 UTC #16 lore raw

PATCH v3 3/9 mm/rmap: refactor some code around lazyfree folio unmapping

In reply to Dev Jain

For lazyfree folio unmapping, after clearing the ptes we must abort the operation if the folio got dirtied or it has unexpected references.

Refactor this logic into a function which will return whether we need to abort or not.

If we abort, we restore the ptes and bail out of try_to_unmap_one. Otherwise adjust the rss stats of the mm and jump to a label.

Also rename that label from "discard" to "finish_unmap"; the former is appropriate in the lazyfree context, but the code following the label is executed for other successful unmap code paths too, so 'discard' does not sound correct for them.

Signed-off-by Dev Jain <[email protected]>

---
 mm/rmap.c | 95 ++++++++++++++++++++++++++++++++-----------------------
 1 file changed, 55 insertions(+), 40 deletions(-)

mm/rmap.c +55 ~~-40~~

index a98acdea0530a..bd4e3639e26ed 100644
@@ -1978,6 +1978,56 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,
 				     FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY);
 }
 
+static inline bool can_unmap_lazyfree_folio_range(struct vm_area_struct *vma,
+		struct folio *folio, unsigned long address, pte_t *ptep,
+		pte_t pteval, unsigned long nr_pages)
+{
+	struct mm_struct *mm = vma->vm_mm;
+	int ref_count, map_count;
+
+	/*
+	 * Synchronize with gup_pte_range():
+	 * - clear PTE; barrier; read refcount
+	 * - inc refcount; barrier; read PTE
+	 */
+	smp_mb();
+
+	ref_count = folio_ref_count(folio);
+	map_count = folio_mapcount(folio);
+
+	/*
+	 * Order reads for page refcount and dirty flag
+	 * (see comments in __remove_mapping()).
+	 */
+	smp_rmb();
+
+	if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) {
+		/*
+		 * redirtied either using the page table or a previously
+		 * obtained GUP reference.
+		 */
+		set_ptes(mm, address, ptep, pteval, nr_pages);
+		folio_set_swapbacked(folio);
+		return false;
+	}
+
+	if (ref_count != 1 + map_count) {
+		/*
+		 * Additional reference. Could be a GUP reference or any
+		 * speculative reference. GUP users must mark the folio
+		 * dirty if there was a modification. This folio cannot be
+		 * reclaimed right now either way, so act just like nothing
+		 * happened.
+		 * We'll come back here later and detect if the folio was
+		 * dirtied when the additional reference is gone.
+		 */
+		set_ptes(mm, address, ptep, pteval, nr_pages);
+		return false;
+	}
+
+	return true;
+}
+
 /* Returns false if unmap needs to be aborted */
 static inline bool unmap_hugetlb_folio(struct vm_area_struct *vma,
 		struct folio *folio, struct page_vma_mapped_walk *pvmw,

@@ -2259,47 +2309,12 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 
 			/* MADV_FREE page check */
 			if (!folio_test_swapbacked(folio)) {
-				int ref_count, map_count;
-
-				/*
-				 * Synchronize with gup_pte_range():
-				 * - clear PTE; barrier; read refcount
-				 * - inc refcount; barrier; read PTE
-				 */
-				smp_mb();
-
-				ref_count = folio_ref_count(folio);
-				map_count = folio_mapcount(folio);
-
-				/*
-				 * Order reads for page refcount and dirty flag
-				 * (see comments in __remove_mapping()).
-				 */
-				smp_rmb();
-
-				if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) {
-					/*
-					 * redirtied either using the page table or a previously
-					 * obtained GUP reference.
-					 */
-					set_ptes(mm, address, pvmw.pte, pteval, nr_pages);
-					folio_set_swapbacked(folio);
+				if (!can_unmap_lazyfree_folio_range(vma, folio, address,
+				    pvmw.pte, pteval, nr_pages))
 					goto walk_abort;
-				} else if (ref_count != 1 + map_count) {
-					/*
-					 * Additional reference. Could be a GUP reference or any
-					 * speculative reference. GUP users must mark the folio
-					 * dirty if there was a modification. This folio cannot be
-					 * reclaimed right now either way, so act just like nothing
-					 * happened.
-					 * We'll come back here later and detect if the folio was
-					 * dirtied when the additional reference is gone.
-					 */
-					set_ptes(mm, address, pvmw.pte, pteval, nr_pages);
-					goto walk_abort;
-				}
+
 				add_mm_counter(mm, MM_ANONPAGES, -nr_pages);
-				goto discard;
+				goto finish_unmap;
 			}
 
 			if (folio_dup_swap(folio, subpage) < 0) {

@@ -2362,7 +2377,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 			 */
 			add_mm_counter(mm, mm_counter_file(folio), -nr_pages);
 		}
-discard:
+finish_unmap:
 		if (unlikely(folio_test_hugetlb(folio))) {
 			hugetlb_remove_rmap(folio);
 		} else {

Signature · 1 line

2.34.1

David Hildenbrand (Arm)

maintainer

2026-05-11 07:28 UTC #17 lore raw

Re: [PATCH v3 3/9] mm/rmap: refactor some code around lazyfree folio unmapping

Quoted 24 lines · Dev Jain

For lazyfree folio unmapping, after clearing the ptes we must abort the operation if the folio got dirtied or it has unexpected references.
Refactor this logic into a function which will return whether we need to abort or not.
If we abort, we restore the ptes and bail out of try_to_unmap_one. Otherwise adjust the rss stats of the mm and jump to a label.
Also rename that label from "discard" to "finish_unmap"; the former is appropriate in the lazyfree context, but the code following the label is executed for other successful unmap code paths too, so 'discard' does not sound correct for them.
Signed-off-by Dev Jain <[email protected]>
---
 mm/rmap.c | 95 ++++++++++++++++++++++++++++++++-----------------------
 1 file changed, 55 insertions(+), 40 deletions(-)
mm/rmap.c +0 -0
index a98acdea0530a..bd4e3639e26ed 100644

mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 1983

Quoted 6 lines

mm/rmap.c +3 -0

@@ -1978,6 +1978,56 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,
 				     FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY);
 }
 
+static inline bool can_unmap_lazyfree_folio_range(struct vm_area_struct *vma,
+		struct folio *folio, unsigned long address, pte_t *ptep,
+		pte_t pteval, unsigned long nr_pages)

Similar comment: ttu_...*

Quoted 35 lines

mm/rmap.c +35 -0

+{
+	struct mm_struct *mm = vma->vm_mm;
+	int ref_count, map_count;
+
+	/*
+	 * Synchronize with gup_pte_range():
+	 * - clear PTE; barrier; read refcount
+	 * - inc refcount; barrier; read PTE
+	 */
+	smp_mb();
+
+	ref_count = folio_ref_count(folio);
+	map_count = folio_mapcount(folio);
+
+	/*
+	 * Order reads for page refcount and dirty flag
+	 * (see comments in __remove_mapping()).
+	 */
+	smp_rmb();
+
+	if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) {
+		/*
+		 * redirtied either using the page table or a previously
+		 * obtained GUP reference.
+		 */
+		set_ptes(mm, address, ptep, pteval, nr_pages);
+		folio_set_swapbacked(folio);
+		return false;
+	}
+
+	if (ref_count != 1 + map_count) {
+		/*
+		 * Additional reference. Could be a GUP reference or any
+		 * speculative reference. GUP users must mark the folio
+		 * dirty if there was a modification. This folio cannot be

mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 2028

Quoted 10 lines

mm/rmap.c +10 -0

+		 * reclaimed right now either way, so act just like nothing
+		 * happened.
+		 * We'll come back here later and detect if the folio was
+		 * dirtied when the additional reference is gone.
+		 */
+		set_ptes(mm, address, ptep, pteval, nr_pages);
+		return false;
+	}
+
+	return true;

Doing the set_ptes() in a function called "can_unmap_lazyfree_folio_range" is not appropriate.

Can we just leave that in the caller? We only do the when we return false.

And hey, then you can call this function ttu_can_unmap_lazyfree_folio() and avoid passing pte ranges. :)

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-12 05:19 UTC #18 lore raw

Re: [PATCH v3 3/9] mm/rmap: refactor some code around lazyfree folio unmapping

Quoted 28 lines · David Hildenbrand (Arm)

Quoted 27 lines · Dev Jain
For lazyfree folio unmapping, after clearing the ptes we must abort the operation if the folio got dirtied or it has unexpected references.
Refactor this logic into a function which will return whether we need to abort or not.
If we abort, we restore the ptes and bail out of try_to_unmap_one. Otherwise adjust the rss stats of the mm and jump to a label.
Also rename that label from "discard" to "finish_unmap"; the former is appropriate in the lazyfree context, but the code following the label is executed for other successful unmap code paths too, so 'discard' does not sound correct for them.
Signed-off-by Dev Jain <[email protected]>
---
 mm/rmap.c | 95 ++++++++++++++++++++++++++++++++-----------------------
 1 file changed, 55 insertions(+), 40 deletions(-)
mm/rmap.c +0 -0
index a98acdea0530a..bd4e3639e26ed 100644
@@ -1978,6 +1978,56 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,
 				     FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY);
 }
 

Quoted 6 lines

mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 1983
Quoted 3 lines
mm/rmap.c +3 -0
+static inline bool can_unmap_lazyfree_folio_range(struct vm_area_struct *vma,
+		struct folio *folio, unsigned long address, pte_t *ptep,
+		pte_t pteval, unsigned long nr_pages)
Similar comment: ttu_...*

Ack

Quoted 51 lines

Quoted 36 lines

mm/rmap.c +35 -0

+{
+	struct mm_struct *mm = vma->vm_mm;
+	int ref_count, map_count;
+
+	/*
+	 * Synchronize with gup_pte_range():
+	 * - clear PTE; barrier; read refcount
+	 * - inc refcount; barrier; read PTE
+	 */
+	smp_mb();
+
+	ref_count = folio_ref_count(folio);
+	map_count = folio_mapcount(folio);
+
+	/*
+	 * Order reads for page refcount and dirty flag
+	 * (see comments in __remove_mapping()).
+	 */
+	smp_rmb();
+
+	if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) {
+		/*
+		 * redirtied either using the page table or a previously
+		 * obtained GUP reference.
+		 */
+		set_ptes(mm, address, ptep, pteval, nr_pages);
+		folio_set_swapbacked(folio);
+		return false;
+	}
+
+	if (ref_count != 1 + map_count) {
+		/*
+		 * Additional reference. Could be a GUP reference or any
+		 * speculative reference. GUP users must mark the folio
+		 * dirty if there was a modification. This folio cannot be

mm/rmap.c static inline unsigned int folio_unmap_pte_batch(struct folio *folio, · line 2028

Quoted 10 lines

mm/rmap.c +10 -0

+		 * reclaimed right now either way, so act just like nothing
+		 * happened.
+		 * We'll come back here later and detect if the folio was
+		 * dirtied when the additional reference is gone.
+		 */
+		set_ptes(mm, address, ptep, pteval, nr_pages);
+		return false;
+	}
+
+	return true;

Doing the set_ptes() in a function called "can_unmap_lazyfree_folio_range" is not appropriate.

Quoted 4 lines

Can we just leave that in the caller? We only do the when we return false.
And hey, then you can call this function ttu_can_unmap_lazyfree_folio() and avoid passing pte ranges. :)

Yep great I'll do that.

Quoted 2 lines

Dev Jain

author

2026-05-06 09:45 UTC #19 lore raw

PATCH v3 4/9 mm/memory: Batch set uffd-wp markers during zapping

In reply to Dev Jain

In preparation for the next patch, enable batch setting of uffd-wp ptes.

The code paths passing nr > 1 to zap_install_uffd_wp_if_needed() produce that nr through either folio_pte_batch or swap_pte_batch, guaranteeing that all ptes are the same w.r.t belonging to the same type of VMA (anonymous or non-anonymous, wp-armed or non-wp-armed), and all being marked with uffd-wp or all being not marked.

Note that we will have to use set_pte_at() in a loop instead of set_ptes() since the latter cannot handle present->non-present conversion for nr_pages > 1.

Convert documentation of install_uffd_wp_ptes_if_needed to kerneldoc format.

No functional change is intended.

Signed-off-by Dev Jain <[email protected]>

---
 include/linux/mm_inline.h | 34 +++++++++++++++++++++-------------
 mm/memory.c               | 20 +-------------------
 mm/rmap.c                 |  2 +-
 3 files changed, 23 insertions(+), 33 deletions(-)

include/linux/mm_inline.h +21 ~~-13~~

index a171070e15f05..6f7ecede2fb45 100644
@@ -566,9 +566,17 @@ static inline pte_marker copy_pte_marker(
 	return dstm;
 }
 
-/*
- * If this pte is wr-protected by uffd-wp in any form, arm the special pte to
- * replace a none pte.  NOTE!  This should only be called when *pte is already
+/**
+ * install_uffd_wp_ptes_if_needed - install uffd-wp marker on PTEs that map
+ *				    consecutive pages of the same large folio.
+ * @vma: The VMA the pages are mapped into.
+ * @addr: Address the first page of this batch is mapped at.
+ * @ptep: Page table pointer for the first entry of this batch.
+ * @pteval: old value of the entry pointed to by ptep.
+ * @nr_ptes: Number of entries to clear (batch size).
+ *
+ * If the ptes were wr-protected by uffd-wp in any form, arm special ptes to
+ * replace none ptes.  NOTE!  This should only be called when *pte is already
  * cleared so we will never accidentally replace something valuable.  Meanwhile
  * none pte also means we are not demoting the pte so tlb flushed is not needed.
  * E.g., when pte cleared the caller should have taken care of the tlb flush.

@@ -576,11 +584,11 @@ static inline pte_marker copy_pte_marker(
  * Must be called with pgtable lock held so that no thread will see the none
  * pte, and if they see it, they'll fault and serialize at the pgtable lock.
  *
- * Returns true if an uffd-wp pte was installed, false otherwise.
+ * Returns true if uffd-wp ptes were installed, false otherwise.
  */
 static inline bool
-pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
-			      pte_t *pte, pte_t pteval)
+install_uffd_wp_ptes_if_needed(struct vm_area_struct *vma, unsigned long addr,
+			      pte_t *ptep, pte_t pteval, unsigned long nr_ptes)
 {
 	bool arm_uffd_pte = false;

@@ -588,7 +596,7 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
 		return false;
 
 	/* The current status of the pte should be "cleared" before calling */
-	WARN_ON_ONCE(!pte_none(ptep_get(pte)));
+	WARN_ON_ONCE(!pte_none(ptep_get(ptep)));
 
 	/*
 	 * NOTE: userfaultfd_wp_unpopulated() doesn't need this whole

@@ -610,13 +618,13 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
 	if (unlikely(pte_swp_uffd_wp_any(pteval)))
 		arm_uffd_pte = true;
 
-	if (unlikely(arm_uffd_pte)) {
-		set_pte_at(vma->vm_mm, addr, pte,
-			   make_pte_marker(PTE_MARKER_UFFD_WP));
-		return true;
-	}
+	if (likely(!arm_uffd_pte))
+		return false;
 
-	return false;
+	for (int i = 0; i < nr_ptes; ++i, ++ptep, addr += PAGE_SIZE)
+		set_pte_at(vma->vm_mm, addr, ptep, make_pte_marker(PTE_MARKER_UFFD_WP));
+
+	return true;
 }
 
 static inline bool vma_has_recency(const struct vm_area_struct *vma)

mm/memory.c +1 ~~-19~~

index 0c9d9c2cbf0e0..f14311c4d2001 100644
@@ -1610,29 +1610,11 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma,
 			      unsigned long addr, pte_t *pte, int nr,
 			      struct zap_details *details, pte_t pteval)
 {
-	bool was_installed = false;
-
-	if (!uffd_supports_wp_marker())
-		return false;
-
-	/* Zap on anonymous always means dropping everything */
-	if (vma_is_anonymous(vma))
-		return false;
-
 	if (zap_drop_markers(details))
 		return false;
 
-	for (;;) {
-		/* the PFN in the PTE is irrelevant. */
-		if (pte_install_uffd_wp_if_needed(vma, addr, pte, pteval))
-			was_installed = true;
-		if (--nr == 0)
-			break;
-		pte++;
-		addr += PAGE_SIZE;
-	}
+	return install_uffd_wp_ptes_if_needed(vma, addr, pte, pteval, nr);
 
-	return was_installed;
 }
 
 static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb,

mm/rmap.c +1 -1

index bd4e3639e26ed..b17dce752a1ea 100644
@@ -2266,7 +2266,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		 * we may want to replace a none pte with a marker pte if
 		 * it's file-backed, so we don't lose the tracking info.
 		 */
-		pte_install_uffd_wp_if_needed(vma, address, pvmw.pte, pteval);
+		install_uffd_wp_ptes_if_needed(vma, address, pvmw.pte, pteval, 1);
 
 		/* Update high watermark before we lower rss */
 		update_hiwater_rss(mm);

Signature · 1 line

2.34.1

David Hildenbrand (Arm)

maintainer

2026-05-11 07:37 UTC #20 lore raw

Re: [PATCH v3 4/9] mm/memory: Batch set uffd-wp markers during zapping

Quoted 8 lines · Dev Jain

In preparation for the next patch, enable batch setting of uffd-wp ptes.
The code paths passing nr > 1 to zap_install_uffd_wp_if_needed() produce that nr through either folio_pte_batch or swap_pte_batch, guaranteeing that all ptes are the same w.r.t belonging to the same type of VMA (anonymous or non-anonymous, wp-armed or non-wp-armed), and all being marked with uffd-wp or all being not marked.

Quoted 6 lines

Note that we will have to use set_pte_at() in a loop instead of set_ptes() since the latter cannot handle present->non-present conversion for nr_pages > 1.
Convert documentation of install_uffd_wp_ptes_if_needed to kerneldoc format.

You should mention why the uffd_supports_wp_marker()+vma_is_anonymous() calls in zap_install_uffd_wp_if_needed can be dropped.

Quoted 37 lines

No functional change is intended.

Signed-off-by Dev Jain <[email protected]>

---
 include/linux/mm_inline.h | 34 +++++++++++++++++++++-------------
 mm/memory.c               | 20 +-------------------
 mm/rmap.c                 |  2 +-
 3 files changed, 23 insertions(+), 33 deletions(-)

include/linux/mm_inline.h +11 -3

index a171070e15f05..6f7ecede2fb45 100644
@@ -566,9 +566,17 @@ static inline pte_marker copy_pte_marker(
 	return dstm;
 }
 
-/*
- * If this pte is wr-protected by uffd-wp in any form, arm the special pte to
- * replace a none pte.  NOTE!  This should only be called when *pte is already
+/**
+ * install_uffd_wp_ptes_if_needed - install uffd-wp marker on PTEs that map
+ *				    consecutive pages of the same large folio.
+ * @vma: The VMA the pages are mapped into.
+ * @addr: Address the first page of this batch is mapped at.
+ * @ptep: Page table pointer for the first entry of this batch.
+ * @pteval: old value of the entry pointed to by ptep.
+ * @nr_ptes: Number of entries to clear (batch size).
+ *
+ * If the ptes were wr-protected by uffd-wp in any form, arm special ptes to
+ * replace none ptes.  NOTE!  This should only be called when *pte is already
  * cleared so we will never accidentally replace something valuable.  Meanwhile
  * none pte also means we are not demoting the pte so tlb flushed is not needed.
  * E.g., when pte cleared the caller should have taken care of the tlb flush.

@@ -576,11 +584,11 @@ static inline pte_marker copy_pte_marker(
  * Must be called with pgtable lock held so that no thread will see the none

include/linux/mm_inline.h static inline pte_marker copy_pte_marker( · line 591

Quoted 10 lines

include/linux/mm_inline.h +3 -3

  * pte, and if they see it, they'll fault and serialize at the pgtable lock.
  *
- * Returns true if an uffd-wp pte was installed, false otherwise.
+ * Returns true if uffd-wp ptes were installed, false otherwise.
  */
 static inline bool
-pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
-			      pte_t *pte, pte_t pteval)
+install_uffd_wp_ptes_if_needed(struct vm_area_struct *vma, unsigned long addr,
+			      pte_t *ptep, pte_t pteval, unsigned long nr_ptes)

If we conditionally do something, what about while at it shorten it to:

cont_install_uffd_wp_ptes()

Also, less churn in this patch if you don't change pte->ptep

(because if you do, you should then also do pteval->pte :) )

So I'd just leave that as is in this patch.

Quoted 20 lines

include/linux/mm_inline.h +1 -5

 {
 	bool arm_uffd_pte = false;

@@ -588,7 +596,7 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
 		return false;
 
 	/* The current status of the pte should be "cleared" before calling */
-	WARN_ON_ONCE(!pte_none(ptep_get(pte)));
+	WARN_ON_ONCE(!pte_none(ptep_get(ptep)));
 
 	/*
 	 * NOTE: userfaultfd_wp_unpopulated() doesn't need this whole

@@ -610,13 +618,13 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
 	if (unlikely(pte_swp_uffd_wp_any(pteval)))
 		arm_uffd_pte = true;
 
-	if (unlikely(arm_uffd_pte)) {
-		set_pte_at(vma->vm_mm, addr, pte,
-			   make_pte_marker(PTE_MARKER_UFFD_WP));
-		return true;

include/linux/mm_inline.h pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr, · line 628

Quoted 10 lines

include/linux/mm_inline.h +6 -2

-	}
+	if (likely(!arm_uffd_pte))
+		return false;
 
-	return false;
+	for (int i = 0; i < nr_ptes; ++i, ++ptep, addr += PAGE_SIZE)
+		set_pte_at(vma->vm_mm, addr, ptep, make_pte_marker(PTE_MARKER_UFFD_WP));
+
+	return true;
 }

I wonder whether this growing function is really appropriate to be in the header file? Can we just move that whole thing into mm/memory.c?

It's called on every try_to_unmap_one() invocation, but I doubt we care about a function call here.

Quoted 50 lines

include/linux/mm_inline.h +0 -0

 static inline bool vma_has_recency(const struct vm_area_struct *vma)

mm/memory.c +1 ~~-19~~

index 0c9d9c2cbf0e0..f14311c4d2001 100644
@@ -1610,29 +1610,11 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma,
 			      unsigned long addr, pte_t *pte, int nr,
 			      struct zap_details *details, pte_t pteval)
 {
-	bool was_installed = false;
-
-	if (!uffd_supports_wp_marker())
-		return false;
-
-	/* Zap on anonymous always means dropping everything */
-	if (vma_is_anonymous(vma))
-		return false;
-
 	if (zap_drop_markers(details))
 		return false;
 
-	for (;;) {
-		/* the PFN in the PTE is irrelevant. */
-		if (pte_install_uffd_wp_if_needed(vma, addr, pte, pteval))
-			was_installed = true;
-		if (--nr == 0)
-			break;
-		pte++;
-		addr += PAGE_SIZE;
-	}
+	return install_uffd_wp_ptes_if_needed(vma, addr, pte, pteval, nr);
 
-	return was_installed;
 }
 
 static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb,

mm/rmap.c +1 -1

index bd4e3639e26ed..b17dce752a1ea 100644
@@ -2266,7 +2266,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		 * we may want to replace a none pte with a marker pte if
 		 * it's file-backed, so we don't lose the tracking info.
 		 */
-		pte_install_uffd_wp_if_needed(vma, address, pvmw.pte, pteval);
+		install_uffd_wp_ptes_if_needed(vma, address, pvmw.pte, pteval, 1);
 
 		/* Update high watermark before we lower rss */
 		update_hiwater_rss(mm);

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-12 05:59 UTC #21 lore raw

Re: [PATCH v3 4/9] mm/memory: Batch set uffd-wp markers during zapping

Quoted 13 lines · David Hildenbrand (Arm)

Quoted 12 lines · Dev Jain
In preparation for the next patch, enable batch setting of uffd-wp ptes.
The code paths passing nr > 1 to zap_install_uffd_wp_if_needed() produce that nr through either folio_pte_batch or swap_pte_batch, guaranteeing that all ptes are the same w.r.t belonging to the same type of VMA (anonymous or non-anonymous, wp-armed or non-wp-armed), and all being marked with uffd-wp or all being not marked.
Note that we will have to use set_pte_at() in a loop instead of set_ptes() since the latter cannot handle present->non-present conversion for nr_pages > 1.

Quoted 5 lines

Quoted 2 lines
Convert documentation of install_uffd_wp_ptes_if_needed to kerneldoc format.
You should mention why the uffd_supports_wp_marker()+vma_is_anonymous() calls in zap_install_uffd_wp_if_needed can be dropped.

Okay.

Quoted 51 lines

Quoted 38 lines

No functional change is intended.

Signed-off-by Dev Jain <[email protected]>

---
 include/linux/mm_inline.h | 34 +++++++++++++++++++++-------------
 mm/memory.c               | 20 +-------------------
 mm/rmap.c                 |  2 +-
 3 files changed, 23 insertions(+), 33 deletions(-)

include/linux/mm_inline.h +11 -3

index a171070e15f05..6f7ecede2fb45 100644
@@ -566,9 +566,17 @@ static inline pte_marker copy_pte_marker(
 	return dstm;
 }
 
-/*
- * If this pte is wr-protected by uffd-wp in any form, arm the special pte to
- * replace a none pte.  NOTE!  This should only be called when *pte is already
+/**
+ * install_uffd_wp_ptes_if_needed - install uffd-wp marker on PTEs that map
+ *				    consecutive pages of the same large folio.
+ * @vma: The VMA the pages are mapped into.
+ * @addr: Address the first page of this batch is mapped at.
+ * @ptep: Page table pointer for the first entry of this batch.
+ * @pteval: old value of the entry pointed to by ptep.
+ * @nr_ptes: Number of entries to clear (batch size).
+ *
+ * If the ptes were wr-protected by uffd-wp in any form, arm special ptes to
+ * replace none ptes.  NOTE!  This should only be called when *pte is already
  * cleared so we will never accidentally replace something valuable.  Meanwhile
  * none pte also means we are not demoting the pte so tlb flushed is not needed.
  * E.g., when pte cleared the caller should have taken care of the tlb flush.

@@ -576,11 +584,11 @@ static inline pte_marker copy_pte_marker(
  * Must be called with pgtable lock held so that no thread will see the none

include/linux/mm_inline.h static inline pte_marker copy_pte_marker( · line 591

Quoted 10 lines

include/linux/mm_inline.h +3 -3

  * pte, and if they see it, they'll fault and serialize at the pgtable lock.
  *
- * Returns true if an uffd-wp pte was installed, false otherwise.
+ * Returns true if uffd-wp ptes were installed, false otherwise.
  */
 static inline bool
-pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
-			      pte_t *pte, pte_t pteval)
+install_uffd_wp_ptes_if_needed(struct vm_area_struct *vma, unsigned long addr,
+			      pte_t *ptep, pte_t pteval, unsigned long nr_ptes)

If we conditionally do something, what about while at it shorten it to:

Quoted 1 line

cont_install_uffd_wp_ptes()

Why "cont_"? Did you mean to say "cond" ("conditionally").

Quoted 6 lines

Also, less churn in this patch if you don't change pte->ptep
(because if you do, you should then also do pteval->pte :) )
So I'd just leave that as is in this patch.

So I'll change pte->ptep and pteval->pte because I hate pte pointers called pte : )

Quoted 35 lines

Quoted 21 lines

include/linux/mm_inline.h +1 -5

 {
 	bool arm_uffd_pte = false;

@@ -588,7 +596,7 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
 		return false;
 
 	/* The current status of the pte should be "cleared" before calling */
-	WARN_ON_ONCE(!pte_none(ptep_get(pte)));
+	WARN_ON_ONCE(!pte_none(ptep_get(ptep)));
 
 	/*
 	 * NOTE: userfaultfd_wp_unpopulated() doesn't need this whole

@@ -610,13 +618,13 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
 	if (unlikely(pte_swp_uffd_wp_any(pteval)))
 		arm_uffd_pte = true;
 
-	if (unlikely(arm_uffd_pte)) {
-		set_pte_at(vma->vm_mm, addr, pte,
-			   make_pte_marker(PTE_MARKER_UFFD_WP));
-		return true;

include/linux/mm_inline.h pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr, · line 628

Quoted 10 lines

include/linux/mm_inline.h +6 -2

-	}
+	if (likely(!arm_uffd_pte))
+		return false;
 
-	return false;
+	for (int i = 0; i < nr_ptes; ++i, ++ptep, addr += PAGE_SIZE)
+		set_pte_at(vma->vm_mm, addr, ptep, make_pte_marker(PTE_MARKER_UFFD_WP));
+
+	return true;
 }

I wonder whether this growing function is really appropriate to be in the header file? Can we just move that whole thing into mm/memory.c?

Quoted 2 lines

It's called on every try_to_unmap_one() invocation, but I doubt we care about a function call here.

Makes sense.

Quoted 53 lines

Quoted 51 lines

include/linux/mm_inline.h +0 -0

 static inline bool vma_has_recency(const struct vm_area_struct *vma)

mm/memory.c +1 ~~-19~~

index 0c9d9c2cbf0e0..f14311c4d2001 100644
@@ -1610,29 +1610,11 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma,
 			      unsigned long addr, pte_t *pte, int nr,
 			      struct zap_details *details, pte_t pteval)
 {
-	bool was_installed = false;
-
-	if (!uffd_supports_wp_marker())
-		return false;
-
-	/* Zap on anonymous always means dropping everything */
-	if (vma_is_anonymous(vma))
-		return false;
-
 	if (zap_drop_markers(details))
 		return false;
 
-	for (;;) {
-		/* the PFN in the PTE is irrelevant. */
-		if (pte_install_uffd_wp_if_needed(vma, addr, pte, pteval))
-			was_installed = true;
-		if (--nr == 0)
-			break;
-		pte++;
-		addr += PAGE_SIZE;
-	}
+	return install_uffd_wp_ptes_if_needed(vma, addr, pte, pteval, nr);
 
-	return was_installed;
 }
 
 static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb,

mm/rmap.c +1 -1

index bd4e3639e26ed..b17dce752a1ea 100644
@@ -2266,7 +2266,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		 * we may want to replace a none pte with a marker pte if
 		 * it's file-backed, so we don't lose the tracking info.
 		 */
-		pte_install_uffd_wp_if_needed(vma, address, pvmw.pte, pteval);
+		install_uffd_wp_ptes_if_needed(vma, address, pvmw.pte, pteval, 1);
 
 		/* Update high watermark before we lower rss */
 		update_hiwater_rss(mm);

David Hildenbrand (Arm)

maintainer

2026-05-12 06:04 UTC #22 lore raw

Re: [PATCH v3 4/9] mm/memory: Batch set uffd-wp markers during zapping

Quoted 7 lines

Quoted 5 lines
If we conditionally do something, what about while at it shorten it to:
cont_install_uffd_wp_ptes()
Why "cont_"? Did you mean to say "cond" ("conditionally").

Yes, like cond_resched().

Quoted 10 lines

Quoted 7 lines
Also, less churn in this patch if you don't change pte->ptep
(because if you do, you should then also do pteval->pte :) )
So I'd just leave that as is in this patch.
So I'll change pte->ptep and pteval->pte because I hate pte pointers called pte : )

Yeah, me too, but it's all over the place happening ... :)

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-06 09:46 UTC #23 lore raw

PATCH v3 5/9 mm/rmap: batch unmap folios belonging to uffd-wp VMAs

In reply to Dev Jain

Commit a67fe41e214f ("mm: rmap: support batched unmapping for file large folios") extended batched unmapping for file folios. That also required making install_uffd_wp_ptes_if_needed() support batching, but that was left out for the time being, and correctness was maintained by stopping batching in case the VMA the folio belongs to is marked uffd-wp.

Now that we have a batched version called install_uffd_wp_ptes_if_needed, simply call that. folio_unmap_pte_batch() ensures that the original state of the ptes is either all uffd or all non-uffd, so we maintain correctness.

If uffd-wp bit is there, we have the following transitions of ptes after unmapping:

1) anon folio: present -> uffd-wp swap
2) file folio: present -> uffd-wp marker

We must ensure that these ptes are not reprocessed by the while loop - if the batch length is less than the number of pages in the folio, then we must skip over this batch.

The page_vma_mapped_walk API ensures this - check_pte() will return true only if any of [pvmw->pfn, pvmw->pfn + nr_pages) is mapped by the pte. There is no pfn underlying either a uffd-wp swap pte or a uffd-wp marker pte, so check_pte returns false and we keep skipping until we hit a present entry, which is where we want to batch from next.

Signed-off-by Dev Jain <[email protected]>

---
 mm/rmap.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

mm/rmap.c +1 -4

index b17dce752a1ea..25813e3605991 100644
@@ -1965,9 +1965,6 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,
 	if (pte_unused(pte))
 		return 1;
 
-	if (userfaultfd_wp(vma))
-		return 1;
-
 	/*
 	 * If unmap fails, we need to restore the ptes. To avoid accidentally
 	 * upgrading write permissions for ptes that were not originally

@@ -2266,7 +2263,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		 * we may want to replace a none pte with a marker pte if
 		 * it's file-backed, so we don't lose the tracking info.
 		 */
-		install_uffd_wp_ptes_if_needed(vma, address, pvmw.pte, pteval, 1);
+		install_uffd_wp_ptes_if_needed(vma, address, pvmw.pte, pteval, nr_pages);
 
 		/* Update high watermark before we lower rss */
 		update_hiwater_rss(mm);

Signature · 1 line

2.34.1

David Hildenbrand (Arm)

maintainer Acked-by on v3 5/9

2026-05-11 07:41 UTC #24 lore raw

Re: [PATCH v3 5/9] mm/rmap: batch unmap folios belonging to uffd-wp VMAs

Quoted 21 lines · Dev Jain

Commit a67fe41e214f ("mm: rmap: support batched unmapping for file large folios") extended batched unmapping for file folios. That also required making install_uffd_wp_ptes_if_needed() support batching, but that was left out for the time being, and correctness was maintained by stopping batching in case the VMA the folio belongs to is marked uffd-wp.
Now that we have a batched version called install_uffd_wp_ptes_if_needed, simply call that. folio_unmap_pte_batch() ensures that the original state of the ptes is either all uffd or all non-uffd, so we maintain correctness.
If uffd-wp bit is there, we have the following transitions of ptes after unmapping:
1) anon folio: present -> uffd-wp swap
2) file folio: present -> uffd-wp marker
We must ensure that these ptes are not reprocessed by the while loop - if the batch length is less than the number of pages in the folio, then we must skip over this batch.

Quoted 5 lines

The page_vma_mapped_walk API ensures this - check_pte() will return true only if any of [pvmw->pfn, pvmw->pfn + nr_pages) is mapped by the pte. There is no pfn underlying either a uffd-wp swap pte or a uffd-wp marker pte, so check_pte returns false and we keep skipping until we hit a present entry, which is where we want to batch from next.

Right. Or a non-present ZONE_DEVICE entry. Your patch shouldn't change anything about that.

Quoted 21 lines

Signed-off-by Dev Jain <[email protected]>

---
 mm/rmap.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

mm/rmap.c +0 -3

index b17dce752a1ea..25813e3605991 100644
@@ -1965,9 +1965,6 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,
 	if (pte_unused(pte))
 		return 1;
 
-	if (userfaultfd_wp(vma))
-		return 1;
-
 	/*
 	 * If unmap fails, we need to restore the ptes. To avoid accidentally
 	 * upgrading write permissions for ptes that were not originally

mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 2269

Quoted 8 lines

mm/rmap.c +1 -1

@@ -2266,7 +2263,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		 * we may want to replace a none pte with a marker pte if
 		 * it's file-backed, so we don't lose the tracking info.
 		 */
-		install_uffd_wp_ptes_if_needed(vma, address, pvmw.pte, pteval, 1);
+		install_uffd_wp_ptes_if_needed(vma, address, pvmw.pte, pteval, nr_pages);
 
 		/* Update high watermark before we lower rss */
 		update_hiwater_rss(mm);

Yes, that should do the trick. folio_pte_batch_flags() will still respect the uffd-wp bit.

Acked-by David Hildenbrand (Arm) <[email protected]>

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-06 09:46 UTC #25 lore raw

PATCH v3 6/9 mm/swapfile: Add batched version of folio_dup_swap

In reply to Dev Jain

Add folio_dup_swap_pages to handle a batch of consecutive pages. Note that folio_dup_swap already can handle a subset of this: nr_pages == 1 and nr_pages == folio_nr_pages(folio). Generalize this to any nr_pages.

Currently we have a not-so-nice logic of passing in subpage == NULL if we mean to exercise the logic on the entire folio, and subpage != NULL if we want to exercise the logic on only that subpage. Remove this
indirection: the caller invokes folio_dup_swap_pages() if it wants to operate on a range of pages in the folio (i.e nr_pages may be anything between 1 till folio_nr_pages()), and invokes folio_dup_swap() if it wants to operate on the entire folio.

Signed-off-by Dev Jain <[email protected]>

---
 mm/rmap.c     |  2 +-
 mm/shmem.c    |  2 +-
 mm/swap.h     | 12 ++++++++++--
 mm/swapfile.c | 20 ++++++++++++--------
 4 files changed, 24 insertions(+), 12 deletions(-)

mm/rmap.c +1 -1

index 25813e3605991..352ba77d90f67 100644
@@ -2314,7 +2314,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 				goto finish_unmap;
 			}
 
-			if (folio_dup_swap(folio, subpage) < 0) {
+			if (folio_dup_swap_pages(folio, subpage, 1) < 0) {
 				set_pte_at(mm, address, pvmw.pte, pteval);
 				goto walk_abort;
 			}

mm/shmem.c +1 -1

index bab3529af23c5..5e4f521399847 100644
@@ -1698,7 +1698,7 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
 			spin_unlock(&shmem_swaplist_lock);
 		}
 
-		folio_dup_swap(folio, NULL);
+		folio_dup_swap(folio);
 		shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap));
 
 		BUG_ON(folio_mapped(folio));

mm/swap.h +10 -2

index a77016f2423b9..3c25f914e908b 100644
@@ -206,7 +206,9 @@ extern int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp);
  * folio_put_swap(): does the opposite thing of folio_dup_swap().
  */
 int folio_alloc_swap(struct folio *folio);
-int folio_dup_swap(struct folio *folio, struct page *subpage);
+int folio_dup_swap(struct folio *folio);
+int folio_dup_swap_pages(struct folio *folio, struct page *page,
+			 unsigned long nr_pages);
 void folio_put_swap(struct folio *folio, struct page *subpage);
 
 /* For internal use */

@@ -390,7 +392,13 @@ static inline int folio_alloc_swap(struct folio *folio)
 	return -EINVAL;
 }
 
-static inline int folio_dup_swap(struct folio *folio, struct page *page)
+static inline int folio_dup_swap(struct folio *folio)
+{
+	return -EINVAL;
+}
+
+static inline int folio_dup_swap_pages(struct folio *folio, struct page *page,
+		unsigned long nr_pages)
 {
 	return -EINVAL;
 }

mm/swapfile.c +12 -8

index c7e173b93e11d..28daf92839e77 100644
@@ -1740,9 +1740,10 @@ int folio_alloc_swap(struct folio *folio)
 }
 
 /**
- * folio_dup_swap() - Increase swap count of swap entries of a folio.
+ * folio_dup_swap_pages() - Increase swap count of swap entries of a folio.
  * @folio: folio with swap entries bounded.
- * @subpage: if not NULL, only increase the swap count of this subpage.
+ * @page: the first page in the folio to increase the swap count for.
+ * @nr_pages: the number of pages in the folio to increase the swap count for.
  *
  * Typically called when the folio is unmapped and have its swap entry to
  * take its place: Swap entries allocated to a folio has count == 0 and pinned

@@ -1756,23 +1757,26 @@ int folio_alloc_swap(struct folio *folio)
  * swap_put_entries_direct on its swap entry before this helper returns, or
  * the swap count may underflow.
  */
-int folio_dup_swap(struct folio *folio, struct page *subpage)
+int folio_dup_swap_pages(struct folio *folio, struct page *page,
+		unsigned long nr_pages)
 {
 	swp_entry_t entry = folio->swap;
-	unsigned long nr_pages = folio_nr_pages(folio);
 
 	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
 	VM_WARN_ON_FOLIO(!folio_test_swapcache(folio), folio);
 
-	if (subpage) {
-		entry.val += folio_page_idx(folio, subpage);
-		nr_pages = 1;
-	}
+	entry.val += folio_page_idx(folio, page);
 
 	return swap_dup_entries_cluster(swap_entry_to_info(entry),
 					swp_offset(entry), nr_pages);
 }
 
+int folio_dup_swap(struct folio *folio)
+{
+	return folio_dup_swap_pages(folio, folio_page(folio, 0),
+				    folio_nr_pages(folio));
+}
+
 /**
  * folio_put_swap() - Decrease swap count of swap entries of a folio.
  * @folio: folio with swap entries bounded, must be in swap cache and locked.

Signature · 1 line

2.34.1

David Hildenbrand (Arm)

maintainer

2026-05-11 07:45 UTC #26 lore raw

Re: [PATCH v3 6/9] mm/swapfile: Add batched version of folio_dup_swap

Quoted 25 lines · Dev Jain

Add folio_dup_swap_pages to handle a batch of consecutive pages. Note that folio_dup_swap already can handle a subset of this: nr_pages == 1 and nr_pages == folio_nr_pages(folio). Generalize this to any nr_pages.
Currently we have a not-so-nice logic of passing in subpage == NULL if we mean to exercise the logic on the entire folio, and subpage != NULL if we want to exercise the logic on only that subpage. Remove this
indirection: the caller invokes folio_dup_swap_pages() if it wants to operate on a range of pages in the folio (i.e nr_pages may be anything between 1 till folio_nr_pages()), and invokes folio_dup_swap() if it wants to operate on the entire folio.
Signed-off-by Dev Jain <[email protected]>
---
 mm/rmap.c     |  2 +-
 mm/shmem.c    |  2 +-
 mm/swap.h     | 12 ++++++++++--
 mm/swapfile.c | 20 ++++++++++++--------
 4 files changed, 24 insertions(+), 12 deletions(-)
mm/rmap.c +0 -0
index 25813e3605991..352ba77d90f67 100644

mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 2317

Quoted 5 lines

mm/rmap.c +1 -1

@@ -2314,7 +2314,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 				goto finish_unmap;
 			}
 
-			if (folio_dup_swap(folio, subpage) < 0) {
+			if (folio_dup_swap_pages(folio, subpage, 1) < 0) {

Can you throw in a patch to rename subpage -> page first?

Quoted 32 lines

mm/rmap.c +0 -0

 				set_pte_at(mm, address, pvmw.pte, pteval);
 				goto walk_abort;
 			}

mm/shmem.c +1 -1

index bab3529af23c5..5e4f521399847 100644
@@ -1698,7 +1698,7 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
 			spin_unlock(&shmem_swaplist_lock);
 		}
 
-		folio_dup_swap(folio, NULL);
+		folio_dup_swap(folio);
 		shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap));
 
 		BUG_ON(folio_mapped(folio));

mm/swap.h +3 -1

index a77016f2423b9..3c25f914e908b 100644
@@ -206,7 +206,9 @@ extern int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp);
  * folio_put_swap(): does the opposite thing of folio_dup_swap().
  */
 int folio_alloc_swap(struct folio *folio);
-int folio_dup_swap(struct folio *folio, struct page *subpage);
+int folio_dup_swap(struct folio *folio);
+int folio_dup_swap_pages(struct folio *folio, struct page *page,
+			 unsigned long nr_pages);
 void folio_put_swap(struct folio *folio, struct page *subpage);
 
 /* For internal use */

mm/swap.h static inline int folio_alloc_swap(struct folio *folio) · line 399

Quoted 9 lines

mm/swap.h +5 -1

@@ -390,7 +392,13 @@ static inline int folio_alloc_swap(struct folio *folio)
 	return -EINVAL;
 }
 
-static inline int folio_dup_swap(struct folio *folio, struct page *page)
+static inline int folio_dup_swap(struct folio *folio)
+{
+	return -EINVAL;
+}
+

Much cleaner.

Can we have a common folio_dup_swap() function instead that does what the variant below does? (call folio_dup_swap_pages())

Quoted 41 lines

mm/swap.h +2 -0

+static inline int folio_dup_swap_pages(struct folio *folio, struct page *page,
+		unsigned long nr_pages)
 {
 	return -EINVAL;
 }

mm/swapfile.c +6 -8

index c7e173b93e11d..28daf92839e77 100644
@@ -1740,9 +1740,10 @@ int folio_alloc_swap(struct folio *folio)
 }
 
 /**
- * folio_dup_swap() - Increase swap count of swap entries of a folio.
+ * folio_dup_swap_pages() - Increase swap count of swap entries of a folio.
  * @folio: folio with swap entries bounded.
- * @subpage: if not NULL, only increase the swap count of this subpage.
+ * @page: the first page in the folio to increase the swap count for.
+ * @nr_pages: the number of pages in the folio to increase the swap count for.
  *
  * Typically called when the folio is unmapped and have its swap entry to
  * take its place: Swap entries allocated to a folio has count == 0 and pinned

@@ -1756,23 +1757,26 @@ int folio_alloc_swap(struct folio *folio)
  * swap_put_entries_direct on its swap entry before this helper returns, or
  * the swap count may underflow.
  */
-int folio_dup_swap(struct folio *folio, struct page *subpage)
+int folio_dup_swap_pages(struct folio *folio, struct page *page,
+		unsigned long nr_pages)
 {
 	swp_entry_t entry = folio->swap;
-	unsigned long nr_pages = folio_nr_pages(folio);
 
 	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
 	VM_WARN_ON_FOLIO(!folio_test_swapcache(folio), folio);
 
-	if (subpage) {
-		entry.val += folio_page_idx(folio, subpage);
-		nr_pages = 1;
-	}
+	entry.val += folio_page_idx(folio, page);

mm/swapfile.c int folio_alloc_swap(struct folio *folio) · line 1778

Quoted 10 lines

mm/swapfile.c +5 -0

 
 	return swap_dup_entries_cluster(swap_entry_to_info(entry),
 					swp_offset(entry), nr_pages);
 }
 
+int folio_dup_swap(struct folio *folio)
+{
+	return folio_dup_swap_pages(folio, folio_page(folio, 0),
+				    folio_nr_pages(folio));
+}

Can you add simplistic kerneldoc for folio_dup_swap() as well, and mostly just link to folio_dup_swap_pages() ?

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-12 06:07 UTC #27 lore raw

Re: [PATCH v3 6/9] mm/swapfile: Add batched version of folio_dup_swap

Quoted 29 lines · David Hildenbrand (Arm)

Quoted 28 lines · Dev Jain
Add folio_dup_swap_pages to handle a batch of consecutive pages. Note that folio_dup_swap already can handle a subset of this: nr_pages == 1 and nr_pages == folio_nr_pages(folio). Generalize this to any nr_pages.
Currently we have a not-so-nice logic of passing in subpage == NULL if we mean to exercise the logic on the entire folio, and subpage != NULL if we want to exercise the logic on only that subpage. Remove this
indirection: the caller invokes folio_dup_swap_pages() if it wants to operate on a range of pages in the folio (i.e nr_pages may be anything between 1 till folio_nr_pages()), and invokes folio_dup_swap() if it wants to operate on the entire folio.
Signed-off-by Dev Jain <[email protected]>
---
 mm/rmap.c     |  2 +-
 mm/shmem.c    |  2 +-
 mm/swap.h     | 12 ++++++++++--
 mm/swapfile.c | 20 ++++++++++++--------
 4 files changed, 24 insertions(+), 12 deletions(-)
mm/rmap.c +0 -0
index 25813e3605991..352ba77d90f67 100644
@@ -2314,7 +2314,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 				goto finish_unmap;
 			}
 

Quoted 4 lines

mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 2317
Quoted 2 lines
mm/rmap.c +1 -1
-			if (folio_dup_swap(folio, subpage) < 0) {
+			if (folio_dup_swap_pages(folio, subpage, 1) < 0) {
Can you throw in a patch to rename subpage -> page first?

Sure.

Quoted 45 lines

Quoted 33 lines

mm/rmap.c +0 -0

 				set_pte_at(mm, address, pvmw.pte, pteval);
 				goto walk_abort;
 			}

mm/shmem.c +1 -1

index bab3529af23c5..5e4f521399847 100644
@@ -1698,7 +1698,7 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
 			spin_unlock(&shmem_swaplist_lock);
 		}
 
-		folio_dup_swap(folio, NULL);
+		folio_dup_swap(folio);
 		shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap));
 
 		BUG_ON(folio_mapped(folio));

mm/swap.h +3 -1

index a77016f2423b9..3c25f914e908b 100644
@@ -206,7 +206,9 @@ extern int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp);
  * folio_put_swap(): does the opposite thing of folio_dup_swap().
  */
 int folio_alloc_swap(struct folio *folio);
-int folio_dup_swap(struct folio *folio, struct page *subpage);
+int folio_dup_swap(struct folio *folio);
+int folio_dup_swap_pages(struct folio *folio, struct page *page,
+			 unsigned long nr_pages);
 void folio_put_swap(struct folio *folio, struct page *subpage);
 
 /* For internal use */

mm/swap.h static inline int folio_alloc_swap(struct folio *folio) · line 399

Quoted 9 lines

mm/swap.h +5 -1

@@ -390,7 +392,13 @@ static inline int folio_alloc_swap(struct folio *folio)
 	return -EINVAL;
 }
 
-static inline int folio_dup_swap(struct folio *folio, struct page *page)
+static inline int folio_dup_swap(struct folio *folio)
+{
+	return -EINVAL;
+}
+

Much cleaner.

Quoted 2 lines

Can we have a common folio_dup_swap() function instead that does what the variant below does? (call folio_dup_swap_pages())

I am guessing you are saying to simply remove the folio_dup_swap stub because we have a stub for folio_dup_swap_pages anyways, I'll do that.

Quoted 47 lines

mm/swap.h +2 -0

+static inline int folio_dup_swap_pages(struct folio *folio, struct page *page,
+		unsigned long nr_pages)
 {
 	return -EINVAL;
 }

mm/swapfile.c +6 -8

index c7e173b93e11d..28daf92839e77 100644
@@ -1740,9 +1740,10 @@ int folio_alloc_swap(struct folio *folio)
 }
 
 /**
- * folio_dup_swap() - Increase swap count of swap entries of a folio.
+ * folio_dup_swap_pages() - Increase swap count of swap entries of a folio.
  * @folio: folio with swap entries bounded.
- * @subpage: if not NULL, only increase the swap count of this subpage.
+ * @page: the first page in the folio to increase the swap count for.
+ * @nr_pages: the number of pages in the folio to increase the swap count for.
  *
  * Typically called when the folio is unmapped and have its swap entry to
  * take its place: Swap entries allocated to a folio has count == 0 and pinned

@@ -1756,23 +1757,26 @@ int folio_alloc_swap(struct folio *folio)
  * swap_put_entries_direct on its swap entry before this helper returns, or
  * the swap count may underflow.
  */
-int folio_dup_swap(struct folio *folio, struct page *subpage)
+int folio_dup_swap_pages(struct folio *folio, struct page *page,
+		unsigned long nr_pages)
 {
 	swp_entry_t entry = folio->swap;
-	unsigned long nr_pages = folio_nr_pages(folio);
 
 	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
 	VM_WARN_ON_FOLIO(!folio_test_swapcache(folio), folio);
 
-	if (subpage) {
-		entry.val += folio_page_idx(folio, subpage);
-		nr_pages = 1;
-	}
+	entry.val += folio_page_idx(folio, page);
 
 	return swap_dup_entries_cluster(swap_entry_to_info(entry),
 					swp_offset(entry), nr_pages);
 }

Quoted 8 lines

mm/swapfile.c int folio_alloc_swap(struct folio *folio) · line 1778
Quoted 5 lines
mm/swapfile.c +5 -0
+int folio_dup_swap(struct folio *folio)
+{
+	return folio_dup_swap_pages(folio, folio_page(folio, 0),
+				    folio_nr_pages(folio));
+}
Can you add simplistic kerneldoc for folio_dup_swap() as well, and mostly just link to folio_dup_swap_pages() ?

Okay.

Quoted 1 line

David Hildenbrand (Arm)

maintainer

2026-05-12 06:36 UTC #28 lore raw

Re: [PATCH v3 6/9] mm/swapfile: Add batched version of folio_dup_swap

Quoted 84 lines · Dev Jain

Quoted 29 lines · David Hildenbrand (Arm)
Quoted 28 lines · Dev Jain
Add folio_dup_swap_pages to handle a batch of consecutive pages. Note that folio_dup_swap already can handle a subset of this: nr_pages == 1 and nr_pages == folio_nr_pages(folio). Generalize this to any nr_pages.
Currently we have a not-so-nice logic of passing in subpage == NULL if we mean to exercise the logic on the entire folio, and subpage != NULL if we want to exercise the logic on only that subpage. Remove this
indirection: the caller invokes folio_dup_swap_pages() if it wants to operate on a range of pages in the folio (i.e nr_pages may be anything between 1 till folio_nr_pages()), and invokes folio_dup_swap() if it wants to operate on the entire folio.
Signed-off-by Dev Jain <[email protected]>
---
 mm/rmap.c     |  2 +-
 mm/shmem.c    |  2 +-
 mm/swap.h     | 12 ++++++++++--
 mm/swapfile.c | 20 ++++++++++++--------
 4 files changed, 24 insertions(+), 12 deletions(-)
mm/rmap.c +0 -0
index 25813e3605991..352ba77d90f67 100644
@@ -2314,7 +2314,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 				goto finish_unmap;
 			}
 
Quoted 4 lines
mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 2317
Quoted 2 lines
mm/rmap.c +1 -1
-			if (folio_dup_swap(folio, subpage) < 0) {
+			if (folio_dup_swap_pages(folio, subpage, 1) < 0) {
Can you throw in a patch to rename subpage -> page first?
Sure.
Quoted 45 lines
Quoted 33 lines
mm/rmap.c +0 -0
 				set_pte_at(mm, address, pvmw.pte, pteval);
 				goto walk_abort;
 			}
mm/shmem.c +1 -1
index bab3529af23c5..5e4f521399847 100644
@@ -1698,7 +1698,7 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
 			spin_unlock(&shmem_swaplist_lock);
 		}
 
-		folio_dup_swap(folio, NULL);
+		folio_dup_swap(folio);
 		shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap));
 
 		BUG_ON(folio_mapped(folio));
mm/swap.h +3 -1
index a77016f2423b9..3c25f914e908b 100644
@@ -206,7 +206,9 @@ extern int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp);
  * folio_put_swap(): does the opposite thing of folio_dup_swap().
  */
 int folio_alloc_swap(struct folio *folio);
-int folio_dup_swap(struct folio *folio, struct page *subpage);
+int folio_dup_swap(struct folio *folio);
+int folio_dup_swap_pages(struct folio *folio, struct page *page,
+			 unsigned long nr_pages);
 void folio_put_swap(struct folio *folio, struct page *subpage);
 
 /* For internal use */
mm/swap.h static inline int folio_alloc_swap(struct folio *folio) · line 399
Quoted 9 lines
mm/swap.h +5 -1
@@ -390,7 +392,13 @@ static inline int folio_alloc_swap(struct folio *folio)
 	return -EINVAL;
 }
 
-static inline int folio_dup_swap(struct folio *folio, struct page *page)
+static inline int folio_dup_swap(struct folio *folio)
+{
+	return -EINVAL;
+}
+
Much cleaner.

Quoted 5 lines

Quoted 2 lines
Can we have a common folio_dup_swap() function instead that does what the variant below does? (call folio_dup_swap_pages())
I am guessing you are saying to simply remove the folio_dup_swap stub because we have a stub for folio_dup_swap_pages anyways, I'll do that.

Right, dropping the "return -EINVAL;" variant, and instead move ...

Quoted 6 lines

Quoted 6 lines
mm/swap.h static inline int folio_alloc_swap(struct folio *folio) · line 404
Quoted 6 lines
mm/swap.h +5 -0
+int folio_dup_swap(struct folio *folio)
+{
+	return folio_dup_swap_pages(folio, folio_page(folio, 0),
+				    folio_nr_pages(folio));
+}

... this variant out of the ifdef, that will call the folio_dup_swap_pages() stub.

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-06 09:46 UTC #29 lore raw

PATCH v3 7/9 mm/swapfile: Add batched version of folio_put_swap

In reply to Dev Jain

Add folio_put_swap_pages to handle a batch of consecutive pages. Note that folio_put_swap already can handle a subset of this: nr_pages == 1 and nr_pages == folio_nr_pages(folio). Generalize this to any nr_pages.

Currently we have a not-so-nice logic of passing in subpage == NULL if we mean to exercise the logic on the entire folio, and subpage != NULL if we want to exercise the logic on only that subpage. Remove this
indirection: the caller invokes folio_put_swap_pages() if it wants to operate on a range of pages in the folio (i.e nr_pages may be anything between 1 till folio_nr_pages()), and invokes folio_put_swap() if it wants to operate on the entire folio.

Signed-off-by Dev Jain <[email protected]>

---
 mm/memory.c   |  6 +++---
 mm/rmap.c     |  4 ++--
 mm/shmem.c    |  6 +++---
 mm/swap.h     | 11 +++++++++--
 mm/swapfile.c | 22 +++++++++++++---------
 5 files changed, 30 insertions(+), 19 deletions(-)

mm/memory.c +3 -3

index f14311c4d2001..c5605a779ce4d 100644
@@ -5104,7 +5104,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 	if (unlikely(folio != swapcache)) {
 		folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE);
 		folio_add_lru_vma(folio, vma);
-		folio_put_swap(swapcache, NULL);
+		folio_put_swap(swapcache);
 	} else if (!folio_test_anon(folio)) {
 		/*
 		 * We currently only expect !anon folios that are fully

@@ -5113,12 +5113,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		VM_WARN_ON_ONCE_FOLIO(folio_nr_pages(folio) != nr_pages, folio);
 		VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio);
 		folio_add_new_anon_rmap(folio, vma, address, rmap_flags);
-		folio_put_swap(folio, NULL);
+		folio_put_swap(folio);
 	} else {
 		VM_WARN_ON_ONCE(nr_pages != 1 && nr_pages != folio_nr_pages(folio));
 		folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address,
 					 rmap_flags);
-		folio_put_swap(folio, nr_pages == 1 ? page : NULL);
+		folio_put_swap_pages(folio, page, nr_pages);
 	}
 
 	VM_BUG_ON(!folio_test_anon(folio) ||

mm/rmap.c +2 -2

index 352ba77d90f67..7cbf850182187 100644
@@ -2325,7 +2325,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 			 * so we'll not check/care.
 			 */
 			if (arch_unmap_one(mm, vma, address, pteval) < 0) {
-				folio_put_swap(folio, subpage);
+				folio_put_swap_pages(folio, subpage, 1);
 				set_pte_at(mm, address, pvmw.pte, pteval);
 				goto walk_abort;
 			}

@@ -2333,7 +2333,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 			/* See folio_try_share_anon_rmap(): clear PTE first. */
 			if (anon_exclusive &&
 			    folio_try_share_anon_rmap_pte(folio, subpage)) {
-				folio_put_swap(folio, subpage);
+				folio_put_swap_pages(folio, subpage, 1);
 				set_pte_at(mm, address, pvmw.pte, pteval);
 				goto walk_abort;
 			}

mm/shmem.c +3 -3

index 5e4f521399847..bb7e0fc305d87 100644
@@ -1719,7 +1719,7 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
 		/* Swap entry might be erased by racing shmem_free_swap() */
 		if (!error) {
 			shmem_recalc_inode(inode, 0, -nr_pages);
-			folio_put_swap(folio, NULL);
+			folio_put_swap(folio);
 		}
 
 		/*

@@ -2199,7 +2199,7 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,
 
 	nr_pages = folio_nr_pages(folio);
 	folio_wait_writeback(folio);
-	folio_put_swap(folio, NULL);
+	folio_put_swap(folio);
 	swap_cache_del_folio(folio);
 	/*
 	 * Don't treat swapin error folio as alloced. Otherwise inode->i_blocks

@@ -2429,7 +2429,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
 	if (sgp == SGP_WRITE)
 		folio_mark_accessed(folio);
 
-	folio_put_swap(folio, NULL);
+	folio_put_swap(folio);
 	swap_cache_del_folio(folio);
 	folio_mark_dirty(folio);
 	put_swap_device(si);

mm/swap.h +9 -2

index 3c25f914e908b..343547469927a 100644
@@ -209,7 +209,9 @@ int folio_alloc_swap(struct folio *folio);
 int folio_dup_swap(struct folio *folio);
 int folio_dup_swap_pages(struct folio *folio, struct page *page,
 			 unsigned long nr_pages);
-void folio_put_swap(struct folio *folio, struct page *subpage);
+void folio_put_swap(struct folio *folio);
+void folio_put_swap_pages(struct folio *folio, struct page *page,
+			  unsigned long nr_pages);
 
 /* For internal use */
 extern void __swap_cluster_free_entries(struct swap_info_struct *si,

@@ -403,7 +405,12 @@ static inline int folio_dup_swap_pages(struct folio *folio, struct page *page,
 	return -EINVAL;
 }
 
-static inline void folio_put_swap(struct folio *folio, struct page *page)
+static inline void folio_put_swap(struct folio *folio)
+{
+}
+
+static inline void folio_put_swap_pages(struct folio *folio, struct page *page,
+				  unsigned long nr_pages)
 {
 }

mm/swapfile.c +13 -9

index 28daf92839e77..ac576cc63b194 100644
@@ -1778,31 +1778,34 @@ int folio_dup_swap(struct folio *folio)
 }
 
 /**
- * folio_put_swap() - Decrease swap count of swap entries of a folio.
+ * folio_put_swap_pages() - Decrease swap count of swap entries of a folio.
  * @folio: folio with swap entries bounded, must be in swap cache and locked.
- * @subpage: if not NULL, only decrease the swap count of this subpage.
+ * @page: the first page in the folio to decrease the swap count for.
+ * @nr_pages: the number of pages in the folio to decrease the swap count for.
  *
  * This won't free the swap slots even if swap count drops to zero, they are
  * still pinned by the swap cache. User may call folio_free_swap to free them.
  * Context: Caller must ensure the folio is locked and in the swap cache.
  */
-void folio_put_swap(struct folio *folio, struct page *subpage)
+void folio_put_swap_pages(struct folio *folio, struct page *page,
+			  unsigned long nr_pages)
 {
 	swp_entry_t entry = folio->swap;
-	unsigned long nr_pages = folio_nr_pages(folio);
 	struct swap_info_struct *si = __swap_entry_to_info(entry);
 
 	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
 	VM_WARN_ON_FOLIO(!folio_test_swapcache(folio), folio);
 
-	if (subpage) {
-		entry.val += folio_page_idx(folio, subpage);
-		nr_pages = 1;
-	}
+	entry.val += folio_page_idx(folio, page);
 
 	swap_put_entries_cluster(si, swp_offset(entry), nr_pages, false);
 }
 
+void folio_put_swap(struct folio *folio)
+{
+	folio_put_swap_pages(folio, folio_page(folio, 0), folio_nr_pages(folio));
+}
+
 /*
  * When we get a swap entry, if there aren't some other ways to
  * prevent swapoff, such as the folio in swap cache is locked, RCU

@@ -2443,7 +2446,8 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,
 		new_pte = pte_mkuffd_wp(new_pte);
 setpte:
 	set_pte_at(vma->vm_mm, addr, pte, new_pte);
-	folio_put_swap(swapcache, folio_file_page(swapcache, swp_offset(entry)));
+	folio_put_swap_pages(swapcache,
+			     folio_file_page(swapcache, swp_offset(entry)), 1);
 out:
 	if (pte)
 		pte_unmap_unlock(pte, ptl);

Signature · 1 line

2.34.1

David Hildenbrand (Arm)

maintainer

2026-05-11 08:07 UTC #30 lore raw

Re: [PATCH v3 7/9] mm/swapfile: Add batched version of folio_put_swap

Quoted 14 lines · Dev Jain

Add folio_put_swap_pages to handle a batch of consecutive pages. Note that folio_put_swap already can handle a subset of this: nr_pages == 1 and nr_pages == folio_nr_pages(folio). Generalize this to any nr_pages.
Currently we have a not-so-nice logic of passing in subpage == NULL if we mean to exercise the logic on the entire folio, and subpage != NULL if we want to exercise the logic on only that subpage. Remove this
indirection: the caller invokes folio_put_swap_pages() if it wants to operate on a range of pages in the folio (i.e nr_pages may be anything between 1 till folio_nr_pages()), and invokes folio_put_swap() if it wants to operate on the entire folio.
Signed-off-by Dev Jain <[email protected]>
---

[...]

mm/swap.h static inline int folio_dup_swap_pages(struct folio *folio, struct page *page, · line 410

Quoted 5 lines

mm/swap.h +3 -1

-static inline void folio_put_swap(struct folio *folio, struct page *page)
+static inline void folio_put_swap(struct folio *folio)
+{
+}

Same comment regarding common function.

Quoted 39 lines

mm/swap.h +3 -0

+
+static inline void folio_put_swap_pages(struct folio *folio, struct page *page,
+				  unsigned long nr_pages)
 {
 }

mm/swapfile.c +5 -7

index 28daf92839e77..ac576cc63b194 100644
@@ -1778,31 +1778,34 @@ int folio_dup_swap(struct folio *folio)
 }
 
 /**
- * folio_put_swap() - Decrease swap count of swap entries of a folio.
+ * folio_put_swap_pages() - Decrease swap count of swap entries of a folio.
  * @folio: folio with swap entries bounded, must be in swap cache and locked.
- * @subpage: if not NULL, only decrease the swap count of this subpage.
+ * @page: the first page in the folio to decrease the swap count for.
+ * @nr_pages: the number of pages in the folio to decrease the swap count for.
  *
  * This won't free the swap slots even if swap count drops to zero, they are
  * still pinned by the swap cache. User may call folio_free_swap to free them.
  * Context: Caller must ensure the folio is locked and in the swap cache.
  */
-void folio_put_swap(struct folio *folio, struct page *subpage)
+void folio_put_swap_pages(struct folio *folio, struct page *page,
+			  unsigned long nr_pages)
 {
 	swp_entry_t entry = folio->swap;
-	unsigned long nr_pages = folio_nr_pages(folio);
 	struct swap_info_struct *si = __swap_entry_to_info(entry);
 
 	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
 	VM_WARN_ON_FOLIO(!folio_test_swapcache(folio), folio);
 
-	if (subpage) {
-		entry.val += folio_page_idx(folio, subpage);
-		nr_pages = 1;

mm/swapfile.c int folio_dup_swap(struct folio *folio) · line 1807

Quoted 10 lines

mm/swapfile.c +5 -1

-	}
+	entry.val += folio_page_idx(folio, page);
 
 	swap_put_entries_cluster(si, swp_offset(entry), nr_pages, false);
 }
 
+void folio_put_swap(struct folio *folio)
+{
+	folio_put_swap_pages(folio, folio_page(folio, 0), folio_nr_pages(folio));
+}

Same comment regarding kerneldoc :)

... and same comment that this definitely looks nicer.

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-06 09:46 UTC #31 lore raw

PATCH v3 8/9 mm/rmap: Add batched version of folio_try_share_anon_rmap_pte

In reply to Dev Jain

To enable batched unmapping of anonymous folios, we need to handle the sharing of exclusive pages. Hence, a batched version of folio_try_share_anon_rmap_pte is required.

Currently, the sole purpose of nr_pages in __folio_try_share_anon_rmap is to do some rmap sanity checks. Add helpers to clear the PageAnonExclusive bit on a batch of nr_pages. Note that __folio_try_share_anon_rmap can receive nr_pages == HPAGE_PMD_NR from the PMD path, but currently we only clear the bit on the head page. Retain this behaviour by setting nr_pages = 1 in case the caller is folio_try_share_anon_rmap_pmd.

While at it, convert nr_pages to unsigned long to future-proof from overflow in case P4D-huge mappings etc get supported down the road. I haven't made such a change in each function receiving nr_pages in try_to_unmap_one - perhaps this can be done incrementally.

Signed-off-by Dev Jain <[email protected]>

---
 include/linux/mm.h   | 11 +++++++++++
 include/linux/rmap.h | 27 ++++++++++++++++++++-------
 2 files changed, 31 insertions(+), 7 deletions(-)

include/linux/mm.h +11 -0

index 31e27ff6a35fa..0b77329cf57a4 100644
@@ -243,6 +243,17 @@ static inline unsigned long folio_page_idx(const struct folio *folio,
 	return page - &folio->page;
 }
 
+static __always_inline void folio_clear_pages_anon_exclusive(struct page *page,
+		unsigned long nr_pages)
+{
+	for (;;) {
+		ClearPageAnonExclusive(page);
+		if (--nr_pages == 0)
+			break;
+		++page;
+	}
+}
+
 static inline struct folio *lru_to_folio(struct list_head *head)
 {
 	return list_entry((head)->prev, struct folio, lru);

include/linux/rmap.h +20 -7

index 8dc0871e5f001..f3b3ee3955afc 100644
@@ -706,15 +706,19 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio,
 }
 
 static __always_inline int __folio_try_share_anon_rmap(struct folio *folio,
-		struct page *page, int nr_pages, enum pgtable_level level)
+		struct page *page, unsigned long nr_pages, enum pgtable_level level)
 {
 	VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);
 	VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio);
 	__folio_rmap_sanity_checks(folio, page, nr_pages, level);
 
+	/* We only clear anon-exclusive from head page of PMD folio */
+	if (level == PGTABLE_LEVEL_PMD)
+		nr_pages = 1;
+
 	/* device private folios cannot get pinned via GUP. */
 	if (unlikely(folio_is_device_private(folio))) {
-		ClearPageAnonExclusive(page);
+		folio_clear_pages_anon_exclusive(page, nr_pages);
 		return 0;
 	}

@@ -766,7 +770,7 @@ static __always_inline int __folio_try_share_anon_rmap(struct folio *folio,
 
 	if (unlikely(folio_maybe_dma_pinned(folio)))
 		return -EBUSY;
-	ClearPageAnonExclusive(page);
+	folio_clear_pages_anon_exclusive(page, nr_pages);
 
 	/*
 	 * This is conceptually a smp_wmb() paired with the smp_rmb() in

@@ -778,11 +782,12 @@ static __always_inline int __folio_try_share_anon_rmap(struct folio *folio,
 }
 
 /**
- * folio_try_share_anon_rmap_pte - try marking an exclusive anonymous page
- *				   mapped by a PTE possibly shared to prepare
+ * folio_try_share_anon_rmap_ptes - try marking exclusive anonymous pages
+ *				   mapped by PTEs possibly shared to prepare
  *				   for KSM or temporary unmapping
  * @folio:	The folio to share a mapping of
- * @page:	The mapped exclusive page
+ * @page:	The first mapped exclusive page of the batch in the folio
+ * @nr_pages:	The number of pages to share in the folio (batch size)
  *
  * The caller needs to hold the page table lock and has to have the page table
  * entries cleared/invalidated.

@@ -797,11 +802,19 @@ static __always_inline int __folio_try_share_anon_rmap(struct folio *folio,
  *
  * Returns 0 if marking the mapped page possibly shared succeeded. Returns
  * -EBUSY otherwise.
+ *
+ * The caller needs to hold the page table lock.
  */
+static inline int folio_try_share_anon_rmap_ptes(struct folio *folio,
+		struct page *page, unsigned long nr_pages)
+{
+	return __folio_try_share_anon_rmap(folio, page, nr_pages, PGTABLE_LEVEL_PTE);
+}
+
 static inline int folio_try_share_anon_rmap_pte(struct folio *folio,
 		struct page *page)
 {
-	return __folio_try_share_anon_rmap(folio, page, 1, PGTABLE_LEVEL_PTE);
+	return folio_try_share_anon_rmap_ptes(folio, page, 1);
 }
 
 /**

Signature · 1 line

2.34.1

David Hildenbrand (Arm)

maintainer

2026-05-11 08:13 UTC #32 lore raw

Re: [PATCH v3 8/9] mm/rmap: Add batched version of folio_try_share_anon_rmap_pte

Quoted 30 lines · Dev Jain

To enable batched unmapping of anonymous folios, we need to handle the sharing of exclusive pages. Hence, a batched version of folio_try_share_anon_rmap_pte is required.
Currently, the sole purpose of nr_pages in __folio_try_share_anon_rmap is to do some rmap sanity checks. Add helpers to clear the PageAnonExclusive bit on a batch of nr_pages. Note that __folio_try_share_anon_rmap can receive nr_pages == HPAGE_PMD_NR from the PMD path, but currently we only clear the bit on the head page. Retain this behaviour by setting nr_pages = 1 in case the caller is folio_try_share_anon_rmap_pmd.
While at it, convert nr_pages to unsigned long to future-proof from overflow in case P4D-huge mappings etc get supported down the road. I haven't made such a change in each function receiving nr_pages in try_to_unmap_one - perhaps this can be done incrementally.
Signed-off-by Dev Jain <[email protected]>
---
 include/linux/mm.h   | 11 +++++++++++
 include/linux/rmap.h | 27 ++++++++++++++++++++-------
 2 files changed, 31 insertions(+), 7 deletions(-)
include/linux/mm.h +0 -0
index 31e27ff6a35fa..0b77329cf57a4 100644
@@ -243,6 +243,17 @@ static inline unsigned long folio_page_idx(const struct folio *folio,
 	return page - &folio->page;
 }
 

include/linux/mm.h static inline unsigned long folio_page_idx(const struct folio *folio, · line 255

Quoted 10 lines

include/linux/mm.h +10 -0

+static __always_inline void folio_clear_pages_anon_exclusive(struct page *page,
+		unsigned long nr_pages)
+{
+	for (;;) {
+		ClearPageAnonExclusive(page);
+		if (--nr_pages == 0)
+			break;
+		++page;
+	}
+}

Something called folio that doesn't consume a folio is odd. I'd prefer we don't add this.

Is there a chance to simply change __folio_try_share_anon_rmap, so we get a single loop inline?

include/linux/rmap.h +11 ~~-12~~

index 8dc0871e5f00..5a1c874b2112 100644
@@ -708,16 +708,13 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio,
 static __always_inline int __folio_try_share_anon_rmap(struct folio *folio,
                struct page *page, int nr_pages, enum pgtable_level level)
 {
+       /* device private folios cannot get pinned via GUP. */
+       const bool pinnable = !folio_is_device_private(folio);
+
        VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);
        VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio);
        __folio_rmap_sanity_checks(folio, page, nr_pages, level);
 
-       /* device private folios cannot get pinned via GUP. */
-       if (unlikely(folio_is_device_private(folio))) {
-               ClearPageAnonExclusive(page);
-               return 0;
-       }
-
        /*
         * We have to make sure that when we clear PageAnonExclusive, that
         * the page is not pinned and that concurrent GUP-fast won't succeed in

@@ -760,19 +757,21 @@ static __always_inline int __folio_try_share_anon_rmap(struct folio *folio,
         * so we use explicit ones here.
         */
 
-       /* Paired with the memory barrier in try_grab_folio(). */
-       if (IS_ENABLED(CONFIG_HAVE_GUP_FAST))
-               smp_mb();
+       if (pinnable) {
+               /* Paired with the memory barrier in try_grab_folio(). */
+               if (IS_ENABLED(CONFIG_HAVE_GUP_FAST))
+                       smp_mb();
 
-       if (unlikely(folio_maybe_dma_pinned(folio)))
-               return -EBUSY;
+               if (unlikely(folio_maybe_dma_pinned(folio)))
+                       return -EBUSY;
+       }
        ClearPageAnonExclusive(page);
 
        /*
         * This is conceptually a smp_wmb() paired with the smp_rmb() in
         * gup_must_unshare().
         */
-       if (IS_ENABLED(CONFIG_HAVE_GUP_FAST))
+       if (pinnable && IS_ENABLED(CONFIG_HAVE_GUP_FAST))
                smp_mb__after_atomic();
        return 0;

Signature · 3 lines

Cheers,

David

David Hildenbrand (Arm)

maintainer

2026-05-11 08:14 UTC #33 lore raw

Re: [PATCH v3 8/9] mm/rmap: Add batched version of folio_try_share_anon_rmap_pte

Quoted 12 lines

Something called folio that doesn't consume a folio is odd. I'd prefer we don't add this.
Is there a chance to simply change __folio_try_share_anon_rmap, so we get a single loop inline?
include/linux/rmap.h +0 -0
index 8dc0871e5f00..5a1c874b2112 100644

include/linux/rmap.h static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, · line 713

Quoted 6 lines

include/linux/rmap.h +3 -0

@@ -708,16 +708,13 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio,
 static __always_inline int __folio_try_share_anon_rmap(struct folio *folio,
                struct page *page, int nr_pages, enum pgtable_level level)
 {
+       /* device private folios cannot get pinned via GUP. */
+       const bool pinnable = !folio_is_device_private(folio);
+

Throw in likely() to keep that unchanged.

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-12 08:57 UTC #34 lore raw

Re: [PATCH v3 8/9] mm/rmap: Add batched version of folio_try_share_anon_rmap_pte

In reply to David Hildenbrand (Arm)

Quoted 90 lines · David Hildenbrand (Arm)

Quoted 30 lines · Dev Jain

To enable batched unmapping of anonymous folios, we need to handle the sharing of exclusive pages. Hence, a batched version of folio_try_share_anon_rmap_pte is required.
Currently, the sole purpose of nr_pages in __folio_try_share_anon_rmap is to do some rmap sanity checks. Add helpers to clear the PageAnonExclusive bit on a batch of nr_pages. Note that __folio_try_share_anon_rmap can receive nr_pages == HPAGE_PMD_NR from the PMD path, but currently we only clear the bit on the head page. Retain this behaviour by setting nr_pages = 1 in case the caller is folio_try_share_anon_rmap_pmd.
While at it, convert nr_pages to unsigned long to future-proof from overflow in case P4D-huge mappings etc get supported down the road. I haven't made such a change in each function receiving nr_pages in try_to_unmap_one - perhaps this can be done incrementally.
Signed-off-by Dev Jain <[email protected]>
---
 include/linux/mm.h   | 11 +++++++++++
 include/linux/rmap.h | 27 ++++++++++++++++++++-------
 2 files changed, 31 insertions(+), 7 deletions(-)
include/linux/mm.h +0 -0
index 31e27ff6a35fa..0b77329cf57a4 100644
@@ -243,6 +243,17 @@ static inline unsigned long folio_page_idx(const struct folio *folio,
 	return page - &folio->page;
 }
 

include/linux/mm.h static inline unsigned long folio_page_idx(const struct folio *folio, · line 255

Quoted 10 lines

include/linux/mm.h +10 -0

+static __always_inline void folio_clear_pages_anon_exclusive(struct page *page,
+		unsigned long nr_pages)
+{
+	for (;;) {
+		ClearPageAnonExclusive(page);
+		if (--nr_pages == 0)
+			break;
+		++page;
+	}
+}

Something called folio that doesn't consume a folio is odd. I'd prefer we don't add this.

Is there a chance to simply change __folio_try_share_anon_rmap, so we get a single loop inline?

include/linux/rmap.h +10 ~~-11~~

index 8dc0871e5f00..5a1c874b2112 100644
@@ -708,16 +708,13 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio,
 static __always_inline int __folio_try_share_anon_rmap(struct folio *folio,
                struct page *page, int nr_pages, enum pgtable_level level)
 {
+       /* device private folios cannot get pinned via GUP. */
+       const bool pinnable = !folio_is_device_private(folio);
+
        VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);
        VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio);
        __folio_rmap_sanity_checks(folio, page, nr_pages, level);
 
-       /* device private folios cannot get pinned via GUP. */
-       if (unlikely(folio_is_device_private(folio))) {
-               ClearPageAnonExclusive(page);
-               return 0;
-       }
-
        /*
         * We have to make sure that when we clear PageAnonExclusive, that
         * the page is not pinned and that concurrent GUP-fast won't succeed in

@@ -760,19 +757,21 @@ static __always_inline int __folio_try_share_anon_rmap(struct folio *folio,
         * so we use explicit ones here.
         */
 
-       /* Paired with the memory barrier in try_grab_folio(). */
-       if (IS_ENABLED(CONFIG_HAVE_GUP_FAST))
-               smp_mb();
+       if (pinnable) {
+               /* Paired with the memory barrier in try_grab_folio(). */
+               if (IS_ENABLED(CONFIG_HAVE_GUP_FAST))
+                       smp_mb();
 
-       if (unlikely(folio_maybe_dma_pinned(folio)))
-               return -EBUSY;
+               if (unlikely(folio_maybe_dma_pinned(folio)))
+                       return -EBUSY;
+       }
        ClearPageAnonExclusive(page);

include/linux/rmap.h static __always_inline int __folio_try_share_anon_rmap(struct folio *folio, · line 777

Quoted 10 lines

include/linux/rmap.h +1 -1

 
        /*
         * This is conceptually a smp_wmb() paired with the smp_rmb() in
         * gup_must_unshare().
         */
-       if (IS_ENABLED(CONFIG_HAVE_GUP_FAST))
+       if (pinnable && IS_ENABLED(CONFIG_HAVE_GUP_FAST))
                smp_mb__after_atomic();
        return 0;

Okay this is better! I'll do this.

Dev Jain

author

2026-05-06 09:46 UTC #35 lore raw

PATCH v3 9/9 mm/rmap: enable batch unmapping of anonymous folios

In reply to Dev Jain

Enable batch clearing of ptes, and batch swap setting of ptes for anon folio unmapping.

Processing all ptes of a large folio in one go helps us batch across atomics (add_mm_counter etc), barriers (in the function __folio_try_share_anon_rmap), repeated calls to page_vma_mapped_walk(), to name a few. In general, batching helps us to execute similar code together, making the execution of the program more memory and CPU friendly.

On arm64-contpte, batching also helps us avoid redundant ptep_get() calls and TLB flushes while breaking the contpte mapping.

The handling of anon-exclusivity is very similar to commit cac1db8c3aad ("mm: optimize mprotect() by PTE batching"). Since folio_unmap_pte_batch() won't look at the bits of the underlying page, we need to process sub-batches of ptes pointing to pages which are same w.r.t exclusivity, and batch set only those ptes to swap ptes in one go. Hence export page_anon_exclusive_sub_batch() to internal.h and reuse it.

arch_unmap_one() is only defined for sparc64; I am not comfortable regarding the nuances between retrieving the pfn from pte_pfn() or from (paddr = pte_val(oldpte) & _PAGE_PADDR_4V).

(And, pte_next_pfn() can't even be called from arch_unmap_one() because that file does not include pgtable.h) So just disable the "sparc64-anon-swapbacked" case for now.

We need to take care of rmap accounting (folio_remove_rmap_ptes) and reference accounting (folio_put_refs) when anon folio unmap succeeds. In case we partially batch the large folio and fail, we need to correctly do the accounting for pages which were successfully unmapped. So, put this accounting code in __unmap_anon_folio() itself, instead of doing some horrible goto jumping at the callsite of unmap_anon_folio().

Add a comment at relevant places to say that we are on a device-exclusive entry and not a present entry.

If the batch length is less than the number of pages in the folio, then we must skip over this batch.

The page_vma_mapped_walk API ensures this - check_pte() will return true only if any of [pvmw->pfn, pvmw->pfn + nr_pages) is mapped by the pte. There is no pfn underlying a swap pte, so check_pte returns false and we keep skipping until we hit a present pte, which is where we want to start unmapping from next.

Signed-off-by Dev Jain <[email protected]>

---
 mm/internal.h |  26 +++++++
 mm/mprotect.c |  17 -----
 mm/rmap.c     | 188 ++++++++++++++++++++++++++++++++++----------------
 3 files changed, 153 insertions(+), 78 deletions(-)

mm/internal.h +26 -0

index 5a2ddcf68e0b6..65ccb959f6220 100644
@@ -393,6 +393,32 @@ static inline unsigned int folio_pte_batch_flags(struct folio *folio,
 unsigned int folio_pte_batch(struct folio *folio, pte_t *ptep, pte_t pte,
 		unsigned int max_nr);
 
+/**
+ * page_anon_exclusive_sub_batch - Determine length of consecutive exclusive
+ * or maybe shared pages
+ * @start_idx: Starting index of the page array to scan from
+ * @max_len: Maximum length to look at
+ * @first_page: First page of the page array
+ * @expected_anon_exclusive: Whether to look for exclusive or !exclusive pages
+ *
+ * Determines length of consecutive ptes, pointing to pages being the same
+ * w.r.t the PageAnonExclusive bit.
+ *
+ * Context: The ptes point to consecutive pages of the same large folio. The
+ * ptes belong to the same PMD and VMA.
+ */
+static __always_inline int page_anon_exclusive_sub_batch(int start_idx, int max_len,
+		struct page *first_page, bool expected_anon_exclusive)
+{
+	int idx;
+
+	for (idx = start_idx + 1; idx < start_idx + max_len; ++idx) {
+		if (expected_anon_exclusive != PageAnonExclusive(first_page + idx))
+			break;
+	}
+	return idx - start_idx;
+}
+
 /**
  * pte_move_swp_offset - Move the swap entry offset field of a swap pte
  *	 forward or backward by delta

mm/mprotect.c +0 ~~-17~~

index 9cbf932b028cf..949fd7022b5cf 100644
@@ -138,23 +138,6 @@ static __always_inline void prot_commit_flush_ptes(struct vm_area_struct *vma,
 		tlb_flush_pte_range(tlb, addr, nr_ptes * PAGE_SIZE);
 }
 
-/*
- * Get max length of consecutive ptes pointing to PageAnonExclusive() pages or
- * !PageAnonExclusive() pages, starting from start_idx. Caller must enforce
- * that the ptes point to consecutive pages of the same anon large folio.
- */
-static __always_inline int page_anon_exclusive_sub_batch(int start_idx, int max_len,
-		struct page *first_page, bool expected_anon_exclusive)
-{
-	int idx;
-
-	for (idx = start_idx + 1; idx < start_idx + max_len; ++idx) {
-		if (expected_anon_exclusive != PageAnonExclusive(first_page + idx))
-			break;
-	}
-	return idx - start_idx;
-}
-
 /*
  * This function is a result of trying our very best to retain the
  * "avoid the write-fault handler" optimization. In can_change_pte_writable(),

mm/rmap.c +127 ~~-61~~

244 lines not on this page · raw message

Signature · 1 line

2.34.1

David Hildenbrand (Arm)

maintainer

2026-05-11 08:16 UTC #36 lore raw

Re: [PATCH v3 9/9] mm/rmap: enable batch unmapping of anonymous folios

Quoted 38 lines · Dev Jain

Enable batch clearing of ptes, and batch swap setting of ptes for anon folio unmapping.
Processing all ptes of a large folio in one go helps us batch across atomics (add_mm_counter etc), barriers (in the function __folio_try_share_anon_rmap), repeated calls to page_vma_mapped_walk(), to name a few. In general, batching helps us to execute similar code together, making the execution of the program more memory and CPU friendly.
On arm64-contpte, batching also helps us avoid redundant ptep_get() calls and TLB flushes while breaking the contpte mapping.
The handling of anon-exclusivity is very similar to commit cac1db8c3aad ("mm: optimize mprotect() by PTE batching"). Since folio_unmap_pte_batch() won't look at the bits of the underlying page, we need to process sub-batches of ptes pointing to pages which are same w.r.t exclusivity, and batch set only those ptes to swap ptes in one go. Hence export page_anon_exclusive_sub_batch() to internal.h and reuse it.
arch_unmap_one() is only defined for sparc64; I am not comfortable regarding the nuances between retrieving the pfn from pte_pfn() or from (paddr = pte_val(oldpte) & _PAGE_PADDR_4V).
(And, pte_next_pfn() can't even be called from arch_unmap_one() because that file does not include pgtable.h) So just disable the "sparc64-anon-swapbacked" case for now.
We need to take care of rmap accounting (folio_remove_rmap_ptes) and reference accounting (folio_put_refs) when anon folio unmap succeeds. In case we partially batch the large folio and fail, we need to correctly do the accounting for pages which were successfully unmapped. So, put this accounting code in __unmap_anon_folio() itself, instead of doing some horrible goto jumping at the callsite of unmap_anon_folio().
Add a comment at relevant places to say that we are on a device-exclusive entry and not a present entry.

Quoted 9 lines

If the batch length is less than the number of pages in the folio, then we must skip over this batch.
The page_vma_mapped_walk API ensures this - check_pte() will return true only if any of [pvmw->pfn, pvmw->pfn + nr_pages) is mapped by the pte. There is no pfn underlying a swap pte, so check_pte returns false and we keep skipping until we hit a present pte, which is where we want to start unmapping from next.

This patch is doing too much. Please separate the cleanups (e.g., moving stuff into helpers -- that likely should have a ttu_ prefix) from the real deal.

Signature · 3 lines

Cheers,

David

Dev Jain

author

2026-05-12 08:59 UTC #37 lore raw

Re: [PATCH v3 9/9] mm/rmap: enable batch unmapping of anonymous folios

Quoted 49 lines · David Hildenbrand (Arm)

Quoted 38 lines · Dev Jain
Enable batch clearing of ptes, and batch swap setting of ptes for anon folio unmapping.
Processing all ptes of a large folio in one go helps us batch across atomics (add_mm_counter etc), barriers (in the function __folio_try_share_anon_rmap), repeated calls to page_vma_mapped_walk(), to name a few. In general, batching helps us to execute similar code together, making the execution of the program more memory and CPU friendly.
On arm64-contpte, batching also helps us avoid redundant ptep_get() calls and TLB flushes while breaking the contpte mapping.
The handling of anon-exclusivity is very similar to commit cac1db8c3aad ("mm: optimize mprotect() by PTE batching"). Since folio_unmap_pte_batch() won't look at the bits of the underlying page, we need to process sub-batches of ptes pointing to pages which are same w.r.t exclusivity, and batch set only those ptes to swap ptes in one go. Hence export page_anon_exclusive_sub_batch() to internal.h and reuse it.
arch_unmap_one() is only defined for sparc64; I am not comfortable regarding the nuances between retrieving the pfn from pte_pfn() or from (paddr = pte_val(oldpte) & _PAGE_PADDR_4V).
(And, pte_next_pfn() can't even be called from arch_unmap_one() because that file does not include pgtable.h) So just disable the "sparc64-anon-swapbacked" case for now.
We need to take care of rmap accounting (folio_remove_rmap_ptes) and reference accounting (folio_put_refs) when anon folio unmap succeeds. In case we partially batch the large folio and fail, we need to correctly do the accounting for pages which were successfully unmapped. So, put this accounting code in __unmap_anon_folio() itself, instead of doing some horrible goto jumping at the callsite of unmap_anon_folio().
Add a comment at relevant places to say that we are on a device-exclusive entry and not a present entry.
Quoted 9 lines
If the batch length is less than the number of pages in the folio, then we must skip over this batch.
The page_vma_mapped_walk API ensures this - check_pte() will return true only if any of [pvmw->pfn, pvmw->pfn + nr_pages) is mapped by the pte. There is no pfn underlying a swap pte, so check_pte returns false and we keep skipping until we hit a present pte, which is where we want to start unmapping from next.

Quoted 2 lines

This patch is doing too much. Please separate the cleanups (e.g., moving stuff into helpers -- that likely should have a ttu_ prefix) from the real deal.

Okay.

Quoted 2 lines

Andrew Morton

maintainer

2026-05-08 23:38 UTC #38 lore raw

In reply to Dev Jain

Quoted 10 lines · On Wed, 6 May 2026 15:14:55 +0530 Dev Jain <[email protected]> wrote:

Speed up unmapping of anonymous large folios by clearing the ptes, and setting swap ptes, in one go.
...
Performance as measured on a Linux VM on Apple M3 (arm64):
Vanilla - Mean: 37401913 ns, std dev: 12%
Patched - Mean: 17420282 ns, std dev: 11%

Quoted 6 lines

No regression observed on 4K folios.
Performance as measured on bare metal x86:
Vanilla - mean: 54986286 ns, std dev: 1.5%
Patched - mean: 51930795 ns, std dev: 3%

That looks nice.

I'll pass at this time, wait for reviewer input. Most reviewers are jetlagged and exhausted, so a resend might be needed ;)

Saskiko said a few things:

	https://sashiko.dev/#/patchset/[email protected]

Dev Jain

author

2026-05-11 06:21 UTC #39 lore raw

Quoted 19 lines · Andrew Morton

Quoted 10 lines · On Wed, 6 May 2026 15:14:55 +0530 Dev Jain <[email protected]> wrote:
Speed up unmapping of anonymous large folios by clearing the ptes, and setting swap ptes, in one go.
...
Performance as measured on a Linux VM on Apple M3 (arm64):
Vanilla - Mean: 37401913 ns, std dev: 12%
Patched - Mean: 17420282 ns, std dev: 11%
Quoted 6 lines
No regression observed on 4K folios.
Performance as measured on bare metal x86:
Vanilla - mean: 54986286 ns, std dev: 1.5%
Patched - mean: 51930795 ns, std dev: 3%

Quoted 7 lines

That looks nice.
I'll pass at this time, wait for reviewer input. Most reviewers are jetlagged and exhausted, so a resend might be needed ;)
Saskiko said a few things:
	https://sashiko.dev/#/patchset/[email protected]

Patch 2:

"In the original code, failing hugetlb_vma_trylock_write() triggered a goto walk_abort, leaving ret set to true."

That is wrong.

Patch 9:

"Since __HAVE_ARCH_UNMAP_ONE is typically defined without a value on sparc64, __is_defined() will evaluate to 0 because it is primarily designed for Kconfig symbols that explicitly evaluate to 1."

Which is again wrong?

Patch 9:

"What happens to the remaining pages in the batch? Since get_and_clear_ptes() cleared all of them upfront, and the loop aborts early without restoring them, it appears the remaining PTEs are left cleared in the page tables and their references are not released"

Yes this is valid. I did see it on the v2 Sashiko review but misread it : )

When unmap fails for a sub-batch, I need to restore all the cleared ptes, not only those of the sub-batch.

This should work:

mm/rmap.c +5 -8

index fc953f36d4527..e54c15a82c504 100644
@@ -2023,10 +2023,8 @@ static inline bool __unmap_anon_folio_range(struct vm_area_struct *vma, struct f
 	swp_entry_t entry = page_swap_entry(subpage);
 	struct mm_struct *mm = vma->vm_mm;
 
-	if (folio_dup_swap_pages(folio, subpage, nr_pages) < 0) {
-		set_ptes(mm, address, ptep, pteval, nr_pages);
+	if (folio_dup_swap_pages(folio, subpage, nr_pages) < 0)
 		return false;
-	}
 
 	/*
 	 * arch_unmap_one() is expected to be a NOP on

@@ -2036,16 +2034,13 @@ static inline bool __unmap_anon_folio_range(struct vm_area_struct *vma, struct f
 	if (arch_unmap_one(mm, vma, address, pteval) < 0) {
 		VM_WARN_ON(nr_pages != 1);
 		folio_put_swap_pages(folio, subpage, nr_pages);
-		set_pte_at(mm, address, ptep, pteval);
 		return false;
 	}
 
 	/* See folio_try_share_anon_rmap(): clear PTE first. */
-	if (anon_exclusive && folio_try_share_anon_rmap_ptes(folio, subpage, nr_pages)) {
+	if (anon_exclusive && folio_try_share_anon_rmap_ptes(folio, subpage, nr_pages))
 		folio_put_swap_pages(folio, subpage, nr_pages);
-		set_ptes(mm, address, ptep, pteval, nr_pages);
 		return false;
-	}
 
 	if (list_empty(&mm->mmlist)) {
 		spin_lock(&mmlist_lock);

@@ -2075,8 +2070,10 @@ static inline bool unmap_anon_folio_range(struct vm_area_struct *vma, struct fol
 						    first_page, expected_anon_exclusive);
 		ret = __unmap_anon_folio_range(vma, folio, first_page + sub_batch_idx,
 					       address, ptep, pteval, len, expected_anon_exclusive);
-		if (!ret)
+		if (!ret) {
+			set_ptes(vma->vm_mm, address, ptep, pteval, nr_pages);
 			return ret;
+		}
 
 		nr_pages -= len;
 		if (!nr_pages)

Quoted 2 lines