Skip to contentSkip to conversation

mm: use per-VMA lock in __access_remote_vm for improved monitoring reliability

RFC PATCH In review Rik van Riel 5 participants 36 messages Jul 24 – Sep 11

Subsystems (12)
Memory management x86 Kselftest BPF +8 more subsystems
lore
Share & tools
lore

Rik van Riel

author reviewer
#0 lore raw

v4 cover mm: use per-VMA lock in __access_remote_vm for improved monitoring reliability

__access_remote_vm() holds mmap_read_lock() for the whole transfer. On large machines, with large multi-threaded applications, the mmap_lock is often contended, leading to things like reads of /proc/PID/cmdline stalling.

This results in system monitoring tools getting stuck, right when system information would be the most helpful.

That problem can be avoided by having __access_remote_vm() access user memory under the per-VMA lock. The old code also looks up the VMA for every page accessed, and walks the page tables for every page inside a large folio. Cleaning that up speeds things up nicely.

Using code that does not include its own lock dance, and does not look up the VMA twice results in a modest speedup.

	read size   baseline    per-VMA     throughput
	8 B          201.7 ns    170.6 ns    ~15% faster
	4 KB         495 ns      442 ns      +12% (8266 -> 9265 MB/s)
	64 KB       4115 ns     3355 ns      +23% (15927 -> 19536 MB/s)
	1 MB       75902 ns    63641 ns      +19% (13815 -> 16477 MB/s)

The multi-threaded reader-versus-writer contention that motivates the lock is not measured.

The rest of the series builds on this to batch per-VMA GUP. Patches 9-12 have follow_page_mask() report a contiguous PTE-mapped run of a large folio, so the slow pin_user_pages() path walks it once instead of per page. Other paths that walk a range benefit too.

That batching pays off on PTE-mapped mTHP. With mm/gup_test.c (PIN_LONGTERM_BENCHMARK, the slow pin_user_pages() path) on a 256 MB MADV_HUGEPAGE region in a 4 CPU VM, median get time over 16 iterations:

	gup_test -L -m 256 -n 65536 -r 16 -t
	                      before      after
	64 kB mTHP            3140 us      412 us   (7.6x)
	2 MB THP (control)      78 us       76 us
	4 kB base (control)   3010 us     3042 us

The PMD-mapped 2 MB THP already returns in one step, so only the PTE-mapped large folio case improves.

Reaching a page under the per-VMA lock means looking up the VMA first, which needs to untag the remote address without the mmap lock. The untag masks (x86 LAM, riscv pointer masking) change only while the target is single-threaded, so a remote reader races at most one such change.

Those masks are already read locklessly elsewhere, so patches 1 and 2 add untagged_addr_remote_unlocked() helpers with READ_ONCE()/WRITE_ONCE() annotations. A stale read sees the old or new mask, never a torn value, and the remote untag is best-effort.

With the VMA in hand we need one page from it under the lock the caller already holds. get_user_pages_remote() does not fit: it hard codes the mmap lock and re-derives the VMA internally.

Rather than reimplement GUP in yet another caller, this version follows David Hildenbrand's v2 guidance: start from a proper GUP interface that takes a locked VMA, not an mm, and decide which faults can resolve under the VMA lock.

Patch 3 renames get_user_page_vma_remote() to get_user_page_lookup_vma() to free the name. Patch 5 adds get_user_page_vma(): a simplified __get_user_pages() that walks the tables with follow_page_mask(), faults a missing page in with faultin_page(), and returns it with a reference and the caller's lock still held.

Like __get_user_pages() it runs check_vma_flags(), so callers need not pre-check the VMA.

A VM_IO or VM_PFNMAP VMA is the exception: a COWed page with a struct page is returned normally, while a raw PFN yields -EFAULT so the caller can reach vma->vm_ops->access(). The caller sets FOLL_VMA_LOCK when it holds the per-VMA lock, which reaches the fault code as FAULT_FLAG_VMA_LOCK.

Anything the per-VMA lock cannot finish releases the lock and returns -EAGAIN, so the caller retries under the mmap lock: a dropped fault, a userfaultfd VMA (which assumes current is the faulting task), a hard error, or ->access() memory. faultin_page() reports the retry the same way for both lock types.

__access_remote_vm() then uses the per-VMA lock when an access fits within one VMA, and falls back to the mmap lock for multi-VMA accesses, stack expansion, or an -EAGAIN from get_user_page_vma().

Walking page tables under only the per-VMA lock is safe against both page table freeing and THP collapse. munmap() frees a VMA's page tables under the VMA write lock that our read lock excludes, so they cannot be torn down under the walk.

THP collapse needs neither lock: it retracts a PTE page under the page table lock and frees it by RCU.

follow_page_pte() takes that same lock through pte_offset_map_lock() and rechecks the pmd, so it either walks an intact table or sees the collapsed pmd and faults the page in, while RCU keeps the retracted page valid.

A COWed page in a VM_PFNMAP mapping was previously unreachable through /proc/pid/mem because generic_access_phys() rejects ioremaps of RAM. With get_user_page_vma() returning the struct page directly, that read now succeeds, which the new selftests cover along with the raw PFN path.

Thanks to Suren for pointing out the need, to David for pushing me to just rework the whole area, to Usama for reviewing the arch patches, and Lorenzo for the changelog and cover letter cleanups.

There is a lot more work that could be done in this area, but the patch series is big enough. More changes can come in follow up work.

---
 arch/arm64/kernel/mte.c                        |   2 +-
 arch/riscv/include/asm/mmu_context.h           |   4 +-
 arch/riscv/include/asm/uaccess.h               |  10 +-
 arch/riscv/kernel/process.c                    |  12 +-
 arch/x86/include/asm/mmu_context.h             |   6 +-
 arch/x86/include/asm/uaccess_64.h              |  15 +-
 arch/x86/kernel/process_64.c                   |   4 +-
 arch/x86/kernel/uprobes.c                      |   2 +-
 include/linux/mm.h                             |  32 +--
 include/linux/uaccess.h                        |   7 +
 mm/gup.c                                       | 360 ++++++++++++++++++-----
 mm/internal.h                                  |   8 +-
 mm/memory.c                                    | 379 +++++++++++++++++--------
 mm/rmap.c                                      |   2 +-
 tools/testing/selftests/mm/Makefile            |   1 +
 tools/testing/selftests/mm/mthp_gup_cow_test.c | 213 ++++++++++++++
 tools/testing/selftests/mm/pfnmap.c            |  66 +++++
 tools/testing/selftests/mm/run_vmtests.sh      |   1 +
 18 files changed, 887 insertions(+), 237 deletions(-)

base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df
---
To: Andrew Morton <[email protected]>
To: [email protected]
To: [email protected]
Cc: Dave Hansen <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Suren Baghdasaryan <[email protected]>
Cc: Lorenzo Stoakes <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: "Liam R. Howlett" <[email protected]>
Cc: Mike Rapoport <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Jason Gunthorpe <[email protected]>
Cc: John Hubbard <[email protected]>
Cc: Peter Xu <[email protected]>
Cc: Matthew Wilcox <[email protected]>
Cc: Usama Arif <[email protected]>
Cc: Rik van Riel <[email protected]>

REVIEWERS NOTES:
Series based on the current Linus tree (v7.2-rc5, commit 248951ddc14d). Patches 1-8 add per-VMA lock remote access; patches 9-12 batch per-VMA GUP over PTE-mapped large folios.

Changes since v3 (RFC v3, 2026-07-17):
https://lore.kernel.org/all/[email protected]/

- Address Usama Arif's v3 review:
- p1: correct the Acked-by address to [email protected].
- p2: rewrite the riscv changelog to describe only this patch's change, and replace the false "pmlen is stable" claim with the accurate story (pmlen may change until MM_CONTEXT_LOCK_PMLEN; READ_ONCE avoids a torn value, and the remote untag is best-effort).
- p3: fix the "name space" typo and note "No functional change intended."
- Split the check_vma_flags() ignore-flags change into its own prep patch (patch 4) so the PROT_NONE PFNMAP read hole fix is bisectable, keeping the VM_READ / FOLL_FORCE / arch-key checks for a PFNMAP VMA.
- get_user_page_vma(): add the flush_anon_page()/flush_dcache_page() cache flushes __get_user_pages() does, and reword the faultin_page() kerneldoc to cover the per-VMA lock, not just the mmap lock.
- New patch 7: read remote strings (bpf_copy_from_user_task_str()) under the per-VMA lock, extending the approach to __copy_remote_vm_str().
- Factor __access_remote_vm() and the string reader onto a shared remote_vm_walk() helper, and fix a NUL-termination regression it exposed:
the partial buffer is now terminated on a mid-walk error.
- Fold the mTHP GUP batching (patches 9-12), previously a separate follow-up, into this series; the cover letter now carries its gup_test numbers.
- Correct two changelog claims. The page table lock plus the pmd recheck, not the VMA write lock, guards the walk against concurrent THP collapse; RCU only keeps the retracted PTE page valid, and the VMA write lock excludes munmap(). The x86 untag mask changes while the target is single-threaded, so the lockless read is best-effort, not "set once and never changes."
- Rebase onto the current Linus tree (v7.2-rc5).

author reviewer
#1 lore raw

RFC PATCH v4 1/12 x86/mm: add untagged_addr_remote_unlocked()

__access_remote_vm() reads another process's memory under the mmap lock. On large machines running big multi-threaded applications, that lock is contended between readers and writers: an mmap() or munmap() in one thread stalls readers like /proc/PID/cmdline, /proc/PID/environ, or process_vm_readv(), even though the memory they read is almost always resident and reachable under the per-VMA lock instead.

Looking up the VMA first requires untagging the address. untagged_addr_remote() asserts the mmap lock only because it reads mm->context.untag_mask, which can race with the write in mm_enable_lam().

That mask changes only while the target is single-threaded, when it enables LAM or resets it at exec. The read need not be stable: it is already read locklessly elsewhere, and a remote untag is best-effort.

Add untagged_addr_remote_unlocked() for callers that have not taken the mmap lock, and annotate access to mm->context.untag_mask with READ_ONCE() and WRITE_ONCE() so the existing lockless reads are explicit and KCSAN-clean. untagged_addr_remote() keeps its assertion and shares the same code.

Assisted-by Claude:claude-opus-4.8
---
 arch/x86/include/asm/mmu_context.h |  6 +++---
 arch/x86/include/asm/uaccess_64.h  | 15 ++++++++++++---
 arch/x86/kernel/process_64.c       |  4 ++--
 include/linux/uaccess.h            |  7 +++++++
 4 files changed, 24 insertions(+), 8 deletions(-)
arch/x86/include/asm/mmu_context.h +3 -3
index ef5b507de34e..cee710f64658 100644
@@ -100,18 +100,18 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm)
 static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm)
 {
 	mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask;
-	mm->context.untag_mask = oldmm->context.untag_mask;
+	WRITE_ONCE(mm->context.untag_mask, READ_ONCE(oldmm->context.untag_mask));
 }
 
 #define mm_untag_mask mm_untag_mask
 static inline unsigned long mm_untag_mask(struct mm_struct *mm)
 {
-	return mm->context.untag_mask;
+	return READ_ONCE(mm->context.untag_mask);
 }
 
 static inline void mm_reset_untag_mask(struct mm_struct *mm)
 {
-	mm->context.untag_mask = -1UL;
+	WRITE_ONCE(mm->context.untag_mask, -1UL);
 }
 
 #define arch_pgtable_dma_compat arch_pgtable_dma_compat
arch/x86/include/asm/uaccess_64.h +12 -3
index 20de34cc9aa6..a095165eb648 100644
@@ -39,18 +39,27 @@ static inline unsigned long __untagged_addr(unsigned long addr)
 	(__force __typeof__(addr))__untagged_addr(__addr);		\
 })
 
+/*
+ * mm->context.untag_mask changes only when the target enables LAM or execs,
+ * always single-threaded. An unlocked remote reader can race that; READ_ONCE
+ * keeps the value whole and the untag is best-effort.
+ */
 static inline unsigned long __untagged_addr_remote(struct mm_struct *mm,
 						   unsigned long addr)
 {
-	mmap_assert_locked(mm);
-	return addr & (mm)->context.untag_mask;
+	return addr & READ_ONCE(mm->context.untag_mask);
 }
 
-#define untagged_addr_remote(mm, addr)	({				\
+#define untagged_addr_remote_unlocked(mm, addr)	({			\
 	unsigned long __addr = (__force unsigned long)(addr);		\
 	(__force __typeof__(addr))__untagged_addr_remote(mm, __addr);	\
 })
 
+#define untagged_addr_remote(mm, addr)	({				\
+	mmap_assert_locked(mm);						\
+	untagged_addr_remote_unlocked(mm, addr);			\
+})
+
 #endif
 
 #define valid_user_address(x) \
arch/x86/kernel/process_64.c +2 -2
index d44afbe005bb..9fa659117f38 100644
@@ -814,7 +814,7 @@ static void enable_lam_func(void *__mm)
 static void mm_enable_lam(struct mm_struct *mm)
 {
 	mm->context.lam_cr3_mask = X86_CR3_LAM_U57;
-	mm->context.untag_mask =  ~GENMASK(62, 57);
+	WRITE_ONCE(mm->context.untag_mask, ~GENMASK(62, 57));
 
 	/*
 	 * Even though the process must still be single-threaded at this
@@ -952,7 +952,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)
 #endif
 #ifdef CONFIG_ADDRESS_MASKING
 	case ARCH_GET_UNTAG_MASK:
-		return put_user(task->mm->context.untag_mask,
+		return put_user(mm_untag_mask(task->mm),
 				(unsigned long __user *)arg2);
 	case ARCH_ENABLE_TAGGED_ADDR:
 		return prctl_enable_tagged_addr(task->mm, arg2);
include/linux/uaccess.h +7 -0
index eddbbb65ccc4..7e6e4c89184c 100644
@@ -34,6 +34,13 @@
 })
 #endif
 
+#ifndef untagged_addr_remote_unlocked
+#define untagged_addr_remote_unlocked(mm, addr)	({	\
+	(void)(mm);					\
+	untagged_addr(addr);				\
+})
+#endif
+
 #ifdef masked_user_access_begin
  #define can_do_masked_user_access() 1
 # ifndef masked_user_write_access_begin

base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df

Signature · 1 line
2.53.0-Meta
reviewer Reviewed-by on v4 1/12
#2 lore raw

Re: [PATCH RFC v4 01/12] x86/mm: add untagged_addr_remote_unlocked()

Quoted 22 lines · Rik van Riel

__access_remote_vm() reads another process's memory under the mmap lock. On large machines running big multi-threaded applications, that lock is contended between readers and writers: an mmap() or munmap() in one thread stalls readers like /proc/PID/cmdline, /proc/PID/environ, or process_vm_readv(), even though the memory they read is almost always resident and reachable under the per-VMA lock instead.

Looking up the VMA first requires untagging the address. untagged_addr_remote() asserts the mmap lock only because it reads mm->context.untag_mask, which can race with the write in mm_enable_lam().

That mask changes only while the target is single-threaded, when it enables LAM or resets it at exec. The read need not be stable: it is already read locklessly elsewhere, and a remote untag is best-effort.

Add untagged_addr_remote_unlocked() for callers that have not taken the mmap lock, and annotate access to mm->context.untag_mask with READ_ONCE() and WRITE_ONCE() so the existing lockless reads are explicit and KCSAN-clean. untagged_addr_remote() keeps its assertion and shares the same code.

Quoted 3 lines
Assisted-by Claude:claude-opus-4.8
Quoted 113 lines
---
 arch/x86/include/asm/mmu_context.h |  6 +++---
 arch/x86/include/asm/uaccess_64.h  | 15 ++++++++++++---
 arch/x86/kernel/process_64.c       |  4 ++--
 include/linux/uaccess.h            |  7 +++++++
 4 files changed, 24 insertions(+), 8 deletions(-)
arch/x86/include/asm/mmu_context.h +3 -3
index ef5b507de34e..cee710f64658 100644
@@ -100,18 +100,18 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm)
 static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm)
 {
        mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask;
-       mm->context.untag_mask = oldmm->context.untag_mask;
+       WRITE_ONCE(mm->context.untag_mask, READ_ONCE(oldmm->context.untag_mask));
 }
 
 #define mm_untag_mask mm_untag_mask
 static inline unsigned long mm_untag_mask(struct mm_struct *mm)
 {
-       return mm->context.untag_mask;
+       return READ_ONCE(mm->context.untag_mask);
 }
 
 static inline void mm_reset_untag_mask(struct mm_struct *mm)
 {
-       mm->context.untag_mask = -1UL;
+       WRITE_ONCE(mm->context.untag_mask, -1UL);
 }
 
 #define arch_pgtable_dma_compat arch_pgtable_dma_compat
arch/x86/include/asm/uaccess_64.h +12 -3
index 20de34cc9aa6..a095165eb648 100644
@@ -39,18 +39,27 @@ static inline unsigned long __untagged_addr(unsigned long addr)
        (__force __typeof__(addr))__untagged_addr(__addr);              \
 })
 
+/*
+ * mm->context.untag_mask changes only when the target enables LAM or execs,
+ * always single-threaded. An unlocked remote reader can race that; READ_ONCE
+ * keeps the value whole and the untag is best-effort.
+ */
 static inline unsigned long __untagged_addr_remote(struct mm_struct *mm,
                                                   unsigned long addr)
 {
-       mmap_assert_locked(mm);
-       return addr & (mm)->context.untag_mask;
+       return addr & READ_ONCE(mm->context.untag_mask);
 }
 
-#define untagged_addr_remote(mm, addr) ({                              \
+#define untagged_addr_remote_unlocked(mm, addr)        ({                      \
        unsigned long __addr = (__force unsigned long)(addr);           \
        (__force __typeof__(addr))__untagged_addr_remote(mm, __addr);   \
 })
 
+#define untagged_addr_remote(mm, addr) ({                              \
+       mmap_assert_locked(mm);                                         \
+       untagged_addr_remote_unlocked(mm, addr);                        \
+})
+
 #endif
 
 #define valid_user_address(x) \
arch/x86/kernel/process_64.c +2 -2
index d44afbe005bb..9fa659117f38 100644
@@ -814,7 +814,7 @@ static void enable_lam_func(void *__mm)
 static void mm_enable_lam(struct mm_struct *mm)
 {
        mm->context.lam_cr3_mask = X86_CR3_LAM_U57;
-       mm->context.untag_mask =  ~GENMASK(62, 57);
+       WRITE_ONCE(mm->context.untag_mask, ~GENMASK(62, 57));
 
        /*
         * Even though the process must still be single-threaded at this
@@ -952,7 +952,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)
 #endif
 #ifdef CONFIG_ADDRESS_MASKING
        case ARCH_GET_UNTAG_MASK:
-               return put_user(task->mm->context.untag_mask,
+               return put_user(mm_untag_mask(task->mm),
                                (unsigned long __user *)arg2);
        case ARCH_ENABLE_TAGGED_ADDR:
                return prctl_enable_tagged_addr(task->mm, arg2);
include/linux/uaccess.h +7 -0
index eddbbb65ccc4..7e6e4c89184c 100644
@@ -34,6 +34,13 @@
 })
 #endif
 
+#ifndef untagged_addr_remote_unlocked
+#define untagged_addr_remote_unlocked(mm, addr)        ({      \
+       (void)(mm);                                     \
+       untagged_addr(addr);                            \
+})
+#endif
+
 #ifdef masked_user_access_begin
  #define can_do_masked_user_access() 1
 # ifndef masked_user_write_access_begin

base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df --
2.53.0-Meta

reviewer
#3 lore raw

Re: [PATCH RFC v4 01/12] x86/mm: add untagged_addr_remote_unlocked()

In reply to Rik van Riel

On 7/24/26 3:29 PM, Rik van Riel wrote:
...

arch/x86/include/asm/uaccess_64.h static inline unsigned long __untagged_addr(unsigned long addr) · line 62

Quoted 11 lines
arch/x86/include/asm/uaccess_64.h +6 -1
-#define untagged_addr_remote(mm, addr)	({				\
+#define untagged_addr_remote_unlocked(mm, addr)	({			\
 	unsigned long __addr = (__force unsigned long)(addr);		\
 	(__force __typeof__(addr))__untagged_addr_remote(mm, __addr);	\
 })
 
+#define untagged_addr_remote(mm, addr)	({				\
+	mmap_assert_locked(mm);						\
+	untagged_addr_remote_unlocked(mm, addr);			\
+})
+

With this diff, mm is now getting evaluated twice.

You could instead do this:

	#define untagged_addr_remote(mm, addr)	({		\
		struct mm_struct *__mm = (mm);			\
		mmap_assert_locked(__mm);			\
		untagged_addr_remote_unlocked(__mm, addr);	\
	})

thanks,

Signature · 1 line
John Hubbard
author reviewer
#4 lore raw

RFC PATCH v4 2/12 riscv/mm: add untagged_addr_remote_unlocked()

In reply to Rik van Riel

__access_remote_vm() reads another task's memory under the mmap lock. That lock is contended on large multi-threaded workloads, where an mmap() or munmap() stalls readers like /proc/PID/cmdline or process_vm_readv() even though the target memory is resident and reachable under the per-VMA lock.

Reaching it under the per-VMA lock means looking up the VMA first, which requires untagging the remote address without the mmap lock. riscv's untagged_addr_remote() asserts that lock and has no unlocked variant; the generic untagged_addr_remote_unlocked() falls back to untagged_addr(), which masks against current->mm, not the target mm.

Add untagged_addr_remote_unlocked(), which masks @addr against the target mm without asserting the lock, and redefine untagged_addr_remote() on top of it so it keeps the assertion.

The mask width is mm->context.pmlen, written while a program is single threaded, and frozen with MM_CONTEXT_LOCK_PMLEN once it spawns a second thread.

That read need not be stable: pmlen is already read locklessly by untagged_addr() and mm_untag_mask(), and a remote untag is best-effort. Annotate the lockless reads with READ_ONCE() and the writes with WRITE_ONCE(), so a reader sees one whole value and the accesses are KCSAN-clean.

Assisted-by Claude:claude-opus-4.8
---
 arch/riscv/include/asm/mmu_context.h |  4 ++--
 arch/riscv/include/asm/uaccess.h     | 10 +++++++---
 arch/riscv/kernel/process.c          | 12 +++++++-----
 3 files changed, 16 insertions(+), 10 deletions(-)
arch/riscv/include/asm/mmu_context.h +2 -2
index dbf27a78df6c..3ce16796e5a2 100644
@@ -21,7 +21,7 @@ static inline void activate_mm(struct mm_struct *prev,
 			       struct mm_struct *next)
 {
 #ifdef CONFIG_RISCV_ISA_SUPM
-	next->context.pmlen = 0;
+	WRITE_ONCE(next->context.pmlen, 0);
 #endif
 	switch_mm(prev, next, NULL);
 }
@@ -44,7 +44,7 @@ DECLARE_STATIC_KEY_FALSE(use_asid_allocator);
 #define mm_untag_mask mm_untag_mask
 static inline unsigned long mm_untag_mask(struct mm_struct *mm)
 {
-	return -1UL >> mm->context.pmlen;
+	return -1UL >> READ_ONCE(mm->context.pmlen);
 }
 #endif
 
arch/riscv/include/asm/uaccess.h +7 -3
index 5d4ec15584cf..53806e0f7dcf 100644
@@ -16,7 +16,7 @@
 static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigned long addr)
 {
 	if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SUPM)) {
-		u8 pmlen = mm->context.pmlen;
+		u8 pmlen = READ_ONCE(mm->context.pmlen);
 
 		/* Virtual addresses are sign-extended; physical addresses are zero-extended. */
 		if (IS_ENABLED(CONFIG_MMU))
@@ -33,12 +33,16 @@ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigne
 	(__force __typeof__(addr))__untagged_addr_remote(current->mm, __addr);	\
 })
 
-#define untagged_addr_remote(mm, addr) ({					\
+#define untagged_addr_remote_unlocked(mm, addr) ({				\
 	unsigned long __addr = (__force unsigned long)(addr);			\
-	mmap_assert_locked(mm);							\
 	(__force __typeof__(addr))__untagged_addr_remote(mm, __addr);		\
 })
 
+#define untagged_addr_remote(mm, addr) ({					\
+	mmap_assert_locked(mm);							\
+	untagged_addr_remote_unlocked(mm, addr);				\
+})
+
 #define access_ok(addr, size) likely(__access_ok(untagged_addr(addr), size))
 #else
 #define untagged_addr(addr) (addr)
arch/riscv/kernel/process.c +7 -5
index b2df7f72241a..6ae7552fed09 100644
@@ -357,13 +357,15 @@ long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg)
 	if (mmap_write_lock_killable(mm))
 		return -EINTR;
 
-	if (test_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags) && mm->context.pmlen != pmlen) {
-		mmap_write_unlock(mm);
-		return -EBUSY;
+	if (test_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags)) {
+		if (READ_ONCE(mm->context.pmlen) != pmlen) {
+			mmap_write_unlock(mm);
+			return -EBUSY;
+		}
 	}
 
 	envcfg_update_bits(task, ENVCFG_PMM, pmm);
-	mm->context.pmlen = pmlen;
+	WRITE_ONCE(mm->context.pmlen, pmlen);
 
 	mmap_write_unlock(mm);
 
@@ -394,7 +396,7 @@ long get_tagged_addr_ctrl(struct task_struct *task)
 		break;
 	}
 
-	if (task->mm->context.pmlen)
+	if (READ_ONCE(task->mm->context.pmlen))
 		ret |= PR_TAGGED_ADDR_ENABLE;
 
 	return ret;
Signature · 1 line
2.53.0-Meta
reviewer Reviewed-by on v4 2/12
#5 lore raw

Re: [PATCH RFC v4 02/12] riscv/mm: add untagged_addr_remote_unlocked()

Quoted 20 lines · Rik van Riel

__access_remote_vm() reads another task's memory under the mmap lock. That lock is contended on large multi-threaded workloads, where an mmap() or munmap() stalls readers like /proc/PID/cmdline or process_vm_readv() even though the target memory is resident and reachable under the per-VMA lock.

Reaching it under the per-VMA lock means looking up the VMA first, which requires untagging the remote address without the mmap lock. riscv's untagged_addr_remote() asserts that lock and has no unlocked variant; the generic untagged_addr_remote_unlocked() falls back to untagged_addr(), which masks against current->mm, not the target mm.

Add untagged_addr_remote_unlocked(), which masks @addr against the target mm without asserting the lock, and redefine untagged_addr_remote() on top of it so it keeps the assertion.

The mask width is mm->context.pmlen, written while a program is single threaded, and frozen with MM_CONTEXT_LOCK_PMLEN once it spawns a second thread.

Quoted 8 lines

That read need not be stable: pmlen is already read locklessly by untagged_addr() and mm_untag_mask(), and a remote untag is best-effort. Annotate the lockless reads with READ_ONCE() and the writes with WRITE_ONCE(), so a reader sees one whole value and the accesses are KCSAN-clean.

Assisted-by Claude:claude-opus-4.8
Quoted 97 lines
---
 arch/riscv/include/asm/mmu_context.h |  4 ++--
 arch/riscv/include/asm/uaccess.h     | 10 +++++++---
 arch/riscv/kernel/process.c          | 12 +++++++-----
 3 files changed, 16 insertions(+), 10 deletions(-)
arch/riscv/include/asm/mmu_context.h +2 -2
index dbf27a78df6c..3ce16796e5a2 100644
@@ -21,7 +21,7 @@ static inline void activate_mm(struct mm_struct *prev,
                               struct mm_struct *next)
 {
 #ifdef CONFIG_RISCV_ISA_SUPM
-       next->context.pmlen = 0;
+       WRITE_ONCE(next->context.pmlen, 0);
 #endif
        switch_mm(prev, next, NULL);
 }
@@ -44,7 +44,7 @@ DECLARE_STATIC_KEY_FALSE(use_asid_allocator);
 #define mm_untag_mask mm_untag_mask
 static inline unsigned long mm_untag_mask(struct mm_struct *mm)
 {
-       return -1UL >> mm->context.pmlen;
+       return -1UL >> READ_ONCE(mm->context.pmlen);
 }
 #endif
 
arch/riscv/include/asm/uaccess.h +7 -3
index 5d4ec15584cf..53806e0f7dcf 100644
@@ -16,7 +16,7 @@
 static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigned long addr)
 {
        if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SUPM)) {
-               u8 pmlen = mm->context.pmlen;
+               u8 pmlen = READ_ONCE(mm->context.pmlen);
 
                /* Virtual addresses are sign-extended; physical addresses are zero-extended. */
                if (IS_ENABLED(CONFIG_MMU))
@@ -33,12 +33,16 @@ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigne
        (__force __typeof__(addr))__untagged_addr_remote(current->mm, __addr);  \
 })
 
-#define untagged_addr_remote(mm, addr) ({                                      \
+#define untagged_addr_remote_unlocked(mm, addr) ({                             \
        unsigned long __addr = (__force unsigned long)(addr);                   \
-       mmap_assert_locked(mm);                                                 \
        (__force __typeof__(addr))__untagged_addr_remote(mm, __addr);           \
 })
 
+#define untagged_addr_remote(mm, addr) ({                                      \
+       mmap_assert_locked(mm);                                                 \
+       untagged_addr_remote_unlocked(mm, addr);                                \
+})
+
 #define access_ok(addr, size) likely(__access_ok(untagged_addr(addr), size))
 #else
 #define untagged_addr(addr) (addr)
arch/riscv/kernel/process.c +7 -6
index b2df7f72241a..6ae7552fed09 100644
@@ -357,13 +357,15 @@ long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg)
        if (mmap_write_lock_killable(mm))
                return -EINTR;
 
-       if (test_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags) && mm->context.pmlen != pmlen) {
-               mmap_write_unlock(mm);
-               return -EBUSY;
+       if (test_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags)) {
+               if (READ_ONCE(mm->context.pmlen) != pmlen) {
+                       mmap_write_unlock(mm);
+                       return -EBUSY;
+               }
        }
 
        envcfg_update_bits(task, ENVCFG_PMM, pmm);
-       mm->context.pmlen = pmlen;
+       WRITE_ONCE(mm->context.pmlen, pmlen);
 
        mmap_write_unlock(mm);
 
@@ -394,7 +396,7 @@ long get_tagged_addr_ctrl(struct task_struct *task)
                break;
        }
 
-       if (task->mm->context.pmlen)
+       if (READ_ONCE(task->mm->context.pmlen))
                ret |= PR_TAGGED_ADDR_ENABLE;
 
        return ret;
--

2.53.0-Meta

reviewer
#6 lore raw

Re: [PATCH RFC v4 02/12] riscv/mm: add untagged_addr_remote_unlocked()

In reply to Rik van Riel

On 7/24/26 3:29 PM, Rik van Riel wrote:
...

arch/riscv/kernel/process.c long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg) · line 361

Quoted 12 lines

index b2df7f72241a..6ae7552fed09 100644

arch/riscv/kernel/process.c +2 -3
@@ -357,13 +357,15 @@ long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg)
 	if (mmap_write_lock_killable(mm))
 		return -EINTR;
 
-	if (test_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags) && mm->context.pmlen != pmlen) {
-		mmap_write_unlock(mm);
-		return -EBUSY;
+	if (test_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags)) {
+		if (READ_ONCE(mm->context.pmlen) != pmlen) {

I think you should remove this misleading READ_ONCE, because it implies a race that is not there. The code is happening under mmap_write_lock_*(), and so does the only writer to pmlen (below), so they cannot race with each other.

Quoted 12 lines
arch/riscv/kernel/process.c +4 -1
+			mmap_write_unlock(mm);
+			return -EBUSY;
+		}
 	}
 
 	envcfg_update_bits(task, ENVCFG_PMM, pmm);
-	mm->context.pmlen = pmlen;
+	WRITE_ONCE(mm->context.pmlen, pmlen);
 
 	mmap_write_unlock(mm);
 

arch/riscv/kernel/process.c long get_tagged_addr_ctrl(struct task_struct *task) · line 402

Quoted 8 lines
arch/riscv/kernel/process.c +1 -1
@@ -394,7 +396,7 @@ long get_tagged_addr_ctrl(struct task_struct *task)
 		break;
 	}
 
-	if (task->mm->context.pmlen)
+	if (READ_ONCE(task->mm->context.pmlen))
 		ret |= PR_TAGGED_ADDR_ENABLE;
 
 	return ret;

thanks,

Signature · 1 line
John Hubbard
author reviewer
#7 lore raw

RFC PATCH v4 3/12 mm: rename get_user_page_vma_remote() to get_user_page_lookup_vma()

In reply to Rik van Riel

get_user_page_vma_remote() faults in the page at @addr in a remote mm and also looks up the VMA that covers it, handing both back to the caller.

This cleans up the namespace for adding a get_user_page_vma() variant where the caller already has the vma.

No functional change intended.

Assisted-by Claude:claude-opus-4.8
---
 arch/arm64/kernel/mte.c   | 2 +-
 arch/x86/kernel/uprobes.c | 2 +-
 include/linux/mm.h        | 2 +-
 mm/memory.c               | 4 ++--
 mm/rmap.c                 | 2 +-
 5 files changed, 6 insertions(+), 6 deletions(-)
arch/arm64/kernel/mte.c +1 -1
index 1a9aad6ef22a..7a6ecc3d9294 100644
@@ -459,7 +459,7 @@ static int __access_remote_tags(struct mm_struct *mm, unsigned long addr,
 		struct vm_area_struct *vma;
 		unsigned long tags, offset;
 		void *maddr;
-		struct page *page = get_user_page_vma_remote(mm, addr,
+		struct page *page = get_user_page_lookup_vma(mm, addr,
 							     gup_flags, &vma);
 		struct folio *folio;
 
arch/x86/kernel/uprobes.c +1 -1
index 3af979fb41d3..329efac0cfb3 100644
@@ -1036,7 +1036,7 @@ static int copy_from_vaddr(struct mm_struct *mm, unsigned long vaddr, void *dst,
 	struct vm_area_struct *vma;
 	struct page *page;
 
-	page = get_user_page_vma_remote(mm, vaddr, gup_flags, &vma);
+	page = get_user_page_lookup_vma(mm, vaddr, gup_flags, &vma);
 	if (IS_ERR(page))
 		return PTR_ERR(page);
 	uprobe_copy_from_page(page, vaddr, dst, len);
include/linux/mm.h +1 -1
index 485df9c2dbdd..24ead14b4790 100644
@@ -3238,7 +3238,7 @@ long pin_user_pages_remote(struct mm_struct *mm,
 /*
  * Retrieves a single page alongside its VMA. Does not support FOLL_NOWAIT.
  */
-static inline struct page *get_user_page_vma_remote(struct mm_struct *mm,
+static inline struct page *get_user_page_lookup_vma(struct mm_struct *mm,
 						    unsigned long addr,
 						    int gup_flags,
 						    struct vm_area_struct **vmap)
mm/memory.c +2 -2
index ff338c2abe92..3b86eeaf084f 100644
@@ -7039,7 +7039,7 @@ static int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
 		void *maddr;
 		struct folio *folio;
 		struct vm_area_struct *vma = NULL;
-		struct page *page = get_user_page_vma_remote(mm, addr,
+		struct page *page = get_user_page_lookup_vma(mm, addr,
 							     gup_flags, &vma);
 
 		if (IS_ERR(page)) {
@@ -7167,7 +7167,7 @@ static int __copy_remote_vm_str(struct mm_struct *mm, unsigned long addr,
 		struct page *page;
 		struct vm_area_struct *vma = NULL;
 
-		page = get_user_page_vma_remote(mm, addr, gup_flags, &vma);
+		page = get_user_page_lookup_vma(mm, addr, gup_flags, &vma);
 		if (IS_ERR(page)) {
 			/*
 			 * Treat as a total failure for now until we decide how
mm/rmap.c +1 -1
index 1c77d5dc06e9..b36f2e219b8f 100644
@@ -2838,7 +2838,7 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr,
 	 * (non-device-exclusive) PTE and issue a MMU_NOTIFY_EXCLUSIVE.
 	 */
 retry:
-	page = get_user_page_vma_remote(mm, addr,
+	page = get_user_page_lookup_vma(mm, addr,
 					FOLL_GET | FOLL_WRITE | FOLL_SPLIT_PMD,
 					&vma);
 	if (IS_ERR(page))
Signature · 1 line
2.53.0-Meta
reviewer Reviewed-by on v4 3/12
#8 lore raw

Re: [PATCH RFC v4 03/12] mm: rename get_user_page_vma_remote() to get_user_page_lookup_vma()

Quoted 6 lines · Rik van Riel

get_user_page_vma_remote() faults in the page at @addr in a remote mm and also looks up the VMA that covers it, handing both back to the caller.

This cleans up the namespace for adding a get_user_page_vma() variant where the caller already has the vma.

nit: Before explaining the results we usually state what this patch is doing. I know the title alread says "rename get_user_page_vma_remote() to get_user_page_lookup_vma()" but it seems appropriate to say that here too.

Quoted 6 lines

No functional change intended.

Assisted-by Claude:claude-opus-4.8
Quoted 85 lines
---
 arch/arm64/kernel/mte.c   | 2 +-
 arch/x86/kernel/uprobes.c | 2 +-
 include/linux/mm.h        | 2 +-
 mm/memory.c               | 4 ++--
 mm/rmap.c                 | 2 +-
 5 files changed, 6 insertions(+), 6 deletions(-)
arch/arm64/kernel/mte.c +1 -1
index 1a9aad6ef22a..7a6ecc3d9294 100644
@@ -459,7 +459,7 @@ static int __access_remote_tags(struct mm_struct *mm, unsigned long addr,
                struct vm_area_struct *vma;
                unsigned long tags, offset;
                void *maddr;
-               struct page *page = get_user_page_vma_remote(mm, addr,
+               struct page *page = get_user_page_lookup_vma(mm, addr,
                                                             gup_flags, &vma);
                struct folio *folio;
 
arch/x86/kernel/uprobes.c +1 -1
index 3af979fb41d3..329efac0cfb3 100644
@@ -1036,7 +1036,7 @@ static int copy_from_vaddr(struct mm_struct *mm, unsigned long vaddr, void *dst,
        struct vm_area_struct *vma;
        struct page *page;
 
-       page = get_user_page_vma_remote(mm, vaddr, gup_flags, &vma);
+       page = get_user_page_lookup_vma(mm, vaddr, gup_flags, &vma);
        if (IS_ERR(page))
                return PTR_ERR(page);
        uprobe_copy_from_page(page, vaddr, dst, len);
include/linux/mm.h +1 -1
index 485df9c2dbdd..24ead14b4790 100644
@@ -3238,7 +3238,7 @@ long pin_user_pages_remote(struct mm_struct *mm,
 /*
  * Retrieves a single page alongside its VMA. Does not support FOLL_NOWAIT.
  */
-static inline struct page *get_user_page_vma_remote(struct mm_struct *mm,
+static inline struct page *get_user_page_lookup_vma(struct mm_struct *mm,
                                                    unsigned long addr,
                                                    int gup_flags,
                                                    struct vm_area_struct **vmap)
mm/memory.c +2 -2
index ff338c2abe92..3b86eeaf084f 100644
@@ -7039,7 +7039,7 @@ static int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
                void *maddr;
                struct folio *folio;
                struct vm_area_struct *vma = NULL;
-               struct page *page = get_user_page_vma_remote(mm, addr,
+               struct page *page = get_user_page_lookup_vma(mm, addr,
                                                             gup_flags, &vma);
 
                if (IS_ERR(page)) {
@@ -7167,7 +7167,7 @@ static int __copy_remote_vm_str(struct mm_struct *mm, unsigned long addr,
                struct page *page;
                struct vm_area_struct *vma = NULL;
 
-               page = get_user_page_vma_remote(mm, addr, gup_flags, &vma);
+               page = get_user_page_lookup_vma(mm, addr, gup_flags, &vma);
                if (IS_ERR(page)) {
                        /*
                         * Treat as a total failure for now until we decide how
mm/rmap.c +1 -2
index 1c77d5dc06e9..b36f2e219b8f 100644
@@ -2838,7 +2838,7 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr,
         * (non-device-exclusive) PTE and issue a MMU_NOTIFY_EXCLUSIVE.
         */
 retry:
-       page = get_user_page_vma_remote(mm, addr,
+       page = get_user_page_lookup_vma(mm, addr,
                                        FOLL_GET | FOLL_WRITE | FOLL_SPLIT_PMD,
                                        &vma);
        if (IS_ERR(page))
--

2.53.0-Meta

author reviewer
#9 lore raw

RFC PATCH v4 4/12 mm/gup: let check_vma_flags() ignore selected VMA flags

In reply to Rik van Riel

check_vma_flags() rejects a VMA whose flags gup cannot handle. A caller that wants to reach a COWed page in a VM_IO or VM_PFNMAP VMA, where that page does have a struct page, cannot express the exception and has to repeat the flag test itself.

Add an ignore_flags argument that check_vma_flags() clears from its local copy of vma->vm_flags before the flag checks. A caller can drop a single rejection this way while keeping every other check. All current callers pass 0.

This prepares for get_user_page_vma(), which passes VM_IO | VM_PFNMAP to reach those COWed pages.

No functional change intended.

Assisted-by Claude:claude-opus-4.8
---
 mm/gup.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
mm/gup.c +11 -3
index 0692119b7904..dcece7b5253c 100644
@@ -1197,13 +1197,21 @@ static bool writable_file_mapping_allowed(struct vm_area_struct *vma,
 	return !vma_needs_dirty_tracking(vma);
 }
 
-static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
+static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
+			   vm_flags_t ignore_flags)
 {
 	vm_flags_t vm_flags = vma->vm_flags;
 	int write = (gup_flags & FOLL_WRITE);
 	int foreign = (gup_flags & FOLL_REMOTE);
 	bool vma_anon = vma_is_anonymous(vma);
 
+	/*
+	 * Opt out of the flag checks that read this local copy (the
+	 * VM_IO/VM_PFNMAP gate and the write/read/cow bits); checks that
+	 * re-read vma->vm_flags through helpers are unaffected.
+	 */
+	vm_flags &= ~ignore_flags;
+
 	if (vm_flags & (VM_IO | VM_PFNMAP))
 		return -EFAULT;
 
@@ -1387,7 +1395,7 @@ static long __get_user_pages(struct mm_struct *mm,
 					ret = -ENOMEM;
 					goto out;
 				}
-				if (check_vma_flags(vma, gup_flags)) {
+				if (check_vma_flags(vma, gup_flags, 0)) {
 					ret = -EINVAL;
 					goto out;
 				}
@@ -1408,7 +1416,7 @@ static long __get_user_pages(struct mm_struct *mm,
 				ret = -EFAULT;
 				goto out;
 			}
-			ret = check_vma_flags(vma, gup_flags);
+			ret = check_vma_flags(vma, gup_flags, 0);
 			if (ret)
 				goto out;
 		}
Signature · 1 line
2.53.0-Meta
reviewer Reviewed-by on v4 4/12
#10 lore raw

Re: [PATCH RFC v4 04/12] mm/gup: let check_vma_flags() ignore selected VMA flags

Quoted 11 lines · Rik van Riel

check_vma_flags() rejects a VMA whose flags gup cannot handle. A caller that wants to reach a COWed page in a VM_IO or VM_PFNMAP VMA, where that page does have a struct page, cannot express the exception and has to repeat the flag test itself.

Add an ignore_flags argument that check_vma_flags() clears from its local copy of vma->vm_flags before the flag checks. A caller can drop a single rejection this way while keeping every other check. All current callers pass 0.

Quoted 7 lines

This prepares for get_user_page_vma(), which passes VM_IO | VM_PFNMAP to reach those COWed pages.

No functional change intended.

Assisted-by Claude:claude-opus-4.8
Quoted 53 lines
---
 mm/gup.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
mm/gup.c +11 -4
index 0692119b7904..dcece7b5253c 100644
@@ -1197,13 +1197,21 @@ static bool writable_file_mapping_allowed(struct vm_area_struct *vma,
        return !vma_needs_dirty_tracking(vma);
 }
 
-static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
+static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
+                          vm_flags_t ignore_flags)
 {
        vm_flags_t vm_flags = vma->vm_flags;
        int write = (gup_flags & FOLL_WRITE);
        int foreign = (gup_flags & FOLL_REMOTE);
        bool vma_anon = vma_is_anonymous(vma);
 
+       /*
+        * Opt out of the flag checks that read this local copy (the
+        * VM_IO/VM_PFNMAP gate and the write/read/cow bits); checks that
+        * re-read vma->vm_flags through helpers are unaffected.
+        */
+       vm_flags &= ~ignore_flags;
+
        if (vm_flags & (VM_IO | VM_PFNMAP))
                return -EFAULT;
 
@@ -1387,7 +1395,7 @@ static long __get_user_pages(struct mm_struct *mm,
                                        ret = -ENOMEM;
                                        goto out;
                                }
-                               if (check_vma_flags(vma, gup_flags)) {
+                               if (check_vma_flags(vma, gup_flags, 0)) {
                                        ret = -EINVAL;
                                        goto out;
                                }
@@ -1408,7 +1416,7 @@ static long __get_user_pages(struct mm_struct *mm,
                                ret = -EFAULT;
                                goto out;
                        }
-                       ret = check_vma_flags(vma, gup_flags);
+                       ret = check_vma_flags(vma, gup_flags, 0);
                        if (ret)
                                goto out;
                }
--

2.53.0-Meta

reviewer
#11 lore raw

Re: [PATCH RFC v4 04/12] mm/gup: let check_vma_flags() ignore selected VMA flags

In reply to Rik van Riel
Quoted 26 lines · Rik van Riel

check_vma_flags() rejects a VMA whose flags gup cannot handle. A caller that wants to reach a COWed page in a VM_IO or VM_PFNMAP VMA, where that page does have a struct page, cannot express the exception and has to repeat the flag test itself.

Add an ignore_flags argument that check_vma_flags() clears from its local copy of vma->vm_flags before the flag checks. A caller can drop a single rejection this way while keeping every other check. All current callers pass 0.

This prepares for get_user_page_vma(), which passes VM_IO | VM_PFNMAP to reach those COWed pages.

No functional change intended.

Assisted-by Claude:claude-opus-4.8
---
 mm/gup.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
mm/gup.c +0 -0
index 0692119b7904..dcece7b5253c 100644

mm/gup.c static bool writable_file_mapping_allowed(struct vm_area_struct *vma, · line 1201

Quoted 6 lines
mm/gup.c +2 -1
@@ -1197,13 +1197,21 @@ static bool writable_file_mapping_allowed(struct vm_area_struct *vma,
 	return !vma_needs_dirty_tracking(vma);
 }
 
-static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
+static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
+			   vm_flags_t ignore_flags)

All callers pass zero in this patch, so there is no behavior change yet. The problem is the interface being added.

The same local vm_flags value is used for the VM_IO/VM_PFNMAP gate, VM_WRITE, VM_SHADOW_STACK, VM_READ, VM_MAYREAD, and is_cow_mapping().

An arbitrary mask can therefore alter permission, shadow-stack, and COW validation. For example, masking VM_SHARED changes the result of is_cow_mapping(), while masking VM_SHADOW_STACK removes its write rejection. Checks implemented through helpers still read the original vma->vm_flags, so the meaning of ignore_flags also depends on how each check happens to be implemented.

Perhaps it is best to replace the mask with a single-purpose internal GUP flag, such as FOLL_PFNMAP_COW, and apply it only to the special-mapping gate:

	if (vm_flags & (VM_IO | VM_PFNMAP) &&
	    (!(gup_flags & FOLL_PFNMAP_COW) ||
	     !(vm_flags & VM_PFNMAP)))
		return -EFAULT;

This permits the later caller to examine a VM_PFNMAP mapping while keeping VM_IO-only mappings and every other VMA check unchanged.

thanks,

Signature · 1 line
John Hubbard
author reviewer
#12 lore raw

Re: [PATCH RFC v4 04/12] mm/gup: let check_vma_flags() ignore selected VMA flags

Quoted 5 lines · John Hubbard

Perhaps it is best to replace the mask with a single-purpose internal GUP flag, such as FOLL_PFNMAP_COW, and apply it only to the special-mapping gate:

Quoted 7 lines
	if (vm_flags & (VM_IO | VM_PFNMAP) &&
	    (!(gup_flags & FOLL_PFNMAP_COW) ||
	     !(vm_flags & VM_PFNMAP)))
		return -EFAULT;

This permits the later caller to examine a VM_PFNMAP mapping while keeping VM_IO-only mappings and every other VMA check unchanged.

Nice idea. I will implement that for v5!

Signature · 1 line
All Rights Reversed.
author reviewer
#13 lore raw

RFC PATCH v4 5/12 mm/gup: add get_user_page_vma() to fault in a page under a held lock

In reply to Rik van Riel

__access_remote_vm() needs a single page from a VMA it has already looked up and locked, faulting it in when necessary, under either the mmap lock or the per-VMA lock. get_user_pages_remote() does not fit: it hard codes the mmap lock and re-looks-up the VMA, neither of which is wanted here.

Add get_user_page_vma(), a simplified __get_user_pages() that walks the page tables with follow_page_mask(), faults a missing page in with faultin_page(), and on success returns it with a reference and the caller's lock still held. Like __get_user_pages() it runs check_vma_flags(), so callers need not pre-check the VMA.

A VM_IO/VM_PFNMAP VMA is the exception to that check: it can still hold COWed pages that have a struct page, so follow_page_mask() is allowed to look for one. Memory with no struct page -- a raw PFN, or a present PFN reported as -EEXIST -- is returned as -EFAULT, so the caller can reach it through vma->vm_ops->access().

The caller sets FOLL_VMA_LOCK when it holds the per-VMA lock rather than the mmap lock, which reaches the fault code as FAULT_FLAG_VMA_LOCK.

Anything that cannot complete under the per-VMA lock -- a dropped fault, a userfaultfd VMA (uffd assumes current is the faulting task), a hard error, or ->access() memory -- releases the lock and returns -EAGAIN, so the caller retries under the mmap lock.

faultin_page() reports these retries as -EAGAIN for both lock types; only the mmap caller records the dropped lock in *locked.

A VM_FAULT_ERROR decoding to no errno now returns -EFAULT instead of BUG(), warning under CONFIG_DEBUG_VM. This relaxes the same case on the existing __get_user_pages() path too.

Assisted-by Claude:claude-opus-4.8
---
 mm/gup.c      | 176 +++++++++++++++++++++++++++++++++++++++++++-------
 mm/internal.h |   8 ++-
 2 files changed, 159 insertions(+), 25 deletions(-)
mm/gup.c +152 -24
index dcece7b5253c..cb7245b7542f 100644
@@ -1080,9 +1080,16 @@ static int get_gate_page(struct mm_struct *mm, unsigned long address,
 }
 
 /*
- * mmap_lock must be held on entry.  If @flags has FOLL_UNLOCKABLE but not
- * FOLL_NOWAIT, the mmap_lock may be released.  If it is, *@locked will be set
- * to 0 and -EBUSY returned.
+ * The caller holds either the mmap lock, or the per-VMA lock (with
+ * FOLL_VMA_LOCK), on entry. If @flags has FOLL_UNLOCKABLE but not FOLL_NOWAIT,
+ * the mmap_lock may be released. If it is, *@locked will be set to 0 and
+ * -EAGAIN returned.
+ *
+ * The return value does not depend on the lock type: a fault that made
+ * progress but needs a retry (VM_FAULT_RETRY / VM_FAULT_COMPLETED) is reported
+ * as -EAGAIN for both the mmap lock and the per-VMA lock (FOLL_VMA_LOCK). Only
+ * the *@locked side effect is lock-type specific, as the per-VMA lock path has
+ * no unlockable mmap_lock to drop.
  */
 static int faultin_page(struct vm_area_struct *vma,
 		unsigned long address, unsigned int flags, bool unshare,
@@ -1097,6 +1104,8 @@ static int faultin_page(struct vm_area_struct *vma,
 		fault_flags |= FAULT_FLAG_WRITE;
 	if (flags & FOLL_REMOTE)
 		fault_flags |= FAULT_FLAG_REMOTE;
+	if (flags & FOLL_VMA_LOCK)
+		fault_flags |= FAULT_FLAG_VMA_LOCK;
 	if (flags & FOLL_UNLOCKABLE) {
 		fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
 		/*
@@ -1125,41 +1134,160 @@ static int faultin_page(struct vm_area_struct *vma,
 
 	ret = handle_mm_fault(vma, address, fault_flags, NULL);
 
+	/*
+	 * A fully completed fault (VM_FAULT_COMPLETED) or one that needs a retry
+	 * (VM_FAULT_RETRY) has released the lock it was holding. Report both as
+	 * -EAGAIN so the caller retries: the mmap lock caller retakes it here,
+	 * the per-VMA lock caller (FOLL_VMA_LOCK) falls back to the mmap lock.
+	 *
+	 * Dropping the mmap lock is recorded in *@locked. There is no such lock
+	 * to drop under the per-VMA lock, where @locked is not used, so leave it
+	 * alone in that case.
+	 */
 	if (ret & VM_FAULT_COMPLETED) {
-		/*
-		 * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
-		 * mmap lock in the page fault handler. Sanity check this.
-		 */
-		WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
-		*locked = 0;
-
-		/*
-		 * We should do the same as VM_FAULT_RETRY, but let's not
-		 * return -EBUSY since that's not reflecting the reality of
-		 * what has happened - we've just fully completed a page
-		 * fault, with the mmap lock released.  Use -EAGAIN to show
-		 * that we want to take the mmap lock _again_.
-		 */
+		if (!(flags & FOLL_VMA_LOCK)) {
+			/*
+			 * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
+			 * mmap lock in the page fault handler. Sanity check this.
+			 */
+			WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
+			*locked = 0;
+		}
 		return -EAGAIN;
 	}
 
 	if (ret & VM_FAULT_ERROR) {
 		int err = vm_fault_to_errno(ret, flags);
 
-		if (err)
-			return err;
-		BUG();
+		/*
+		 * VM_FAULT_ERROR always decodes to an errno; a zero here would
+		 * mean handle_mm_fault() returned an unexpected combination.
+		 * Report -EFAULT rather than crash: under the per-VMA lock the
+		 * mmap lock retry produces the definitive result.
+		 */
+		VM_WARN_ON_ONCE(!err);
+		return err ? err : -EFAULT;
 	}
 
 	if (ret & VM_FAULT_RETRY) {
-		if (!(fault_flags & FAULT_FLAG_RETRY_NOWAIT))
+		if (!(flags & FOLL_VMA_LOCK) &&
+		    !(fault_flags & FAULT_FLAG_RETRY_NOWAIT))
 			*locked = 0;
-		return -EBUSY;
+		return -EAGAIN;
 	}
 
 	return 0;
 }
 
+/*
+ * get_user_page_vma - get one page from @vma, whose lock the caller already
+ * holds: the mmap lock, or (with FOLL_VMA_LOCK) the per-VMA lock. Walks the
+ * page tables, faulting the page in if needed, and on success returns it with
+ * a reference and the lock still held.
+ *
+ * Runs check_vma_flags() like __get_user_pages(), so callers need not pre-check
+ * the VMA; most rejections are returned as their error. A VM_IO/VM_PFNMAP VMA
+ * is the exception: a COWed page with a struct page is returned, while a raw
+ * PFN has none and yields -EFAULT, to be reached via vma->vm_ops->access().
+ *
+ * Under FOLL_VMA_LOCK, anything that cannot be finished under the per-VMA lock
+ * (a dropped fault, userfaultfd, a hard error, or ->access() memory) releases
+ * the lock and returns -EAGAIN, so the caller retries under the mmap lock.
+ */
+struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,
+			       unsigned int gup_flags)
+{
+	bool vma_locked = gup_flags & FOLL_VMA_LOCK;
+	unsigned long page_mask;
+	struct page *page;
+	int locked = 1;
+	bool pfnmap;
+	int ret;
+
+	/*
+	 * Two lock modes are supported: the mmap lock, with neither flag, or
+	 * the per-VMA lock, with both FOLL_VMA_LOCK and FOLL_UNLOCKABLE. An
+	 * unlockable mmap fault would drop the lock and report it in *locked,
+	 * which this function does not relay, so reject a lone flag.
+	 */
+	VM_WARN_ON_ONCE(!(gup_flags & FOLL_VMA_LOCK) != !(gup_flags & FOLL_UNLOCKABLE));
+
+	/*
+	 * Validate the VMA up front, like __get_user_pages(). A VM_IO/VM_PFNMAP
+	 * VMA is not rejected outright: it can hold COWed pages that have a
+	 * struct page; let follow_page_mask() look for them, and treat only
+	 * its struct-page-less PFNs as unreachable. Other rejections
+	 * (secretmem, permissions, etc) result in immediate failure.
+	 */
+	ret = check_vma_flags(vma, gup_flags, VM_IO | VM_PFNMAP);
+	if (ret)
+		goto fail;
+	pfnmap = vma->vm_flags & (VM_IO | VM_PFNMAP);
+
+	for (;;) {
+		if (fatal_signal_pending(current)) {
+			ret = -EINTR;
+			goto fail;
+		}
+		cond_resched();
+
+		/* follow_page_mask() requires @page_mask; it is unused here. */
+		page = follow_page_mask(vma, addr,
+					gup_flags | FOLL_TOUCH | FOLL_GET,
+					&page_mask);
+		if (!IS_ERR_OR_NULL(page)) {
+			/* Match __get_user_pages(): flush for VIVT/aliasing caches. */
+			flush_anon_page(vma, page, addr);
+			flush_dcache_page(page);
+			return page;
+		}
+
+		/*
+		 * No struct page: a raw PFN of a VM_IO/VM_PFNMAP VMA, whether
+		 * seen by the up-front check (@pfnmap) or reported as -EEXIST
+		 * for a present PFN. Return -EFAULT so the caller reaches it
+		 * through vma->vm_ops->access().
+		 */
+		if (pfnmap || PTR_ERR(page) == -EEXIST) {
+			ret = -EFAULT;
+			goto fail;
+		}
+		/* A hard error from the walk itself. */
+		if (page && PTR_ERR(page) != -EMLINK) {
+			ret = PTR_ERR(page);
+			goto fail;
+		}
+
+		/*
+		 * The page is not present, or needs unsharing. A remote fault
+		 * under the per-VMA lock cannot deliver userfaultfd (which
+		 * assumes current is the faulting task), so fall back for those.
+		 */
+		if (vma_locked && userfaultfd_armed(vma)) {
+			ret = -EAGAIN;
+			goto fail;
+		}
+		ret = faultin_page(vma, addr, gup_flags | FOLL_REMOTE | FOLL_GET,
+				   PTR_ERR(page) == -EMLINK, &locked);
+		if (ret == -EAGAIN)
+			return ERR_PTR(-EAGAIN);	/* fault released the per-VMA lock */
+		if (ret)
+			goto fail;
+	}
+
+fail:
+	/*
+	 * Under the per-VMA lock the caller cannot reach ->access() or act on a
+	 * hard error (both need the mmap lock), so release the lock and have it
+	 * retry there; the mmap-lock pass produces the definitive error.
+	 */
+	if (vma_locked) {
+		vma_end_read(vma);
+		return ERR_PTR(-EAGAIN);
+	}
+	return ERR_PTR(ret);
+}
+
 /*
  * Writing to file-backed mappings which require folio dirty tracking using GUP
  * is a fundamentally broken operation, as kernel write access to GUP mappings
@@ -1197,8 +1325,8 @@ static bool writable_file_mapping_allowed(struct vm_area_struct *vma,
 	return !vma_needs_dirty_tracking(vma);
 }
 
-static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
-			   vm_flags_t ignore_flags)
+int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
+		    vm_flags_t ignore_flags)
 {
 	vm_flags_t vm_flags = vma->vm_flags;
 	int write = (gup_flags & FOLL_WRITE);
mm/internal.h +7 -1
index 181e79f1d6a2..706f7f08fd18 100644
@@ -1595,6 +1595,10 @@ struct vm_struct *__get_vm_area_node(unsigned long size,
  */
 int __must_check try_grab_folio(struct folio *folio, int refs,
 				unsigned int flags);
+int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
+		    vm_flags_t ignore_flags);
+struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,
+			       unsigned int gup_flags);
 
 /*
  * mm/huge_memory.c
@@ -1641,11 +1645,13 @@ enum {
 	FOLL_UNLOCKABLE = 1 << 21,
 	/* VMA lookup+checks compatible with MADV_POPULATE_(READ|WRITE) */
 	FOLL_MADV_POPULATE = 1 << 22,
+	/* caller holds the per-VMA lock, not the mmap lock */
+	FOLL_VMA_LOCK = 1 << 23,
 };
 
 #define INTERNAL_GUP_FLAGS (FOLL_TOUCH | FOLL_TRIED | FOLL_REMOTE | FOLL_PIN | \
 			    FOLL_FAST_ONLY | FOLL_UNLOCKABLE | \
-			    FOLL_MADV_POPULATE)
+			    FOLL_MADV_POPULATE | FOLL_VMA_LOCK)
 
 /*
  * Indicates for which pages that are write-protected in the page table,
Signature · 1 line
2.53.0-Meta
#14 lore raw

Re: [PATCH RFC v4 05/12] mm/gup: add get_user_page_vma() to fault in a page under a held lock

Quoted 27 lines · Rik van Riel

__access_remote_vm() needs a single page from a VMA it has already looked up and locked, faulting it in when necessary, under either the mmap lock or the per-VMA lock. get_user_pages_remote() does not fit: it hard codes the mmap lock and re-looks-up the VMA, neither of which is wanted here.

Add get_user_page_vma(), a simplified __get_user_pages() that walks the page tables with follow_page_mask(), faults a missing page in with faultin_page(), and on success returns it with a reference and the caller's lock still held. Like __get_user_pages() it runs check_vma_flags(), so callers need not pre-check the VMA.

A VM_IO/VM_PFNMAP VMA is the exception to that check: it can still hold COWed pages that have a struct page, so follow_page_mask() is allowed to look for one. Memory with no struct page -- a raw PFN, or a present PFN reported as -EEXIST -- is returned as -EFAULT, so the caller can reach it through vma->vm_ops->access().

The caller sets FOLL_VMA_LOCK when it holds the per-VMA lock rather than the mmap lock, which reaches the fault code as FAULT_FLAG_VMA_LOCK.

Anything that cannot complete under the per-VMA lock -- a dropped fault, a userfaultfd VMA (uffd assumes current is the faulting task), a hard error, or ->access() memory -- releases the lock and returns -EAGAIN, so the caller retries under the mmap lock.

Quoted 4 lines

faultin_page() reports these retries as -EAGAIN for both lock types; only the mmap caller records the dropped lock in *locked.

A VM_FAULT_ERROR decoding to no errno now returns -EFAULT instead of

In "decoding to no errno" sounds weird. Maybe drop "no"?

Quoted 46 lines

BUG(), warning under CONFIG_DEBUG_VM. This relaxes the same case on the existing __get_user_pages() path too.

Assisted-by Claude:claude-opus-4.8
---
 mm/gup.c      | 176 +++++++++++++++++++++++++++++++++++++++++++-------
 mm/internal.h |   8 ++-
 2 files changed, 159 insertions(+), 25 deletions(-)
mm/gup.c +12 -3
index dcece7b5253c..cb7245b7542f 100644
@@ -1080,9 +1080,16 @@ static int get_gate_page(struct mm_struct *mm, unsigned long address,
 }
 
 /*
- * mmap_lock must be held on entry.  If @flags has FOLL_UNLOCKABLE but not
- * FOLL_NOWAIT, the mmap_lock may be released.  If it is, *@locked will be set
- * to 0 and -EBUSY returned.
+ * The caller holds either the mmap lock, or the per-VMA lock (with
+ * FOLL_VMA_LOCK), on entry. If @flags has FOLL_UNLOCKABLE but not FOLL_NOWAIT,
+ * the mmap_lock may be released. If it is, *@locked will be set to 0 and
+ * -EAGAIN returned.
+ *
+ * The return value does not depend on the lock type: a fault that made
+ * progress but needs a retry (VM_FAULT_RETRY / VM_FAULT_COMPLETED) is reported
+ * as -EAGAIN for both the mmap lock and the per-VMA lock (FOLL_VMA_LOCK). Only
+ * the *@locked side effect is lock-type specific, as the per-VMA lock path has
+ * no unlockable mmap_lock to drop.
  */
 static int faultin_page(struct vm_area_struct *vma,
                unsigned long address, unsigned int flags, bool unshare,
@@ -1097,6 +1104,8 @@ static int faultin_page(struct vm_area_struct *vma,
                fault_flags |= FAULT_FLAG_WRITE;
        if (flags & FOLL_REMOTE)
                fault_flags |= FAULT_FLAG_REMOTE;
+       if (flags & FOLL_VMA_LOCK)
+               fault_flags |= FAULT_FLAG_VMA_LOCK;
        if (flags & FOLL_UNLOCKABLE) {
                fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
                /*
@@ -1125,41 +1134,160 @@ static int faultin_page(struct vm_area_struct *vma,
 
        ret = handle_mm_fault(vma, address, fault_flags, NULL);

mm/gup.c static int faultin_page(struct vm_area_struct *vma, · line 1145

Quoted 10 lines
mm/gup.c +9 -0
 
+       /*
+        * A fully completed fault (VM_FAULT_COMPLETED) or one that needs a retry
+        * (VM_FAULT_RETRY) has released the lock it was holding. Report both as
+        * -EAGAIN so the caller retries: the mmap lock caller retakes it here,
+        * the per-VMA lock caller (FOLL_VMA_LOCK) falls back to the mmap lock.
+        *
+        * Dropping the mmap lock is recorded in *@locked. There is no such lock
+        * to drop under the per-VMA lock, where @locked is not used, so leave it
+        * alone in that case.

"There is no such lock to drop under the per-VMA lock" is a bit confusing. Maybe change it to "Under per-VMA lock we are not holding mmap_lock, therefore @locked is not modified."
To avoid more confusion I would rename @locked parameter in this function to @mmap_locked. I think that would make things more clear.

Quoted 5 lines
mm/gup.c +1 -3
+        */
        if (ret & VM_FAULT_COMPLETED) {
-               /*
-                * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
-                * mmap lock in the page fault handler. Sanity check this.

mm/gup.c static int faultin_page(struct vm_area_struct *vma, · line 1141

Quoted 10 lines
mm/gup.c +0 -10
-                */
-               WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
-               *locked = 0;
-
-               /*
-                * We should do the same as VM_FAULT_RETRY, but let's not
-                * return -EBUSY since that's not reflecting the reality of
-                * what has happened - we've just fully completed a page
-                * fault, with the mmap lock released.  Use -EAGAIN to show
-                * that we want to take the mmap lock _again_.

This comment explains why after completing the fault handling we still return -EAGAIN. I think it's still relevant for when we use mmap_lock. Why are you removing it?

Quoted 13 lines
mm/gup.c +8 -1
-                */
+               if (!(flags & FOLL_VMA_LOCK)) {
+                       /*
+                        * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
+                        * mmap lock in the page fault handler. Sanity check this.
+                        */
+                       WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
+                       *locked = 0;
+               }
                return -EAGAIN;
        }
 
        if (ret & VM_FAULT_ERROR) {

mm/gup.c static int faultin_page(struct vm_area_struct *vma, · line 1166

Quoted 10 lines
mm/gup.c +5 -3
                int err = vm_fault_to_errno(ret, flags);
 
-               if (err)
-                       return err;
-               BUG();
+               /*
+                * VM_FAULT_ERROR always decodes to an errno; a zero here would
+                * mean handle_mm_fault() returned an unexpected combination.
+                * Report -EFAULT rather than crash: under the per-VMA lock the
+                * mmap lock retry produces the definitive result.

I don't follow this logic. If handle_mm_fault() under per-VMA locks returns something unexpected, shouldn't we return -EAGAIN to retry under mmap_lock and get the definitive result?

mm/gup.c static int faultin_page(struct vm_area_struct *vma, · line 1176

Quoted 12 lines
mm/gup.c +6 -2
+                */
+               VM_WARN_ON_ONCE(!err);
+               return err ? err : -EFAULT;
        }
 
        if (ret & VM_FAULT_RETRY) {
-               if (!(fault_flags & FAULT_FLAG_RETRY_NOWAIT))
+               if (!(flags & FOLL_VMA_LOCK) &&
+                   !(fault_flags & FAULT_FLAG_RETRY_NOWAIT))
                        *locked = 0;
-               return -EBUSY;
+               return -EAGAIN;

This changes the returned error code even when we operate under mmpa_lock. What's the reason for it?

Quoted 36 lines
mm/gup.c +31 -0
        }
 
        return 0;
 }
 
+/*
+ * get_user_page_vma - get one page from @vma, whose lock the caller already
+ * holds: the mmap lock, or (with FOLL_VMA_LOCK) the per-VMA lock. Walks the
+ * page tables, faulting the page in if needed, and on success returns it with
+ * a reference and the lock still held.
+ *
+ * Runs check_vma_flags() like __get_user_pages(), so callers need not pre-check
+ * the VMA; most rejections are returned as their error. A VM_IO/VM_PFNMAP VMA
+ * is the exception: a COWed page with a struct page is returned, while a raw
+ * PFN has none and yields -EFAULT, to be reached via vma->vm_ops->access().
+ *
+ * Under FOLL_VMA_LOCK, anything that cannot be finished under the per-VMA lock
+ * (a dropped fault, userfaultfd, a hard error, or ->access() memory) releases
+ * the lock and returns -EAGAIN, so the caller retries under the mmap lock.
+ */
+struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,
+                              unsigned int gup_flags)
+{
+       bool vma_locked = gup_flags & FOLL_VMA_LOCK;
+       unsigned long page_mask;
+       struct page *page;
+       int locked = 1;
+       bool pfnmap;
+       int ret;
+
+       /*
+        * Two lock modes are supported: the mmap lock, with neither flag, or
+        * the per-VMA lock, with both FOLL_VMA_LOCK and FOLL_UNLOCKABLE. An
+        * unlockable mmap fault would drop the lock and report it in *locked,
+        * which this function does not relay, so reject a lone flag.
+        */

mm/gup.c static int faultin_page(struct vm_area_struct *vma, · line 1222

Quoted 10 lines
mm/gup.c +10 -0
+       VM_WARN_ON_ONCE(!(gup_flags & FOLL_VMA_LOCK) != !(gup_flags & FOLL_UNLOCKABLE));
+
+       /*
+        * Validate the VMA up front, like __get_user_pages(). A VM_IO/VM_PFNMAP
+        * VMA is not rejected outright: it can hold COWed pages that have a
+        * struct page; let follow_page_mask() look for them, and treat only
+        * its struct-page-less PFNs as unreachable. Other rejections
+        * (secretmem, permissions, etc) result in immediate failure.
+        */
+       ret = check_vma_flags(vma, gup_flags, VM_IO | VM_PFNMAP);

I assume get_user_page_vma() will be called multiple times for the same VMA. Would it make sense to validate the VMA only once (at the level of the caller) instead of doing that for each page? Of you don't want to rely on the caller to do that?

Quoted 115 lines
mm/gup.c +70 -2
+       if (ret)
+               goto fail;
+       pfnmap = vma->vm_flags & (VM_IO | VM_PFNMAP);
+
+       for (;;) {
+               if (fatal_signal_pending(current)) {
+                       ret = -EINTR;
+                       goto fail;
+               }
+               cond_resched();
+
+               /* follow_page_mask() requires @page_mask; it is unused here. */
+               page = follow_page_mask(vma, addr,
+                                       gup_flags | FOLL_TOUCH | FOLL_GET,
+                                       &page_mask);
+               if (!IS_ERR_OR_NULL(page)) {
+                       /* Match __get_user_pages(): flush for VIVT/aliasing caches. */
+                       flush_anon_page(vma, page, addr);
+                       flush_dcache_page(page);
+                       return page;
+               }
+
+               /*
+                * No struct page: a raw PFN of a VM_IO/VM_PFNMAP VMA, whether
+                * seen by the up-front check (@pfnmap) or reported as -EEXIST
+                * for a present PFN. Return -EFAULT so the caller reaches it
+                * through vma->vm_ops->access().
+                */
+               if (pfnmap || PTR_ERR(page) == -EEXIST) {
+                       ret = -EFAULT;
+                       goto fail;
+               }
+               /* A hard error from the walk itself. */
+               if (page && PTR_ERR(page) != -EMLINK) {
+                       ret = PTR_ERR(page);
+                       goto fail;
+               }
+
+               /*
+                * The page is not present, or needs unsharing. A remote fault
+                * under the per-VMA lock cannot deliver userfaultfd (which
+                * assumes current is the faulting task), so fall back for those.
+                */
+               if (vma_locked && userfaultfd_armed(vma)) {
+                       ret = -EAGAIN;
+                       goto fail;
+               }
+               ret = faultin_page(vma, addr, gup_flags | FOLL_REMOTE | FOLL_GET,
+                                  PTR_ERR(page) == -EMLINK, &locked);
+               if (ret == -EAGAIN)
+                       return ERR_PTR(-EAGAIN);        /* fault released the per-VMA lock */
+               if (ret)
+                       goto fail;
+       }
+
+fail:
+       /*
+        * Under the per-VMA lock the caller cannot reach ->access() or act on a
+        * hard error (both need the mmap lock), so release the lock and have it
+        * retry there; the mmap-lock pass produces the definitive error.
+        */
+       if (vma_locked) {
+               vma_end_read(vma);
+               return ERR_PTR(-EAGAIN);
+       }
+       return ERR_PTR(ret);
+}
+
 /*
  * Writing to file-backed mappings which require folio dirty tracking using GUP
  * is a fundamentally broken operation, as kernel write access to GUP mappings
@@ -1197,8 +1325,8 @@ static bool writable_file_mapping_allowed(struct vm_area_struct *vma,
        return !vma_needs_dirty_tracking(vma);
 }
 
-static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
-                          vm_flags_t ignore_flags)
+int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
+                   vm_flags_t ignore_flags)
 {
        vm_flags_t vm_flags = vma->vm_flags;
        int write = (gup_flags & FOLL_WRITE);
mm/internal.h +7 -2
index 181e79f1d6a2..706f7f08fd18 100644
@@ -1595,6 +1595,10 @@ struct vm_struct *__get_vm_area_node(unsigned long size,
  */
 int __must_check try_grab_folio(struct folio *folio, int refs,
                                unsigned int flags);
+int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
+                   vm_flags_t ignore_flags);
+struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,
+                              unsigned int gup_flags);
 
 /*
  * mm/huge_memory.c
@@ -1641,11 +1645,13 @@ enum {
        FOLL_UNLOCKABLE = 1 << 21,
        /* VMA lookup+checks compatible with MADV_POPULATE_(READ|WRITE) */
        FOLL_MADV_POPULATE = 1 << 22,
+       /* caller holds the per-VMA lock, not the mmap lock */
+       FOLL_VMA_LOCK = 1 << 23,
 };
 
 #define INTERNAL_GUP_FLAGS (FOLL_TOUCH | FOLL_TRIED | FOLL_REMOTE | FOLL_PIN | \
                            FOLL_FAST_ONLY | FOLL_UNLOCKABLE | \
-                           FOLL_MADV_POPULATE)
+                           FOLL_MADV_POPULATE | FOLL_VMA_LOCK)
 
 /*
  * Indicates for which pages that are write-protected in the page table,
--

2.53.0-Meta

reviewer
#15 lore raw

Re: [PATCH RFC v4 05/12] mm/gup: add get_user_page_vma() to fault in a page under a held lock

In reply to Rik van Riel

On 7/24/26 3:29 PM, Rik van Riel wrote:
...

Quoted 1 line

+struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,

...

mm/gup.c static int faultin_page(struct vm_area_struct *vma, · line 1225

Quoted 4 lines
mm/gup.c +4 -0
+	ret = check_vma_flags(vma, gup_flags, VM_IO | VM_PFNMAP);
+	if (ret)
+		goto fail;
+	pfnmap = vma->vm_flags & (VM_IO | VM_PFNMAP);

This treats VM_IO and VM_PFNMAP as the same exception, but they have different meanings. VM_IO tells MM callers not to inspect the mapped pages because accesses can have side effects. It does not imply that the VMA contains raw PFNs!

For example, sg_mmap() creates a VM_IO-only VMA whose fault handler installs ordinary pages from the SCSI reserve buffer. If such a page is already present, follow_page_mask() returns it before the pfnmap test below is reached. Patch 6 then lets /proc/PID/mem and ptrace copy that page directly, even though sg provides no ->access callback and the operation fails today. Writes can also modify the buffer when the mapping is writable. Existing ptrace authorization still applies, so this is not a privilege bypass, but it removes the VM_IO exclusion for a driver that did not opt in through ->access.

An untouched page in the same VMA still fails because pfnmap is also true for VM_IO and this function refuses to fault it. The result therefore depends on whether the target previously touched the page. The required exception is narrower. A remap_pfn_range() VMA has both VM_IO and VM_PFNMAP, but VM_PFNMAP is what identifies the mapping whose COWed page should be returned.

So if you use the FOLL_PFNMAP_COW flag suggested on patch 4, and test only VM_PFNMAP here:

	ret = check_vma_flags(vma, gup_flags);
	if (ret)
		goto fail;
	pfnmap = vma->vm_flags & VM_PFNMAP;

Patch 6 can set FOLL_PFNMAP_COW for __access_remote_vm(). check_vma_flags() then permits VM_IO | VM_PFNMAP for that caller while continuing to reject VM_IO-only mappings. vm_normal_page() supplies the remaining distinction: it returns the COWed normal page and rejects the raw PFN.

Something like that.

thanks,

Signature · 1 line
John Hubbard
author reviewer
#16 lore raw

RFC PATCH v4 6/12 mm: use per-VMA lock in __access_remote_vm() for single-VMA accesses

In reply to Rik van Riel

__access_remote_vm() holds mmap_read_lock() for the whole transfer. On large machines running big multi-threaded applications, that lock is contended between readers and writers: an mmap() or munmap() stalls readers like /proc/PID/cmdline, /proc/PID/environ, or /proc/PID/mem, even though the memory they read is almost always resident.

Use the per-VMA lock to access a remote process's memory when the access fits in one VMA. Fall back to the mmap_lock if the access crosses VMA boundaries, or when get_user_page_vma() cannot finish the access under the per-VMA lock.

Factor the walk into remote_vm_walk(): it selects the lock, faults in each page with get_user_page_vma(), and hands the page to a per-page action. __access_remote_vm() passes access_vm_page(), which copies it; a later patch shares the walk for the remote string reader.

remote_access_lock() and remote_access_unlock() keep the lock selection out of the walk. remote_access_lock() takes the per-VMA lock when the range fits one VMA whose flags permit the access, and the mmap lock otherwise, returning an ERR_PTR() when the mmap lock cannot be taken.

Looking up the VMA first untags the remote address with untagged_addr_remote_unlocked(), added earlier in this series, so the untag needs no mmap lock.

Walking the page tables under only the per-VMA lock is safe against both page table freeing and THP collapse.

munmap() frees a VMA's page tables through free_pgtables(), which uses neither the page table lock nor RCU. But it runs only after the VMA is detached under the VMA write lock, which our per-VMA read lock excludes, so no page table of this VMA is torn down under the walk.

THP collapse instead needs neither lock: file-backed collapse retracts a page table under i_mmap_lock and the page table lock alone. It frees the retracted PTE page by RCU, and pte_offset_map() holds the RCU read lock, so the page stays valid for the walk.

follow_page_pte() takes that same page table lock through pte_offset_map_lock() and rechecks the pmd, so it either walks an intact table or sees the collapsed pmd and faults the THP in.

get_user_page_vma() returns -EFAULT for memory with no struct page: the raw PFNs of a VM_IO/VM_PFNMAP VMA, ioremapped device memory reached through ptrace and /proc/PID/mem.

access_vm_page() reaches that memory under the mmap lock through vma->vm_ops->access(), via a new access_remote_vma_ops() helper, as get_user_pages_remote() and the old ->access() fallback did before.

A COWed page in such a VMA now reads normally through get_user_page_vma(); previously it was routed to ->access(), whose generic_access_phys() rejects the ioremap of a RAM page.

A single-threaded microbenchmark reading a remote process's memory through /proc/PID/mem shows the per-VMA path costs less than the old mmap_read_lock() plus get_user_pages_remote() route. The gain grows with the read size as the per-page VMA re-lookup drops away. Median of three pinned repetitions in a VM:

	read size   baseline    per-VMA     throughput
	8 B          201.7 ns    170.6 ns    ~15% faster
	4 KB         495 ns      442 ns      +12% (8266 -> 9265 MB/s)
	64 KB       4115 ns     3355 ns      +23% (15927 -> 19536 MB/s)
	1 MB       75902 ns    63641 ns      +19% (13815 -> 16477 MB/s)

This does not measure the multi-threaded reader-versus-writer contention that motivates the per-VMA lock; that case remains to be quantified.

Assisted-by Claude:claude-opus-4.8
---
 mm/memory.c | 268 +++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 213 insertions(+), 55 deletions(-)
mm/memory.c +213 -55
index 3b86eeaf084f..273dfe12bc6d 100644
@@ -7015,86 +7015,244 @@ EXPORT_SYMBOL_GPL(generic_access_phys);
 #endif
 
 /*
- * Access another process' address space as given in mm.
+ * VM_IO / VM_PFNMAP memory, such as an ioremapped device mapping, maps
+ * PFNs that have no struct page, so get_user_page_vma() cannot fetch it
+ * even though the page tables are populated. It can still be reached
+ * through vma->vm_ops->access().
+ *
+ * Returns the number of bytes transferred, or <= 0 if @vma cannot be
+ * accessed this way.
  */
-static int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
-			      void *buf, int len, unsigned int gup_flags)
+static int access_remote_vma_ops(struct vm_area_struct *vma, unsigned long addr,
+				 void *buf, int len, int write)
+{
+#ifdef CONFIG_HAVE_IOREMAP_PROT
+	if (vma->vm_ops && vma->vm_ops->access)
+		return vma->vm_ops->access(vma, addr, buf, len, write);
+#endif
+	return 0;
+}
+
+/*
+ * Lock @mm to reach the remote range [@addr, @addr + @len).
+ *
+ * Take the per-VMA lock when the whole range fits in a single VMA whose
+ * flags permit the access. The RCU freed page tables then keep page table
+ * memory from being reused with unexpected contents while the lock is held.
+ * Otherwise fall back to the mmap lock, which also covers multi-VMA ranges,
+ * stack expansion, and ->access() memory.
+ *
+ * Return whether the mmap lock is held. The per-VMA locked VMA, when one is
+ * taken, is stored in *@vmap; it is NULL on the mmap lock path. *@vmap is
+ * set to an ERR_PTR() when the mmap lock could not be taken, so callers must
+ * check IS_ERR(*@vmap) before using either result.
+ */
+static bool remote_access_lock(struct mm_struct *mm, unsigned long addr,
+			       int len, unsigned int gup_flags,
+			       struct vm_area_struct **vmap)
+{
+	struct vm_area_struct *vma = NULL;
+
+#if defined(CONFIG_PER_VMA_LOCK) && defined(CONFIG_MMU_GATHER_RCU_TABLE_FREE)
+	vma = lock_vma_under_rcu(mm, addr);
+	if (vma) {
+		/* addr + len must not wrap, and must fit within the one VMA. */
+		if (addr + len < addr || addr + len > vma->vm_end ||
+		    check_vma_flags(vma, gup_flags, 0)) {
+			vma_end_read(vma);
+			vma = NULL;
+		}
+	}
+#endif
+
+	if (!vma) {
+		if (mmap_read_lock_killable(mm)) {
+			*vmap = ERR_PTR(-EINTR);
+			return false;
+		}
+		*vmap = NULL;
+		return true;
+	}
+
+	*vmap = vma;
+	return false;
+}
+
+/* Release the lock taken by remote_access_lock(). */
+static void remote_access_unlock(struct mm_struct *mm,
+				 struct vm_area_struct *vma, bool have_mmap_lock)
+{
+	if (have_mmap_lock)
+		mmap_read_unlock(mm);
+	else if (vma)
+		vma_end_read(vma);
+}
+
+/*
+ * Per-page action for a remote VM walk. Handle up to @len bytes at @addr on
+ * @page, advancing *@buf past the bytes read from or written to it. @page is
+ * NULL for struct-page-less memory (VM_IO / VM_PFNMAP) reached under the mmap
+ * lock.
+ *
+ * Return the number of source bytes handled at @addr, 0 to end the walk (a
+ * string reached its NUL, or ->access() memory could not be reached), or a
+ * negative errno to abort.
+ */
+typedef int (*remote_vm_action)(struct vm_area_struct *vma, struct page *page,
+				unsigned long addr, void **buf, int len,
+				int write);
+
+/*
+ * Walk the remote range [@addr, @addr + @len) of @mm, handing each page to
+ * @action. Use the per-VMA lock when the range fits one VMA, and fall back to
+ * the mmap lock for multi-VMA ranges, stack expansion (when @can_expand_stack
+ * is set), or when the per-VMA lock cannot finish a fault.
+ *
+ * Each page is faulted in with get_user_page_vma() under whichever lock is
+ * held. Return the number of bytes @action consumed; *@err is a negative
+ * errno when the walk aborted, else 0.
+ */
+static int remote_vm_walk(struct mm_struct *mm, unsigned long addr, void *buf,
+			  int len, unsigned int gup_flags, bool can_expand_stack,
+			  remote_vm_action action, int *err)
 {
 	void *old_buf = buf;
 	int write = gup_flags & FOLL_WRITE;
+	bool have_mmap_lock;
+	struct vm_area_struct *vma;
 
-	if (mmap_read_lock_killable(mm))
-		return 0;
+	*err = 0;
 
-	/* Untag the address before looking up the VMA */
-	addr = untagged_addr_remote(mm, addr);
+	/*
+	 * Set FOLL_REMOTE so check_vma_flags() applies the same protection key
+	 * rules as get_user_pages_remote() did: the current PKRU is not checked
+	 * against a VMA reached on @mm's behalf.
+	 */
+	gup_flags |= FOLL_REMOTE;
 
-	/* Avoid triggering the temporary warning in __get_user_pages */
-	if (!vma_lookup(mm, addr) && !expand_stack(mm, addr))
+	addr = untagged_addr_remote_unlocked(mm, addr);
+
+	have_mmap_lock = remote_access_lock(mm, addr, len, gup_flags, &vma);
+	if (IS_ERR(vma)) {
+		*err = -EFAULT;
 		return 0;
+	}
 
-	/* ignore errors, just check how much was successfully transferred */
 	while (len) {
-		int bytes, offset;
-		void *maddr;
-		struct folio *folio;
-		struct vm_area_struct *vma = NULL;
-		struct page *page = get_user_page_lookup_vma(mm, addr,
-							     gup_flags, &vma);
+		unsigned int foll_flags = gup_flags;
+		struct page *page;
+		int ret;
 
-		if (IS_ERR(page)) {
-			/* We might need to expand the stack to access it */
+		if (!vma || addr >= vma->vm_end) {
+			/* Any lookup here holds the mmap lock. */
+			VM_BUG_ON(!have_mmap_lock);
 			vma = vma_lookup(mm, addr);
-			if (!vma) {
+			if (!vma && can_expand_stack) {
+				/* expand_stack() drops the mmap lock if it fails */
 				vma = expand_stack(mm, addr);
-
-				/* mmap_lock was dropped on failure */
 				if (!vma)
-					return buf - old_buf;
-
-				/* Try again if stack expansion worked */
-				continue;
+					have_mmap_lock = false;
 			}
+			if (!vma) {
+				*err = -EFAULT;
+				break;
+			}
+		}
 
+		/*
+		 * FOLL_UNLOCKABLE lets the per-VMA fault retry, dropping the
+		 * lock, so the walk can fall back to the mmap lock.
+		 */
+		if (!have_mmap_lock)
+			foll_flags |= FOLL_VMA_LOCK | FOLL_UNLOCKABLE;
+
+		page = get_user_page_vma(vma, addr, foll_flags);
+		if (IS_ERR(page)) {
 			/*
-			 * Check if this is a VM_IO | VM_PFNMAP VMA, which
-			 * we can access using slightly different code.
+			 * get_user_page_vma() returns -EAGAIN, with the per-VMA
+			 * lock released, for anything it could not finish under
+			 * it; retake the mmap lock and retry. A different error
+			 * therefore only arrives under the mmap lock, where
+			 * struct-page-less memory can be reached via ->access().
 			 */
-			bytes = 0;
-#ifdef CONFIG_HAVE_IOREMAP_PROT
-			if (vma->vm_ops && vma->vm_ops->access)
-				bytes = vma->vm_ops->access(vma, addr, buf,
-							    len, write);
-#endif
-			if (bytes <= 0)
-				break;
-		} else {
-			folio = page_folio(page);
-			bytes = len;
-			offset = addr & (PAGE_SIZE-1);
-			if (bytes > PAGE_SIZE-offset)
-				bytes = PAGE_SIZE-offset;
-
-			maddr = kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE);
-			if (write) {
-				copy_to_user_page(vma, page, addr,
-						  maddr + offset, buf, bytes);
-				folio_mark_dirty_lock(folio);
-			} else {
-				copy_from_user_page(vma, page, addr,
-						    buf, maddr + offset, bytes);
+			if (PTR_ERR(page) == -EAGAIN) {
+				vma = NULL;
+				if (mmap_read_lock_killable(mm)) {
+					*err = -EFAULT;
+					break;
+				}
+				have_mmap_lock = true;
+				continue;
 			}
-			folio_release_kmap(folio, maddr);
+			if (WARN_ON_ONCE(!have_mmap_lock))
+				break;
+			page = NULL;
 		}
-		len -= bytes;
-		buf += bytes;
-		addr += bytes;
+
+		ret = action(vma, page, addr, &buf, len, write);
+		if (ret <= 0) {
+			if (ret < 0)
+				*err = ret;
+			break;
+		}
+		addr += ret;
+		len -= ret;
 	}
-	mmap_read_unlock(mm);
+
+	remote_access_unlock(mm, vma, have_mmap_lock);
 
 	return buf - old_buf;
 }
 
+/*
+ * Copy one page's worth of [@addr, @addr + @len) to or from *@buf. Reaches
+ * struct-page-less VM_IO / VM_PFNMAP memory through vma->vm_ops->access().
+ */
+static int access_vm_page(struct vm_area_struct *vma, struct page *page,
+			  unsigned long addr, void **buf, int len, int write)
+{
+	struct folio *folio;
+	int bytes, offset;
+	void *maddr;
+
+	if (!page) {
+		bytes = access_remote_vma_ops(vma, addr, *buf, len, write);
+		if (bytes > 0)
+			*buf += bytes;
+		return bytes;
+	}
+
+	bytes = len;
+	offset = addr & (PAGE_SIZE - 1);
+	if (bytes > PAGE_SIZE - offset)
+		bytes = PAGE_SIZE - offset;
+
+	folio = page_folio(page);
+	maddr = kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE);
+	if (write) {
+		copy_to_user_page(vma, page, addr, maddr + offset, *buf, bytes);
+		folio_mark_dirty_lock(folio);
+	} else {
+		copy_from_user_page(vma, page, addr, *buf, maddr + offset, bytes);
+	}
+	folio_release_kmap(folio, maddr);
+
+	*buf += bytes;
+	return bytes;
+}
+
+/*
+ * Access another process' address space as given in mm.
+ */
+static int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
+			      void *buf, int len, unsigned int gup_flags)
+{
+	int err;
+
+	return remote_vm_walk(mm, addr, buf, len, gup_flags, true,
+			      access_vm_page, &err);
+}
+
 /**
  * access_remote_vm - access another process' address space
  * @mm:		the mm_struct of the target address space
Signature · 1 line
2.53.0-Meta
#17 lore raw

Re: [PATCH RFC v4 06/12] mm: use per-VMA lock in __access_remote_vm() for single-VMA accesses

Quoted 67 lines · Rik van Riel

__access_remote_vm() holds mmap_read_lock() for the whole transfer. On large machines running big multi-threaded applications, that lock is contended between readers and writers: an mmap() or munmap() stalls readers like /proc/PID/cmdline, /proc/PID/environ, or /proc/PID/mem, even though the memory they read is almost always resident.

Use the per-VMA lock to access a remote process's memory when the access fits in one VMA. Fall back to the mmap_lock if the access crosses VMA boundaries, or when get_user_page_vma() cannot finish the access under the per-VMA lock.

Factor the walk into remote_vm_walk(): it selects the lock, faults in each page with get_user_page_vma(), and hands the page to a per-page action. __access_remote_vm() passes access_vm_page(), which copies it; a later patch shares the walk for the remote string reader.

remote_access_lock() and remote_access_unlock() keep the lock selection out of the walk. remote_access_lock() takes the per-VMA lock when the range fits one VMA whose flags permit the access, and the mmap lock otherwise, returning an ERR_PTR() when the mmap lock cannot be taken.

Looking up the VMA first untags the remote address with untagged_addr_remote_unlocked(), added earlier in this series, so the untag needs no mmap lock.

Walking the page tables under only the per-VMA lock is safe against both page table freeing and THP collapse.

munmap() frees a VMA's page tables through free_pgtables(), which uses neither the page table lock nor RCU. But it runs only after the VMA is detached under the VMA write lock, which our per-VMA read lock excludes, so no page table of this VMA is torn down under the walk.

THP collapse instead needs neither lock: file-backed collapse retracts a page table under i_mmap_lock and the page table lock alone. It frees the retracted PTE page by RCU, and pte_offset_map() holds the RCU read lock, so the page stays valid for the walk.

follow_page_pte() takes that same page table lock through pte_offset_map_lock() and rechecks the pmd, so it either walks an intact table or sees the collapsed pmd and faults the THP in.

get_user_page_vma() returns -EFAULT for memory with no struct page: the raw PFNs of a VM_IO/VM_PFNMAP VMA, ioremapped device memory reached through ptrace and /proc/PID/mem.

access_vm_page() reaches that memory under the mmap lock through vma->vm_ops->access(), via a new access_remote_vma_ops() helper, as get_user_pages_remote() and the old ->access() fallback did before.

A COWed page in such a VMA now reads normally through get_user_page_vma(); previously it was routed to ->access(), whose generic_access_phys() rejects the ioremap of a RAM page.

A single-threaded microbenchmark reading a remote process's memory through /proc/PID/mem shows the per-VMA path costs less than the old mmap_read_lock() plus get_user_pages_remote() route. The gain grows with the read size as the per-page VMA re-lookup drops away. Median of three pinned repetitions in a VM:

        read size   baseline    per-VMA     throughput
        8 B          201.7 ns    170.6 ns    ~15% faster
        4 KB         495 ns      442 ns      +12% (8266 -> 9265 MB/s)
        64 KB       4115 ns     3355 ns      +23% (15927 -> 19536 MB/s)
        1 MB       75902 ns    63641 ns      +19% (13815 -> 16477 MB/s)
Quoted 6 lines

This does not measure the multi-threaded reader-versus-writer contention that motivates the per-VMA lock; that case remains to be quantified.

Assisted-by Claude:claude-opus-4.8

Some nits below but otherwise looks very nice!

Quoted 36 lines
---
 mm/memory.c | 268 +++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 213 insertions(+), 55 deletions(-)
mm/memory.c +20 -3
index 3b86eeaf084f..273dfe12bc6d 100644
@@ -7015,86 +7015,244 @@ EXPORT_SYMBOL_GPL(generic_access_phys);
 #endif
 
 /*
- * Access another process' address space as given in mm.
+ * VM_IO / VM_PFNMAP memory, such as an ioremapped device mapping, maps
+ * PFNs that have no struct page, so get_user_page_vma() cannot fetch it
+ * even though the page tables are populated. It can still be reached
+ * through vma->vm_ops->access().
+ *
+ * Returns the number of bytes transferred, or <= 0 if @vma cannot be
+ * accessed this way.
  */
-static int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
-                             void *buf, int len, unsigned int gup_flags)
+static int access_remote_vma_ops(struct vm_area_struct *vma, unsigned long addr,
+                                void *buf, int len, int write)
+{
+#ifdef CONFIG_HAVE_IOREMAP_PROT
+       if (vma->vm_ops && vma->vm_ops->access)
+               return vma->vm_ops->access(vma, addr, buf, len, write);
+#endif
+       return 0;
+}
+
+/*
+ * Lock @mm to reach the remote range [@addr, @addr + @len).
+ *

mm/memory.c EXPORT_SYMBOL_GPL(generic_access_phys); · line 7048

Quoted 10 lines
mm/memory.c +10 -0
+ * Take the per-VMA lock when the whole range fits in a single VMA whose
+ * flags permit the access. The RCU freed page tables then keep page table
+ * memory from being reused with unexpected contents while the lock is held.
+ * Otherwise fall back to the mmap lock, which also covers multi-VMA ranges,
+ * stack expansion, and ->access() memory.
+ *
+ * Return whether the mmap lock is held. The per-VMA locked VMA, when one is
+ * taken, is stored in *@vmap; it is NULL on the mmap lock path. *@vmap is
+ * set to an ERR_PTR() when the mmap lock could not be taken, so callers must
+ * check IS_ERR(*@vmap) before using either result.

This is an unusual way of returning an error. Why not return an int with 0 for success, errno for any error and use *vmap to indicate whether we took per-VMA lock (*vmap!=NULL) or mmap_lock (*vmap==NULL)? That seems to me less confusing.

Quoted 6 lines
mm/memory.c +6 -0
+ */
+static bool remote_access_lock(struct mm_struct *mm, unsigned long addr,
+                              int len, unsigned int gup_flags,
+                              struct vm_area_struct **vmap)
+{
+       struct vm_area_struct *vma = NULL;

mm/memory.c EXPORT_SYMBOL_GPL(generic_access_phys); · line 7064

Quoted 10 lines
mm/memory.c +10 -0
+
+#if defined(CONFIG_PER_VMA_LOCK) && defined(CONFIG_MMU_GATHER_RCU_TABLE_FREE)
+       vma = lock_vma_under_rcu(mm, addr);
+       if (vma) {
+               /* addr + len must not wrap, and must fit within the one VMA. */
+               if (addr + len < addr || addr + len > vma->vm_end ||
+                   check_vma_flags(vma, gup_flags, 0)) {
+                       vma_end_read(vma);
+                       vma = NULL;
+               }

nit: This function can be simplified if we set vmap=NULL at the beginning and then if the vma we found is fine then set *vmap=vma and return early.

Quoted 27 lines
mm/memory.c +27 -0
+       }
+#endif
+
+       if (!vma) {
+               if (mmap_read_lock_killable(mm)) {
+                       *vmap = ERR_PTR(-EINTR);
+                       return false;
+               }
+               *vmap = NULL;
+               return true;
+       }
+
+       *vmap = vma;
+       return false;
+}
+
+/* Release the lock taken by remote_access_lock(). */
+static void remote_access_unlock(struct mm_struct *mm,
+                                struct vm_area_struct *vma, bool have_mmap_lock)
+{
+       if (have_mmap_lock)
+               mmap_read_unlock(mm);
+       else if (vma)
+               vma_end_read(vma);
+}
+
+/*

mm/memory.c EXPORT_SYMBOL_GPL(generic_access_phys); · line 7101

Quoted 10 lines
mm/memory.c +10 -0
+ * Per-page action for a remote VM walk. Handle up to @len bytes at @addr on
+ * @page, advancing *@buf past the bytes read from or written to it. @page is
+ * NULL for struct-page-less memory (VM_IO / VM_PFNMAP) reached under the mmap
+ * lock.
+ *
+ * Return the number of source bytes handled at @addr, 0 to end the walk (a
+ * string reached its NUL, or ->access() memory could not be reached), or a
+ * negative errno to abort.
+ */
+typedef int (*remote_vm_action)(struct vm_area_struct *vma, struct page *page,

Maybe we should use folio instead of page from the get go? I think you can convert page to folio right after you get it from get_user_page_vma().

Quoted 10 lines
mm/memory.c +10 -0
+                               unsigned long addr, void **buf, int len,
+                               int write);
+
+/*
+ * Walk the remote range [@addr, @addr + @len) of @mm, handing each page to
+ * @action. Use the per-VMA lock when the range fits one VMA, and fall back to
+ * the mmap lock for multi-VMA ranges, stack expansion (when @can_expand_stack
+ * is set), or when the per-VMA lock cannot finish a fault.
+ *
+ * Each page is faulted in with get_user_page_vma() under whichever lock is

mm/memory.c EXPORT_SYMBOL_GPL(generic_access_phys); · line 7121

Quoted 10 lines
mm/memory.c +7 -0
+ * held. Return the number of bytes @action consumed; *@err is a negative
+ * errno when the walk aborted, else 0.
+ */
+static int remote_vm_walk(struct mm_struct *mm, unsigned long addr, void *buf,
+                         int len, unsigned int gup_flags, bool can_expand_stack,
+                         remote_vm_action action, int *err)
 {
        void *old_buf = buf;
        int write = gup_flags & FOLL_WRITE;
+       bool have_mmap_lock;

nit: I always associate have_xxx with features supported by a given architecture. Maybe it's just me but I would prefer "mmap_locked" or "mmap_lock_held".

Quoted 193 lines
mm/memory.c +113 -53
+       struct vm_area_struct *vma;
 
-       if (mmap_read_lock_killable(mm))
-               return 0;
+       *err = 0;
 
-       /* Untag the address before looking up the VMA */
-       addr = untagged_addr_remote(mm, addr);
+       /*
+        * Set FOLL_REMOTE so check_vma_flags() applies the same protection key
+        * rules as get_user_pages_remote() did: the current PKRU is not checked
+        * against a VMA reached on @mm's behalf.
+        */
+       gup_flags |= FOLL_REMOTE;
 
-       /* Avoid triggering the temporary warning in __get_user_pages */
-       if (!vma_lookup(mm, addr) && !expand_stack(mm, addr))
+       addr = untagged_addr_remote_unlocked(mm, addr);
+
+       have_mmap_lock = remote_access_lock(mm, addr, len, gup_flags, &vma);
+       if (IS_ERR(vma)) {
+               *err = -EFAULT;
                return 0;
+       }
 
-       /* ignore errors, just check how much was successfully transferred */
        while (len) {
-               int bytes, offset;
-               void *maddr;
-               struct folio *folio;
-               struct vm_area_struct *vma = NULL;
-               struct page *page = get_user_page_lookup_vma(mm, addr,
-                                                            gup_flags, &vma);
+               unsigned int foll_flags = gup_flags;
+               struct page *page;
+               int ret;
 
-               if (IS_ERR(page)) {
-                       /* We might need to expand the stack to access it */
+               if (!vma || addr >= vma->vm_end) {
+                       /* Any lookup here holds the mmap lock. */
+                       VM_BUG_ON(!have_mmap_lock);
                        vma = vma_lookup(mm, addr);
-                       if (!vma) {
+                       if (!vma && can_expand_stack) {
+                               /* expand_stack() drops the mmap lock if it fails */
                                vma = expand_stack(mm, addr);
-
-                               /* mmap_lock was dropped on failure */
                                if (!vma)
-                                       return buf - old_buf;
-
-                               /* Try again if stack expansion worked */
-                               continue;
+                                       have_mmap_lock = false;
                        }
+                       if (!vma) {
+                               *err = -EFAULT;
+                               break;
+                       }
+               }
 
+               /*
+                * FOLL_UNLOCKABLE lets the per-VMA fault retry, dropping the
+                * lock, so the walk can fall back to the mmap lock.
+                */
+               if (!have_mmap_lock)
+                       foll_flags |= FOLL_VMA_LOCK | FOLL_UNLOCKABLE;
+
+               page = get_user_page_vma(vma, addr, foll_flags);
+               if (IS_ERR(page)) {
                        /*
-                        * Check if this is a VM_IO | VM_PFNMAP VMA, which
-                        * we can access using slightly different code.
+                        * get_user_page_vma() returns -EAGAIN, with the per-VMA
+                        * lock released, for anything it could not finish under
+                        * it; retake the mmap lock and retry. A different error
+                        * therefore only arrives under the mmap lock, where
+                        * struct-page-less memory can be reached via ->access().
                         */
-                       bytes = 0;
-#ifdef CONFIG_HAVE_IOREMAP_PROT
-                       if (vma->vm_ops && vma->vm_ops->access)
-                               bytes = vma->vm_ops->access(vma, addr, buf,
-                                                           len, write);
-#endif
-                       if (bytes <= 0)
-                               break;
-               } else {
-                       folio = page_folio(page);
-                       bytes = len;
-                       offset = addr & (PAGE_SIZE-1);
-                       if (bytes > PAGE_SIZE-offset)
-                               bytes = PAGE_SIZE-offset;
-
-                       maddr = kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE);
-                       if (write) {
-                               copy_to_user_page(vma, page, addr,
-                                                 maddr + offset, buf, bytes);
-                               folio_mark_dirty_lock(folio);
-                       } else {
-                               copy_from_user_page(vma, page, addr,
-                                                   buf, maddr + offset, bytes);
+                       if (PTR_ERR(page) == -EAGAIN) {
+                               vma = NULL;
+                               if (mmap_read_lock_killable(mm)) {
+                                       *err = -EFAULT;
+                                       break;
+                               }
+                               have_mmap_lock = true;
+                               continue;
                        }
-                       folio_release_kmap(folio, maddr);
+                       if (WARN_ON_ONCE(!have_mmap_lock))
+                               break;
+                       page = NULL;
                }
-               len -= bytes;
-               buf += bytes;
-               addr += bytes;
+
+               ret = action(vma, page, addr, &buf, len, write);
+               if (ret <= 0) {
+                       if (ret < 0)
+                               *err = ret;
+                       break;
+               }
+               addr += ret;
+               len -= ret;
        }
-       mmap_read_unlock(mm);
+
+       remote_access_unlock(mm, vma, have_mmap_lock);
 
        return buf - old_buf;
 }
 
+/*
+ * Copy one page's worth of [@addr, @addr + @len) to or from *@buf. Reaches
+ * struct-page-less VM_IO / VM_PFNMAP memory through vma->vm_ops->access().
+ */
+static int access_vm_page(struct vm_area_struct *vma, struct page *page,
+                         unsigned long addr, void **buf, int len, int write)
+{
+       struct folio *folio;
+       int bytes, offset;
+       void *maddr;
+
+       if (!page) {
+               bytes = access_remote_vma_ops(vma, addr, *buf, len, write);
+               if (bytes > 0)
+                       *buf += bytes;
+               return bytes;
+       }
+
+       bytes = len;
+       offset = addr & (PAGE_SIZE - 1);
+       if (bytes > PAGE_SIZE - offset)
+               bytes = PAGE_SIZE - offset;
+
+       folio = page_folio(page);
+       maddr = kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE);
+       if (write) {
+               copy_to_user_page(vma, page, addr, maddr + offset, *buf, bytes);
+               folio_mark_dirty_lock(folio);
+       } else {
+               copy_from_user_page(vma, page, addr, *buf, maddr + offset, bytes);
+       }
+       folio_release_kmap(folio, maddr);
+
+       *buf += bytes;
+       return bytes;
+}
+
+/*
+ * Access another process' address space as given in mm.
+ */
+static int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
+                             void *buf, int len, unsigned int gup_flags)
+{
+       int err;
+
+       return remote_vm_walk(mm, addr, buf, len, gup_flags, true,
+                             access_vm_page, &err);
+}
+
 /**
  * access_remote_vm - access another process' address space
  * @mm:                the mm_struct of the target address space
--

2.53.0-Meta

author reviewer
#18 lore raw

RFC PATCH v4 7/12 mm: read remote strings under the per-VMA lock

In reply to Rik van Riel

__copy_remote_vm_str() reads another process's memory under the mmap read lock for the whole copy, and looks the VMA up again for every page through get_user_page_lookup_vma().

This is the mmap-lock-only form that __access_remote_vm() used before it moved to the per-VMA lock, and it shares the same contention: an mmap() or munmap() stalls a bpf_copy_from_user_task_str() reader even though the string is resident.

Read the string through remote_vm_walk(), the walk __access_remote_vm() already uses, with a copy_vm_str() action that strscpy()s each page and stops at the NUL. Bounding the copy by its maximum length keeps the per-VMA path inside a single VMA, so no VMA is looked up again while that lock is held.

A copy the per-VMA lock cannot finish falls back to the mmap lock: a fault that dropped the lock, or a string running past the VMA. Memory with no struct page (VM_IO/VM_PFNMAP) and stack expansion stay on the mmap lock path and still return -EFAULT, as before; ->access() support is left for later.

Assisted-by Claude:claude-opus-4.8
---
 mm/memory.c | 111 ++++++++++++++++++++++------------------------------
 1 file changed, 47 insertions(+), 64 deletions(-)
mm/memory.c +47 -64
index 273dfe12bc6d..aaf620017f34 100644
@@ -7296,84 +7296,67 @@ EXPORT_SYMBOL_GPL(access_process_vm);
 
 #ifdef CONFIG_BPF_SYSCALL
 /*
- * Copy a string from another process's address space as given in mm.
- * If there is any error return -EFAULT.
+ * Copy a NUL-terminated string from @addr into *@buf, up to @len bytes,
+ * stopping at the NUL. strscpy() always NUL terminates, so recopy the last
+ * byte of a page when more pages follow. A string is never read from
+ * struct-page-less VM_IO / VM_PFNMAP memory.
  */
-static int __copy_remote_vm_str(struct mm_struct *mm, unsigned long addr,
-				void *buf, int len, unsigned int gup_flags)
+static int copy_vm_str(struct vm_area_struct *vma, struct page *page,
+		       unsigned long addr, void **buf, int len, int write)
 {
-	void *old_buf = buf;
-	int err = 0;
-
-	*(char *)buf = '\0';
+	struct folio *folio;
+	int bytes, offset, retval;
+	void *maddr;
 
-	if (mmap_read_lock_killable(mm))
+	if (!page)
 		return -EFAULT;
 
-	addr = untagged_addr_remote(mm, addr);
+	bytes = len;
+	offset = addr & (PAGE_SIZE - 1);
+	if (bytes > PAGE_SIZE - offset)
+		bytes = PAGE_SIZE - offset;
 
-	/* Avoid triggering the temporary warning in __get_user_pages */
-	if (!vma_lookup(mm, addr)) {
-		err = -EFAULT;
-		goto out;
+	folio = page_folio(page);
+	maddr = kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE);
+	retval = strscpy(*buf, maddr + offset, bytes);
+	if (retval >= 0) {
+		/* Found the end of the string. */
+		*buf += retval;
+		folio_release_kmap(folio, maddr);
+		return 0;
 	}
 
-	while (len) {
-		int bytes, offset, retval;
-		void *maddr;
-		struct folio *folio;
-		struct page *page;
-		struct vm_area_struct *vma = NULL;
+	*buf += bytes - 1;
+	if (bytes != len) {
+		copy_from_user_page(vma, page, addr + bytes - 1, *buf,
+				    maddr + (PAGE_SIZE - 1), 1);
+		*buf += 1;
+	}
+	folio_release_kmap(folio, maddr);
 
-		page = get_user_page_lookup_vma(mm, addr, gup_flags, &vma);
-		if (IS_ERR(page)) {
-			/*
-			 * Treat as a total failure for now until we decide how
-			 * to handle the CONFIG_HAVE_IOREMAP_PROT case and
-			 * stack expansion.
-			 */
-			*(char *)buf = '\0';
-			err = -EFAULT;
-			goto out;
-		}
+	return bytes;
+}
 
-		folio = page_folio(page);
-		bytes = len;
-		offset = addr & (PAGE_SIZE - 1);
-		if (bytes > PAGE_SIZE - offset)
-			bytes = PAGE_SIZE - offset;
-
-		maddr = kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE);
-		retval = strscpy(buf, maddr + offset, bytes);
-		if (retval >= 0) {
-			/* Found the end of the string */
-			buf += retval;
-			folio_release_kmap(folio, maddr);
-			break;
-		}
+/*
+ * Copy a string from another process's address space as given in mm.
+ * If there is any error return -EFAULT.
+ */
+static int __copy_remote_vm_str(struct mm_struct *mm, unsigned long addr,
+				void *buf, int len, unsigned int gup_flags)
+{
+	int bytes, err;
 
-		buf += bytes - 1;
-		/*
-		 * Because strscpy always NUL terminates we need to
-		 * copy the last byte in the page if we are going to
-		 * load more pages
-		 */
-		if (bytes != len) {
-			addr += bytes - 1;
-			copy_from_user_page(vma, page, addr, buf, maddr + (PAGE_SIZE - 1), 1);
-			buf += 1;
-			addr += 1;
-		}
-		len -= bytes;
+	*(char *)buf = '\0';
 
-		folio_release_kmap(folio, maddr);
+	bytes = remote_vm_walk(mm, addr, buf, len, gup_flags, false,
+			       copy_vm_str, &err);
+	if (err) {
+		/* The contract guarantees a terminated buffer even on error. */
+		((char *)buf)[bytes] = '\0';
+		return err;
 	}
 
-out:
-	mmap_read_unlock(mm);
-	if (err)
-		return err;
-	return buf - old_buf;
+	return bytes;
 }
 
 /**
Signature · 1 line
2.53.0-Meta
author reviewer
#19 lore raw

RFC PATCH v4 8/12 selftests/mm: cover /proc/pid/mem access to VM_PFNMAP memory

In reply to Rik van Riel

Reading a VM_PFNMAP mapping through /proc/pid/mem exercises __access_remote_vm() two ways: a COWed page has a struct page and is returned by get_user_page_vma(), while a raw PFN has none and is reached through vma->vm_ops->access().

Add two tests to pfnmap.c, both reading VM_PFNMAP memory through /proc/self/mem.

procmem_cow_read maps the file MAP_PRIVATE and writable, writes to COW a page, then reads it back. Without the struct-page path in get_user_page_vma() this read is short: the access falls back to generic_access_phys(), which ioremaps the PFN, and ioremap of a COWed RAM page is rejected.

procmem_pfn_read reads a raw PFN back through ->access(). ioremap rejects RAM, so it runs only for genuine device memory and is skipped for the default /dev/mem System RAM target.

Assisted-by Claude:claude-opus-4.8
---
 tools/testing/selftests/mm/pfnmap.c | 66 +++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
tools/testing/selftests/mm/pfnmap.c +66 -0
index 4f550822385a..6ff5d1029517 100644
@@ -31,6 +31,7 @@ static sigjmp_buf sigjmp_buf_env;
 static char *file = "/dev/mem";
 static off_t file_offset;
 static int fd;
+static int target_is_ram;
 
 static void signal_handler(int sig)
 {
@@ -113,6 +114,7 @@ static void pfnmap_init(void)
 		if (err)
 			ksft_exit_skip("Cannot find ram target in '/proc/iomem': %s\n",
 				       strerror(-err));
+		target_is_ram = 1;
 	} else {
 		file_offset = 0;
 	}
@@ -271,6 +273,70 @@ TEST_F(pfnmap, fork)
 	ASSERT_EQ(ret, 0);
 }
 
+TEST_F(pfnmap, procmem_cow_read)
+{
+	char *priv, *buf;
+	ssize_t rc;
+	int mem_fd;
+
+	/*
+	 * A COWed page in a VM_PFNMAP mapping has a struct page, so reading it
+	 * through /proc/self/mem -- __access_remote_vm() -> get_user_page_vma()
+	 * -- returns it directly, instead of routing to vma->vm_ops->access(),
+	 * which ioremaps the PFN and cannot reach a COWed RAM page.
+	 *
+	 * Map the file MAP_PRIVATE and writable, write to COW a page into anon
+	 * memory, then read the page back through /proc/self/mem.
+	 */
+	self->size2 = self->pagesize;
+	self->addr2 = mmap(NULL, self->size2, PROT_READ | PROT_WRITE,
+			   MAP_PRIVATE, fd, file_offset);
+	if (self->addr2 == MAP_FAILED)
+		SKIP(return, "Cannot create a writable private pfnmap mapping");
+	priv = self->addr2;
+
+	/* COW the page and stamp known bytes into the anon copy. */
+	priv[0] = 0x42;
+	priv[self->pagesize - 1] = 0x24;
+
+	buf = malloc(self->pagesize);
+	ASSERT_NE(buf, NULL);
+
+	mem_fd = open("/proc/self/mem", O_RDONLY);
+	ASSERT_GE(mem_fd, 0);
+	rc = pread(mem_fd, buf, self->pagesize, (off_t)(uintptr_t)priv);
+	close(mem_fd);
+
+	ASSERT_EQ(rc, (ssize_t)self->pagesize);
+	EXPECT_EQ(buf[0], 0x42);
+	EXPECT_EQ(buf[self->pagesize - 1], 0x24);
+
+	free(buf);
+}
+
+TEST_F(pfnmap, procmem_pfn_read)
+{
+	char buf[64];
+	ssize_t rc;
+	int mem_fd;
+
+	/*
+	 * A raw PFN of a VM_IO/VM_PFNMAP mapping has no struct page, so
+	 * __access_remote_vm() reaches it through vma->vm_ops->access()
+	 * (generic_access_phys()). That ioremaps the PFN, which is rejected for
+	 * RAM, so this only applies to genuine device memory.
+	 */
+	if (target_is_ram)
+		SKIP(return, "Target is System RAM; ->access() cannot ioremap RAM");
+
+	mem_fd = open("/proc/self/mem", O_RDONLY);
+	ASSERT_GE(mem_fd, 0);
+	rc = pread(mem_fd, buf, sizeof(buf), (off_t)(uintptr_t)self->addr1);
+	close(mem_fd);
+
+	ASSERT_EQ(rc, (ssize_t)sizeof(buf));
+}
+
 int main(int argc, char **argv)
 {
 	for (int i = 1; i < argc; i++) {
Signature · 1 line
2.53.0-Meta
reviewer
#20 lore raw

Re: [PATCH RFC v4 08/12] selftests/mm: cover /proc/pid/mem access to VM_PFNMAP memory

Quoted 5 lines · Rik van Riel

Reading a VM_PFNMAP mapping through /proc/pid/mem exercises __access_remote_vm() two ways: a COWed page has a struct page and is returned by get_user_page_vma(), while a raw PFN has none and is reached through vma->vm_ops->access().

Quoted 8 lines

Add two tests to pfnmap.c, both reading VM_PFNMAP memory through /proc/self/mem.

procmem_cow_read maps the file MAP_PRIVATE and writable, writes to COW a page, then reads it back. Without the struct-page path in get_user_page_vma() this read is short: the access falls back to generic_access_phys(), which ioremaps the PFN, and ioremap of a COWed RAM page is rejected.

This is architecture-dependent. RISC-V uses generic_ioremap_prot(), which does not reject RAM, so the old ->access() path can read the COWed page and this test can pass without exercising get_user_page_vma().

The raw test has the same observability problem: checking only the byte count cannot distinguish ->access() from an erroneous GUP result for a PFNMAP of normal RAM.

Should we instead use a deterministic PFNMAP test provider whose mapped page contains one pattern and whose ->access() callback returns another? The COW test could require the anonymous data and the raw test could require the ->access() pattern. That would also avoid reading arbitrary MMIO and skipping the raw case in the default run.

Quoted 74 lines

procmem_pfn_read reads a raw PFN back through ->access(). ioremap rejects RAM, so it runs only for genuine device memory and is skipped for the default /dev/mem System RAM target.

Assisted-by Claude:claude-opus-4.8
---
 tools/testing/selftests/mm/pfnmap.c | 66 +++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
tools/testing/selftests/mm/pfnmap.c +41 -0
index 4f550822385a..6ff5d1029517 100644
@@ -31,6 +31,7 @@ static sigjmp_buf sigjmp_buf_env;
 static char *file = "/dev/mem";
 static off_t file_offset;
 static int fd;
+static int target_is_ram;
 
 static void signal_handler(int sig)
 {
@@ -113,6 +114,7 @@ static void pfnmap_init(void)
 		if (err)
 			ksft_exit_skip("Cannot find ram target in '/proc/iomem': %s\n",
 				       strerror(-err));
+		target_is_ram = 1;
 	} else {
 		file_offset = 0;
 	}
@@ -271,6 +273,70 @@ TEST_F(pfnmap, fork)
 	ASSERT_EQ(ret, 0);
 }
 
+TEST_F(pfnmap, procmem_cow_read)
+{
+	char *priv, *buf;
+	ssize_t rc;
+	int mem_fd;
+
+	/*
+	 * A COWed page in a VM_PFNMAP mapping has a struct page, so reading it
+	 * through /proc/self/mem -- __access_remote_vm() -> get_user_page_vma()
+	 * -- returns it directly, instead of routing to vma->vm_ops->access(),
+	 * which ioremaps the PFN and cannot reach a COWed RAM page.
+	 *
+	 * Map the file MAP_PRIVATE and writable, write to COW a page into anon
+	 * memory, then read the page back through /proc/self/mem.
+	 */
+	self->size2 = self->pagesize;
+	self->addr2 = mmap(NULL, self->size2, PROT_READ | PROT_WRITE,
+			   MAP_PRIVATE, fd, file_offset);
+	if (self->addr2 == MAP_FAILED)
+		SKIP(return, "Cannot create a writable private pfnmap mapping");
+	priv = self->addr2;
+
+	/* COW the page and stamp known bytes into the anon copy. */
+	priv[0] = 0x42;
+	priv[self->pagesize - 1] = 0x24;
+
+	buf = malloc(self->pagesize);
+	ASSERT_NE(buf, NULL);
+
+	mem_fd = open("/proc/self/mem", O_RDONLY);
+	ASSERT_GE(mem_fd, 0);
+	rc = pread(mem_fd, buf, self->pagesize, (off_t)(uintptr_t)priv);
+	close(mem_fd);
+
+	ASSERT_EQ(rc, (ssize_t)self->pagesize);
+	EXPECT_EQ(buf[0], 0x42);
+	EXPECT_EQ(buf[self->pagesize - 1], 0x24);
+
+	free(buf);

tools/testing/selftests/mm/pfnmap.c TEST_F(pfnmap, fork) · line 324

Quoted 10 lines
tools/testing/selftests/mm/pfnmap.c +10 -0
+}
+
+TEST_F(pfnmap, procmem_pfn_read)
+{
+	char buf[64];
+	ssize_t rc;
+	int mem_fd;
+
+	/*
+	 * A raw PFN of a VM_IO/VM_PFNMAP mapping has no struct page, so

That is not the VM_PFNMAP invariant. A raw mapping is treated as special even when its PFN has a valid struct page. Please describe this as GUP not returning a page for the raw mapping, here and in patches 5 and 6.

The new pread() offsets also need 64-bit off_t. On a 32-bit ABI, a mapping at or above 2 GiB converts to a negative off_t, and pread64() rejects it before /proc/self/mem sees the offset.

thanks,

Signature · 1 line
John Hubbard
author reviewer
#21 lore raw

RFC PATCH v4 9/12 mm/gup: build get_user_page_lookup_vma() on get_user_page_vma()

In reply to Rik van Riel

get_user_page_lookup_vma() faults in one page of a remote mm and returns it together with the VMA that covers it. It open-codes that with get_user_pages_remote() followed by vma_lookup(), duplicating the single-page walk that get_user_page_vma() now provides.

Every caller already holds the mmap lock, and the helper needs it anyway:
get_user_pages_remote(locked=NULL) and vma_lookup() both rely on it.

Look the VMA up under the held mmap lock and fault the page in through get_user_page_vma() without FOLL_VMA_LOCK, so the lock is held for the whole call and never dropped. Force FOLL_REMOTE and FOLL_TOUCH to preserve the foreign-access permission check and accessed-bit behavior that get_user_pages_remote() applied before.

Add mmap_assert_locked() so the lock requirement is documented and any future caller that forgets it trips the assertion rather than walking page tables unlocked.

Reject FOLL_UNLOCKABLE alongside the existing FOLL_NOWAIT check: both let the fault handler drop the mmap lock, which would invalidate the VMA looked up here. get_user_page_vma() passes neither, so the lock is held for the whole walk and the returned VMA stays valid.

The one behavioral change is that the old path could drop and retake the mmap lock across a fault, while get_user_page_vma() holds it throughout. None of the callers rely on the lock being dropped.

Assisted-by Claude:claude-opus-4.8
---
 include/linux/mm.h | 32 +++-----------------------------
 mm/gup.c           | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+), 29 deletions(-)
include/linux/mm.h +3 -29
index 24ead14b4790..0f66d76e6ca7 100644
@@ -3235,35 +3235,9 @@ long pin_user_pages_remote(struct mm_struct *mm,
 			   unsigned int gup_flags, struct page **pages,
 			   int *locked);
 
-/*
- * Retrieves a single page alongside its VMA. Does not support FOLL_NOWAIT.
- */
-static inline struct page *get_user_page_lookup_vma(struct mm_struct *mm,
-						    unsigned long addr,
-						    int gup_flags,
-						    struct vm_area_struct **vmap)
-{
-	struct page *page;
-	struct vm_area_struct *vma;
-	int got;
-
-	if (WARN_ON_ONCE(unlikely(gup_flags & FOLL_NOWAIT)))
-		return ERR_PTR(-EINVAL);
-
-	got = get_user_pages_remote(mm, addr, 1, gup_flags, &page, NULL);
-
-	if (got < 0)
-		return ERR_PTR(got);
-
-	vma = vma_lookup(mm, addr);
-	if (WARN_ON_ONCE(!vma)) {
-		put_page(page);
-		return ERR_PTR(-EINVAL);
-	}
-
-	*vmap = vma;
-	return page;
-}
+struct page *get_user_page_lookup_vma(struct mm_struct *mm, unsigned long addr,
+				      int gup_flags,
+				      struct vm_area_struct **vmap);
 
 long get_user_pages(unsigned long start, unsigned long nr_pages,
 		    unsigned int gup_flags, struct page **pages);
mm/gup.c +35 -0
index cb7245b7542f..7f4107f7ff10 100644
@@ -1288,6 +1288,41 @@ struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,
 	return ERR_PTR(ret);
 }
 
+/*
+ * get_user_page_lookup_vma - fault in one page of a remote mm and hand back the
+ * page along with the VMA that covers it. The caller must hold the mmap_lock.
+ * Returns with the mmap_lock still held.
+ *
+ * Looks up the VMA, and gets a reference to the page through
+ * get_user_page_vma(), faulting in the page if needed.
+ *
+ * FOLL_NOWAIT and FOLL_UNLOCKABLE are rejected: both let the fault handler
+ * drop the mmap lock, which could invalidate the looked-up VMA.
+ */
+struct page *get_user_page_lookup_vma(struct mm_struct *mm, unsigned long addr,
+				      int gup_flags,
+				      struct vm_area_struct **vmap)
+{
+	struct vm_area_struct *vma;
+	struct page *page;
+
+	if (WARN_ON_ONCE(unlikely(gup_flags & (FOLL_NOWAIT | FOLL_UNLOCKABLE))))
+		return ERR_PTR(-EINVAL);
+
+	mmap_assert_locked(mm);
+
+	vma = vma_lookup(mm, addr);
+	if (!vma)
+		return ERR_PTR(-EFAULT);
+
+	page = get_user_page_vma(vma, addr, gup_flags | FOLL_REMOTE | FOLL_TOUCH);
+	if (IS_ERR(page))
+		return page;
+
+	*vmap = vma;
+	return page;
+}
+
 /*
  * Writing to file-backed mappings which require folio dirty tracking using GUP
  * is a fundamentally broken operation, as kernel write access to GUP mappings
Signature · 1 line
2.53.0-Meta
author reviewer
#22 lore raw

RFC PATCH v4 10/12 mm/gup: pass an end address to follow_page_mask() and return a page count

In reply to Rik van Riel

follow_page_mask() reports how many pages the returned page covers with a
page_mask: a bitmask of the enclosing naturally-aligned huge page. Callers turn that into a stride, which assumes the run is a power-of-two block aligned to its own size.

That form cannot describe an arbitrary contiguous run, which a later patch needs to batch PTE-mapped large folios.

Replace the page_mask output with nr_pages, the number of contiguous pages the returned page covers starting at @address, and add an @end argument bounding how far the walk looks, so a small request does not scan an entire large folio.

The huge PMD and PUD paths report the same stride as before, now as a count clamped to @end. __get_user_pages() uses the count directly. get_user_page_vma() passes @address + PAGE_SIZE and still returns one page.

No functional change intended.

Assisted-by Claude:claude-opus-4.8
---
 mm/gup.c | 102 ++++++++++++++++++++++++++++++-------------------------
 1 file changed, 56 insertions(+), 46 deletions(-)
mm/gup.c +56 -46
index 7f4107f7ff10..96b00ddf7e04 100644
@@ -647,8 +647,9 @@ static inline bool can_follow_write_pud(pud_t pud, struct page *page,
 }
 
 static struct page *follow_huge_pud(struct vm_area_struct *vma,
-				    unsigned long addr, pud_t *pudp,
-				    int flags, unsigned long *page_mask)
+				    unsigned long addr, unsigned long end,
+				    pud_t *pudp, int flags,
+				    unsigned long *nr_pages)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	struct page *page;
@@ -672,10 +673,13 @@ static struct page *follow_huge_pud(struct vm_area_struct *vma,
 		return ERR_PTR(-EMLINK);
 
 	ret = try_grab_folio(page_folio(page), 1, flags);
-	if (ret)
+	if (ret) {
 		page = ERR_PTR(ret);
-	else
-		*page_mask = HPAGE_PUD_NR - 1;
+	} else {
+		unsigned long off = (addr & ~PUD_MASK) >> PAGE_SHIFT;
+
+		*nr_pages = min(HPAGE_PUD_NR - off, (end - addr) >> PAGE_SHIFT);
+	}
 
 	return page;
 }
@@ -699,9 +703,9 @@ static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page,
 }
 
 static struct page *follow_huge_pmd(struct vm_area_struct *vma,
-				    unsigned long addr, pmd_t *pmd,
-				    unsigned int flags,
-				    unsigned long *page_mask)
+				    unsigned long addr, unsigned long end,
+				    pmd_t *pmd, unsigned int flags,
+				    unsigned long *nr_pages)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	pmd_t pmdval = *pmd;
@@ -738,23 +742,25 @@ static struct page *follow_huge_pmd(struct vm_area_struct *vma,
 #endif	/* CONFIG_TRANSPARENT_HUGEPAGE */
 
 	page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;
-	*page_mask = HPAGE_PMD_NR - 1;
+	*nr_pages = min(HPAGE_PMD_NR - ((addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT),
+			(end - addr) >> PAGE_SHIFT);
 
 	return page;
 }
 
 #else  /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */
 static struct page *follow_huge_pud(struct vm_area_struct *vma,
-				    unsigned long addr, pud_t *pudp,
-				    int flags, unsigned long *page_mask)
+				    unsigned long addr, unsigned long end,
+				    pud_t *pudp, int flags,
+				    unsigned long *nr_pages)
 {
 	return NULL;
 }
 
 static struct page *follow_huge_pmd(struct vm_area_struct *vma,
-				    unsigned long addr, pmd_t *pmd,
-				    unsigned int flags,
-				    unsigned long *page_mask)
+				    unsigned long addr, unsigned long end,
+				    pmd_t *pmd, unsigned int flags,
+				    unsigned long *nr_pages)
 {
 	return NULL;
 }
@@ -800,7 +806,8 @@ static inline bool can_follow_write_pte(pte_t pte, struct page *page,
 }
 
 static struct page *follow_page_pte(struct vm_area_struct *vma,
-		unsigned long address, pmd_t *pmd, unsigned int flags)
+		unsigned long address, unsigned long end, pmd_t *pmd,
+		unsigned int flags, unsigned long *nr_pages)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	struct folio *folio;
@@ -885,6 +892,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
 		 */
 		folio_mark_accessed(folio);
 	}
+
 out:
 	pte_unmap_unlock(ptep, ptl);
 	return page;
@@ -896,9 +904,9 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
 }
 
 static struct page *follow_pmd_mask(struct vm_area_struct *vma,
-				    unsigned long address, pud_t *pudp,
-				    unsigned int flags,
-				    unsigned long *page_mask)
+				    unsigned long address, unsigned long end,
+				    pud_t *pudp, unsigned int flags,
+				    unsigned long *nr_pages)
 {
 	pmd_t *pmd, pmdval;
 	spinlock_t *ptl;
@@ -912,7 +920,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma,
 	if (!pmd_present(pmdval))
 		return no_page_table(vma, flags, address);
 	if (likely(!pmd_leaf(pmdval)))
-		return follow_page_pte(vma, address, pmd, flags);
+		return follow_page_pte(vma, address, end, pmd, flags, nr_pages);
 
 	if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags))
 		return no_page_table(vma, flags, address);
@@ -925,24 +933,24 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma,
 	}
 	if (unlikely(!pmd_leaf(pmdval))) {
 		spin_unlock(ptl);
-		return follow_page_pte(vma, address, pmd, flags);
+		return follow_page_pte(vma, address, end, pmd, flags, nr_pages);
 	}
 	if (pmd_trans_huge(pmdval) && (flags & FOLL_SPLIT_PMD)) {
 		spin_unlock(ptl);
 		split_huge_pmd(vma, pmd, address);
 		/* If pmd was left empty, stuff a page table in there quickly */
 		return pte_alloc(mm, pmd) ? ERR_PTR(-ENOMEM) :
-			follow_page_pte(vma, address, pmd, flags);
+			follow_page_pte(vma, address, end, pmd, flags, nr_pages);
 	}
-	page = follow_huge_pmd(vma, address, pmd, flags, page_mask);
+	page = follow_huge_pmd(vma, address, end, pmd, flags, nr_pages);
 	spin_unlock(ptl);
 	return page;
 }
 
 static struct page *follow_pud_mask(struct vm_area_struct *vma,
-				    unsigned long address, p4d_t *p4dp,
-				    unsigned int flags,
-				    unsigned long *page_mask)
+				    unsigned long address, unsigned long end,
+				    p4d_t *p4dp, unsigned int flags,
+				    unsigned long *nr_pages)
 {
 	pud_t *pudp, pud;
 	spinlock_t *ptl;
@@ -955,7 +963,7 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma,
 		return no_page_table(vma, flags, address);
 	if (pud_leaf(pud)) {
 		ptl = pud_lock(mm, pudp);
-		page = follow_huge_pud(vma, address, pudp, flags, page_mask);
+		page = follow_huge_pud(vma, address, end, pudp, flags, nr_pages);
 		spin_unlock(ptl);
 		if (page)
 			return page;
@@ -964,13 +972,13 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma,
 	if (unlikely(pud_bad(pud)))
 		return no_page_table(vma, flags, address);
 
-	return follow_pmd_mask(vma, address, pudp, flags, page_mask);
+	return follow_pmd_mask(vma, address, end, pudp, flags, nr_pages);
 }
 
 static struct page *follow_p4d_mask(struct vm_area_struct *vma,
-				    unsigned long address, pgd_t *pgdp,
-				    unsigned int flags,
-				    unsigned long *page_mask)
+				    unsigned long address, unsigned long end,
+				    pgd_t *pgdp, unsigned int flags,
+				    unsigned long *nr_pages)
 {
 	p4d_t *p4dp, p4d;
 
@@ -981,15 +989,16 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma,
 	if (!p4d_present(p4d) || p4d_bad(p4d))
 		return no_page_table(vma, flags, address);
 
-	return follow_pud_mask(vma, address, p4dp, flags, page_mask);
+	return follow_pud_mask(vma, address, end, p4dp, flags, nr_pages);
 }
 
 /**
  * follow_page_mask - look up a page descriptor from a user-virtual address
  * @vma: vm_area_struct mapping @address
  * @address: virtual address to look up
+ * @end: virtual address at which to stop batching contiguous pages
  * @flags: flags modifying lookup behaviour
- * @page_mask: a pointer to output page_mask
+ * @nr_pages: output; number of contiguous pages the caller can read
  *
  * @flags can have FOLL_ flags set, defined in <linux/mm.h>
  *
@@ -998,15 +1007,17 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma,
  * trigger a fault with FAULT_FLAG_UNSHARE set. Note that unsharing is only
  * relevant with FOLL_PIN and !FOLL_WRITE.
  *
- * On output, @page_mask is set according to the size of the page.
+ * On output, @nr_pages holds how many contiguous pages the folio that includes
+ * the returned page has mapped into this process, so the caller can advance
+ * over a large folio in one step.
  *
  * Return: the mapped (struct page *), %NULL if no mapping exists, or
  * an error pointer if there is a mapping to something not represented
  * by a page descriptor (see also vm_normal_page()).
  */
 static struct page *follow_page_mask(struct vm_area_struct *vma,
-			      unsigned long address, unsigned int flags,
-			      unsigned long *page_mask)
+			      unsigned long address, unsigned long end,
+			      unsigned int flags, unsigned long *nr_pages)
 {
 	pgd_t *pgd;
 	struct mm_struct *mm = vma->vm_mm;
@@ -1014,13 +1025,13 @@ static struct page *follow_page_mask(struct vm_area_struct *vma,
 
 	vma_pgtable_walk_begin(vma);
 
-	*page_mask = 0;
+	*nr_pages = 1;
 	pgd = pgd_offset(mm, address);
 
 	if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd)))
 		page = no_page_table(vma, flags, address);
 	else
-		page = follow_p4d_mask(vma, address, pgd, flags, page_mask);
+		page = follow_p4d_mask(vma, address, end, pgd, flags, nr_pages);
 
 	vma_pgtable_walk_end(vma);
 
@@ -1198,7 +1209,7 @@ struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,
 			       unsigned int gup_flags)
 {
 	bool vma_locked = gup_flags & FOLL_VMA_LOCK;
-	unsigned long page_mask;
+	unsigned long nr_pages;
 	struct page *page;
 	int locked = 1;
 	bool pfnmap;
@@ -1231,10 +1242,10 @@ struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,
 		}
 		cond_resched();
 
-		/* follow_page_mask() requires @page_mask; it is unused here. */
-		page = follow_page_mask(vma, addr,
+		/* This helper hands back a single page; cap the batch at one. */
+		page = follow_page_mask(vma, addr, addr + PAGE_SIZE,
 					gup_flags | FOLL_TOUCH | FOLL_GET,
-					&page_mask);
+					&nr_pages);
 		if (!IS_ERR_OR_NULL(page)) {
 			/* Match __get_user_pages(): flush for VIVT/aliasing caches. */
 			flush_anon_page(vma, page, addr);
@@ -1529,7 +1540,6 @@ static long __get_user_pages(struct mm_struct *mm,
 {
 	long ret = 0, i = 0;
 	struct vm_area_struct *vma = NULL;
-	unsigned long page_mask = 0;
 
 	if (!nr_pages)
 		return 0;
@@ -1544,7 +1554,7 @@ static long __get_user_pages(struct mm_struct *mm,
 
 	do {
 		struct page *page;
-		unsigned int page_increm;
+		unsigned long page_increm;
 
 		/* first iteration or cross vma bound */
 		if (!vma || start >= vma->vm_end) {
@@ -1571,7 +1581,7 @@ static long __get_user_pages(struct mm_struct *mm,
 						pages ? &page : NULL);
 				if (ret)
 					goto out;
-				page_mask = 0;
+				page_increm = 1;
 				goto next_page;
 			}
 
@@ -1594,7 +1604,8 @@ static long __get_user_pages(struct mm_struct *mm,
 		}
 		cond_resched();
 
-		page = follow_page_mask(vma, start, gup_flags, &page_mask);
+		page = follow_page_mask(vma, start, start + nr_pages * PAGE_SIZE,
+					gup_flags, &page_increm);
 		if (!page || PTR_ERR(page) == -EMLINK) {
 			ret = faultin_page(vma, start, gup_flags,
 					   PTR_ERR(page) == -EMLINK, locked);
@@ -1627,7 +1638,6 @@ static long __get_user_pages(struct mm_struct *mm,
 			goto out;
 		}
 next_page:
-		page_increm = 1 + (~(start >> PAGE_SHIFT) & page_mask);
 		if (page_increm > nr_pages)
 			page_increm = nr_pages;
 
Signature · 1 line
2.53.0-Meta
author reviewer
#23 lore raw

RFC PATCH v4 11/12 mm/gup: batch contiguous PTE-mapped large folios in follow_page_mask()

In reply to Rik van Riel

follow_page_mask() returns one page per call for a PTE-mapped large folio, so __get_user_pages() re-walks the page tables for every page of an mTHP even though the folio maps a contiguous run. The huge PMD and PUD paths already return the whole mapping in one step.

Report the contiguous run for the PTE case too. follow_pte_batch() uses folio_pte_batch_flags() to count consecutive present PTEs that map consecutive pages of the same folio with a uniform write bit, bounded by the page table, @end, the VMA, and the folio itself.

Keep the per-PTE guarantees that follow_page_pte() makes for the head page. folio_pte_batch_flags() with FPB_RESPECT_WRITE stops the run at a change in the write bit, so the whole run matches the head.

A writable run is safe for any access: a writable anon page is exclusive, so gup_must_unshare() cannot fire, and FOLL_WRITE is satisfied.

A read-only run is batched only for a plain read, since FOLL_WRITE would need a COW fault per page and FOLL_PIN would need a per-page gup_must_unshare() check.

Measured with mm/gup_test.c (PIN_LONGTERM_BENCHMARK, the slow pin_user_pages() path) on a 256 MB MADV_HUGEPAGE anonymous region in a 4 CPU VM, median get time over 16 iterations. Each folio size was confirmed through the per-size anon_fault_alloc counters (4096 folios for 64 kB, 128 for 2 MB):

gup_test -L -m 256 -n 65536 -r 16 -t
before after
64 kB mTHP 3140 us 412 us (7.6x)
2 MB THP (control) 78 us 76 us
4 kB base (control) 3010 us 3042 us

The PMD-mapped 2 MB THP already returns the whole mapping in one step, so it stays fast and unchanged. The 4 kB baseline shows the per-page walk cost that the 64 kB case paid before this change; only the PTE-mapped large folio case improves.

Assisted-by Claude:claude-opus-4.8
---
 mm/gup.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
mm/gup.c +45 -0
index 96b00ddf7e04..4910c93ba003 100644
@@ -805,6 +805,43 @@ static inline bool can_follow_write_pte(pte_t pte, struct page *page,
 	return !userfaultfd_pte_wp(vma, pte);
 }
 
+/*
+ * Count the pages, starting at @address and bounded by @end, that a PTE-mapped
+ * large @folio maps contiguously and that can be returned together with the
+ * page at @address: consecutive present PTEs mapping consecutive pages of
+ * @folio with a uniform write bit, within this VMA and a single page table.
+ * Returns at least 1.
+ *
+ * gup_must_unshare() and the write-fault check are per PTE. A writable run is
+ * always safe: a writable anon page is exclusive, and FOLL_WRITE is satisfied.
+ * A read-only run is only safe for a plain read; FOLL_WRITE would need a COW
+ * fault per page and FOLL_PIN would need a per-page gup_must_unshare() check,
+ * so those fall back to a single page.
+ */
+static unsigned long follow_pte_batch(struct vm_area_struct *vma,
+		unsigned long address, unsigned long end, struct folio *folio,
+		struct page *page, pte_t *ptep, pte_t pte, unsigned int flags)
+{
+	pte_t batch_pte = pte;
+	unsigned long max;
+
+	if (!pte_write(pte) && (flags & (FOLL_WRITE | FOLL_PIN)))
+		return 1;
+
+	/*
+	 * folio_pte_batch_flags() scans forward from @ptep, so the run must
+	 * stay within this page table: bound it by the PMD as well as @end and
+	 * the VMA, since a large folio can be PTE-mapped across a PMD boundary.
+	 */
+	max = min((pmd_addr_end(address, end) - address) >> PAGE_SHIFT,
+		  (vma->vm_end - address) >> PAGE_SHIFT);
+	if (max <= 1)
+		return 1;
+
+	return folio_pte_batch_flags(folio, vma, ptep, &batch_pte, max,
+				     FPB_RESPECT_WRITE);
+}
+
 static struct page *follow_page_pte(struct vm_area_struct *vma,
 		unsigned long address, unsigned long end, pmd_t *pmd,
 		unsigned int flags, unsigned long *nr_pages)
@@ -893,6 +930,14 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
 		folio_mark_accessed(folio);
 	}
 
+	/*
+	 * A PTE-mapped large folio can be handed back as a contiguous batch,
+	 * so the caller advances over the whole run in one step instead of
+	 * walking the page tables for every page.
+	 */
+	if (folio_test_large(folio))
+		*nr_pages = follow_pte_batch(vma, address, end, folio, page,
+					     ptep, pte, flags);
 out:
 	pte_unmap_unlock(ptep, ptl);
 	return page;
Signature · 1 line
2.53.0-Meta
#24 lore raw

Re: [PATCH RFC v4 11/12] mm/gup: batch contiguous PTE-mapped large folios in follow_page_mask()

Quoted 33 lines · Rik van Riel

follow_page_mask() returns one page per call for a PTE-mapped large folio, so __get_user_pages() re-walks the page tables for every page of an mTHP even though the folio maps a contiguous run. The huge PMD and PUD paths already return the whole mapping in one step.

Report the contiguous run for the PTE case too. follow_pte_batch() uses folio_pte_batch_flags() to count consecutive present PTEs that map consecutive pages of the same folio with a uniform write bit, bounded by the page table, @end, the VMA, and the folio itself.

Keep the per-PTE guarantees that follow_page_pte() makes for the head page. folio_pte_batch_flags() with FPB_RESPECT_WRITE stops the run at a change in the write bit, so the whole run matches the head.

A writable run is safe for any access: a writable anon page is exclusive, so gup_must_unshare() cannot fire, and FOLL_WRITE is satisfied.

A read-only run is batched only for a plain read, since FOLL_WRITE would need a COW fault per page and FOLL_PIN would need a per-page gup_must_unshare() check.

Measured with mm/gup_test.c (PIN_LONGTERM_BENCHMARK, the slow pin_user_pages() path) on a 256 MB MADV_HUGEPAGE anonymous region in a 4 CPU VM, median get time over 16 iterations. Each folio size was confirmed through the per-size anon_fault_alloc counters (4096 folios for 64 kB, 128 for 2 MB):

gup_test -L -m 256 -n 65536 -r 16 -t
before after
64 kB mTHP 3140 us 412 us (7.6x)
2 MB THP (control) 78 us 76 us
4 kB base (control) 3010 us 3042 us

Quoted 7 lines

The PMD-mapped 2 MB THP already returns the whole mapping in one step, so it stays fast and unchanged. The 4 kB baseline shows the per-page walk cost that the 64 kB case paid before this change; only the PTE-mapped large folio case improves.

Assisted-by Claude:claude-opus-4.8

Why is this patch part of this patch set?

This makes perfect sense independently, no?

Signature · 3 lines
Cheers,

David
author reviewer
#25 lore raw

Re: [PATCH RFC v4 11/12] mm/gup: batch contiguous PTE-mapped large folios in follow_page_mask()

Quoted 16 lines · David Hildenbrand (Arm)
Quoted 15 lines · Rik van Riel

  gup_test -L -m 256 -n 65536 -r 16 -t before      after
  64 kB mTHP            3140 us      412 us   (7.6x) 2 MB THP (control)      78 us       76 us 4 kB base (control)   3010 us     3042 us

The PMD-mapped 2 MB THP already returns the whole mapping in one step, so
it stays fast and unchanged. The 4 kB baseline shows the per-page walk cost
that the 64 kB case paid before this change; only the PTE-mapped large
folio case improves.

Quoted 6 lines
Quoted 2 lines
Assisted-by Claude:claude-opus-4.8

Why is this patch part of this patch set?

This makes perfect sense independently, no?

I'm happy to drop this for now. I think it would fit in better with another optimization to only take one refcount per continuously mapped extent of pages from the same folio.

I can't really untangle it because it modifies some of the same code as other patches in the series, but I'm happy to save it for a follow-up.

Signature · 1 line
All Rights Reversed.
#26 lore raw

Re: [PATCH RFC v4 11/12] mm/gup: batch contiguous PTE-mapped large folios in follow_page_mask()

Quoted 24 lines · Rik van Riel
Quoted 16 lines · David Hildenbrand (Arm)
Quoted 15 lines · Rik van Riel

  gup_test -L -m 256 -n 65536 -r 16 -t before      after
  64 kB mTHP            3140 us      412 us   (7.6x) 2 MB THP (control)      78 us       76 us 4 kB base (control)   3010 us     3042 us

The PMD-mapped 2 MB THP already returns the whole mapping in one step, so
it stays fast and unchanged. The 4 kB baseline shows the per-page walk cost
that the 64 kB case paid before this change; only the PTE-mapped large
folio case improves.

Quoted 6 lines
Quoted 2 lines
Assisted-by Claude:claude-opus-4.8

Why is this patch part of this patch set?

This makes perfect sense independently, no?

Quoted 8 lines

I'm happy to drop this for now. I think it would fit in better with another optimization to only take one refcount per continuously mapped extent of pages from the same folio.

I can't really untangle it because it modifies some of the same code as other patches in the series, but I'm happy to save it for a follow-up.

Given this series here is RFC, can't we have that optimization upfront? I'd assume it will benefit quite some use cases that pin multiple consecutive pages.

IIUC, it'd be mostly patch 10+11. No need to worry about the test case right now, we can do that later.

Signature · 3 lines
Cheers,

David
author reviewer
#27 lore raw

Re: [PATCH RFC v4 11/12] mm/gup: batch contiguous PTE-mapped large folios in follow_page_mask()

Quoted 37 lines · David Hildenbrand (Arm)
Quoted 27 lines · Rik van Riel
Quoted 19 lines · David Hildenbrand (Arm)
Quoted 18 lines · Rik van Riel

  gup_test -L -m 256 -n 65536 -r 16 -t before      after
  64 kB mTHP            3140 us      412 us   (7.6x) 2 MB THP (control)      78 us       76 us 4 kB base (control)   3010 us     3042 us

The PMD-mapped 2 MB THP already returns the whole mapping in one
step, so
it stays fast and unchanged. The 4 kB baseline shows the per- page
walk cost
that the 64 kB case paid before this change; only the PTE- mapped
large
folio case improves.

Quoted 6 lines
Quoted 2 lines
Assisted-by Claude:claude-opus-4.8

Why is this patch part of this patch set?

This makes perfect sense independently, no?

Quoted 8 lines

I'm happy to drop this for now. I think it would fit in better with another optimization to only take one refcount per continuously mapped extent of pages from the same folio.

I can't really untangle it because it modifies some of the same code as other patches in the series, but I'm happy to save it for a follow-up.

Quoted 8 lines

Given this series here is RFC, can't we have that optimization upfront? I'd
assume it will benefit quite some use cases that pin multiple consecutive pages.

IIUC, it'd be mostly patch 10+11. No need to worry about the test case right
now, we can do that later.

I'm happy to send patches 10-11 by themselves.

We can save a self test for after the future optimization of only taking a single refcount for an entire extent of pages from the same folio mapped in the same process, at which point we'll have something more serious to test.

I'll get these to you soon.

Signature · 1 line
All Rights Reversed.
author reviewer
#28 lore raw

RFC PATCH v4 12/12 selftests/mm: add a slow-GUP content and COW test for mTHP

In reply to Rik van Riel

follow_page_mask() now batches a PTE-mapped large folio (mTHP) into one contiguous run for the slow get_user_pages() path. A mis-batched run would hand back the wrong pages or a stale COW copy, which the existing tests do not catch: gup_test checks only pin/unpin integrity, and cow exercises COW mostly at PMD size.

Add mthp_gup_cow_test. It forces 64kB-only mTHP, writes a per-page-distinct pattern, then pins the region on the slow path (PIN_LONGTERM without USE_FAST) and compares the bytes the kernel copies back from the pinned pages against that pattern.

The test covers a read pin, a write pin, and COW after fork(): a child write-pins to force per-page unshare and checks the copied contents, then rewrites its copy while the parent verifies its own contents are intact.

It also reports how many pages sit in contiguous large-folio runs, so a kernel without mTHP shows light coverage rather than passing vacuously.

Assisted-by Claude:claude-opus-4.8
---
 tools/testing/selftests/mm/Makefile           |   1 +
 .../testing/selftests/mm/mthp_gup_cow_test.c  | 213 ++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh     |   1 +
 3 files changed, 215 insertions(+)
 create mode 100644 tools/testing/selftests/mm/mthp_gup_cow_test.c
tools/testing/selftests/mm/Makefile +1 -0
index e6df968f0971..6b917a4f73e0 100644
@@ -83,6 +83,7 @@ TEST_GEN_FILES += mrelease_test
 TEST_GEN_FILES += mremap_dontunmap
 TEST_GEN_FILES += mremap_test
 TEST_GEN_FILES += mseal_test
+TEST_GEN_FILES += mthp_gup_cow_test
 TEST_GEN_FILES += on-fault-limit
 TEST_GEN_FILES += pagemap_ioctl
 TEST_GEN_FILES += pfnmap
tools/testing/selftests/mm/mthp_gup_cow_test.c new file +213 -0

213 lines not on this page · raw message

tools/testing/selftests/mm/run_vmtests.sh +1 -0
index 8c296dedf047..b49a3eb0c205 100755
@@ -289,6 +289,7 @@ fi
 # Dump pages 0, 19, and 4096, using pin_user_pages:
 CATEGORY="gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000
 CATEGORY="gup_test" run_test ./gup_longterm
+CATEGORY="gup_test" run_test ./mthp_gup_cow_test
 
 CATEGORY="userfaultfd" run_test ./uffd-unit-tests
 uffd_stress_bin=./uffd-stress
Signature · 1 line
2.53.0-Meta
reviewer
#29 lore raw

Re: [PATCH RFC v4 12/12] selftests/mm: add a slow-GUP content and COW test for mTHP

Hi Rik,

Quoted 27 lines · Rik van Riel

follow_page_mask() now batches a PTE-mapped large folio (mTHP) into one contiguous run for the slow get_user_pages() path. A mis-batched run would hand back the wrong pages or a stale COW copy, which the existing tests do not catch: gup_test checks only pin/unpin integrity, and cow exercises COW mostly at PMD size.

Add mthp_gup_cow_test. It forces 64kB-only mTHP, writes a per-page-distinct pattern, then pins the region on the slow path (PIN_LONGTERM without USE_FAST) and compares the bytes the kernel copies back from the pinned pages against that pattern.

The test covers a read pin, a write pin, and COW after fork(): a child write-pins to force per-page unshare and checks the copied contents, then rewrites its copy while the parent verifies its own contents are intact.

It also reports how many pages sit in contiguous large-folio runs, so a kernel without mTHP shows light coverage rather than passing vacuously.

Assisted-by Claude:claude-opus-4.8
---
 tools/testing/selftests/mm/Makefile           |   1 +
 .../testing/selftests/mm/mthp_gup_cow_test.c  | 213 ++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh     |   1 +
 3 files changed, 215 insertions(+)
 create mode 100644 tools/testing/selftests/mm/mthp_gup_cow_test.c

tools/testing/selftests/mm/mthp_gup_cow_test.c line 50

Quoted 8 lines
tools/testing/selftests/mm/mthp_gup_cow_test.c new file +8 -0
+
+static long PS;
+static int fails;
+static int tap;
+
+static void ok(int cond, const char *desc)
+{
+	printf("%s %d %s\n", cond ? "ok" : "not ok", ++tap, desc);

Please (instruct claude to) use ksft_* helpers for printing messages and tracking failed and passed tests.

Quoted 7 lines
tools/testing/selftests/mm/mthp_gup_cow_test.c new file +7 -0
+	if (!cond)
+		fails++;
+}
+
+/* Deterministic, per-page-distinct pattern so any mis-order or leak shows. */
+static void fill(char *base, size_t sz, uint32_t salt)
+{

tools/testing/selftests/mm/mthp_gup_cow_test.c line 67

Quoted 10 lines
tools/testing/selftests/mm/mthp_gup_cow_test.c new file +10 -0
+	for (size_t off = 0; off < sz; off += PS) {
+		uint32_t k = off / PS;
+		uint64_t v = ((uint64_t)salt << 32) ^ (k * 0x9E3779B1u + 0x1234);
+
+		for (size_t i = 0; i < PS; i += sizeof(v))
+			memcpy(base + off + i, &v, sizeof(v));
+	}
+}
+
+static int wsysfs(const char *path, const char *val)

There are existing helpers to read/write sysfs in vm_util.h

Quoted 10 lines
tools/testing/selftests/mm/mthp_gup_cow_test.c new file +10 -0
+{
+	int fd = open(path, O_WRONLY);
+
+	if (fd < 0)
+		return -1;
+	int r = write(fd, val, strlen(val));
+
+	close(fd);
+	return r < 0 ? -1 : 0;
+}
Signature · 2 lines
Sincerely yours,
Mike.
#30 lore raw

Re: [PATCH RFC v4 12/12] selftests/mm: add a slow-GUP content and COW test for mTHP

Quoted 30 lines · Mike Rapoport

Hi Rik,

Quoted 27 lines · Rik van Riel

follow_page_mask() now batches a PTE-mapped large folio (mTHP) into one contiguous run for the slow get_user_pages() path. A mis-batched run would hand back the wrong pages or a stale COW copy, which the existing tests do not catch: gup_test checks only pin/unpin integrity, and cow exercises COW mostly at PMD size.

Add mthp_gup_cow_test. It forces 64kB-only mTHP, writes a per-page-distinct pattern, then pins the region on the slow path (PIN_LONGTERM without USE_FAST) and compares the bytes the kernel copies back from the pinned pages against that pattern.

The test covers a read pin, a write pin, and COW after fork(): a child write-pins to force per-page unshare and checks the copied contents, then rewrites its copy while the parent verifies its own contents are intact.

It also reports how many pages sit in contiguous large-folio runs, so a kernel without mTHP shows light coverage rather than passing vacuously.

Assisted-by Claude:claude-opus-4.8
---
 tools/testing/selftests/mm/Makefile           |   1 +
 .../testing/selftests/mm/mthp_gup_cow_test.c  | 213 ++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh     |   1 +
 3 files changed, 215 insertions(+)
 create mode 100644 tools/testing/selftests/mm/mthp_gup_cow_test.c
Quoted 11 lines

tools/testing/selftests/mm/mthp_gup_cow_test.c line 50

Quoted 8 lines
tools/testing/selftests/mm/mthp_gup_cow_test.c new file +8 -0
+
+static long PS;
+static int fails;
+static int tap;
+
+static void ok(int cond, const char *desc)
+{
+	printf("%s %d %s\n", cond ? "ok" : "not ok", ++tap, desc);

Please (instruct claude to) use ksft_* helpers for printing messages and tracking failed and passed tests.

I'm afraid we won't have time to keep repeating the following:

Documentation/process/coding-assistants.rst

* Reviewing all AI-generated code
[...]
* Taking full responsibility for the contribution

Signature · 3 lines
Cheers,

David
author reviewer
#31 lore raw

Re: [PATCH RFC v4 12/12] selftests/mm: add a slow-GUP content and COW test for mTHP

Quoted 6 lines · David Hildenbrand (Arm)
Quoted 4 lines · Mike Rapoport

Please (instruct claude to) use ksft_* helpers for printing messages and
tracking failed and passed tests.

Quoted 7 lines

I'm afraid we won't have time to keep repeating the following:

Documentation/process/coding-assistants.rst

* Reviewing all AI-generated code
[...]
* Taking full responsibility for the contribution

I did review the code. I missed this one, largely because I was focused more on the mm bits.

Not only will I clean it up, I will also make sure that other people using the available Linux kernel AI tooling will automatically avoid repeating this mistake.

I submitted some self test checks to review-prompts:

https://github.com/masoncl/review-prompts/pull/89

And added some commits to kernel-style to create clearer comments:

https://github.com/rikvanriel/kernel-style

I also tested the patch series against the updated review-prompts and kernel-style guidelines, and the issues that were pointed got caught with the updated tooling.

With the gigablock patch series, I tried to edit the default LLM output, and create something vaguely decent. That was a lost cause, which I do not plan to repeat.

It was an especially big waste of time with the code that I was never planning to keep, but where I had not thought of an alternative approach yet.

The current plan of attack is to add LLM rules every time I get a comment on a patch series that addresses something generalizable that I missed.

That way anybody using those tools should have those mistakes caught, and either raised or fixed by tooling, before they even start editing their code.

You don't want to review the same mistakes over and over again, and I want to help people avoid sending the same mistakes that were previously made.

Signature · 1 line
All Rights Reversed.
#32 lore raw

Re: [PATCH RFC v4 12/12] selftests/mm: add a slow-GUP content and COW test for mTHP

In reply to Rik van Riel
Quoted 11 lines · Rik van Riel

follow_page_mask() now batches a PTE-mapped large folio (mTHP) into one contiguous run for the slow get_user_pages() path. A mis-batched run would hand back the wrong pages or a stale COW copy, which the existing tests do not catch: gup_test checks only pin/unpin integrity, and cow exercises COW mostly at PMD size.

Add mthp_gup_cow_test. It forces 64kB-only mTHP, writes a per-page-distinct pattern, then pins the region on the slow path (PIN_LONGTERM without USE_FAST) and compares the bytes the kernel copies back from the pinned pages against that pattern.

Quoted 8 lines

The test covers a read pin, a write pin, and COW after fork(): a child write-pins to force per-page unshare and checks the copied contents, then rewrites its copy while the parent verifies its own contents are intact.

It also reports how many pages sit in contiguous large-folio runs, so a kernel without mTHP shows light coverage rather than passing vacuously.

Assisted-by Claude:claude-opus-4.8

I'm sorry, I cannot be convinced to review this AI slop here. You should have known better to no send such a ... patch ... out.

It completely violates to how we are supposed to write mm selftests in 2026.

Also, testing some implementation details is not really what we want.

Extend other tests to just naturally test mthp as well, please, if they don't already do that.

Having a separate selftest just for this is just extremely ... odd.

Signature · 3 lines
Cheers,

David
#33 lore raw
In reply to Rik van Riel
Quoted 7 lines · Rik van Riel

__access_remote_vm() holds mmap_read_lock() for the whole transfer. On large machines, with large multi-threaded applications, the mmap_lock is often contended, leading to things like reads of /proc/PID/cmdline stalling.

This results in system monitoring tools getting stuck, right when system information would be the most helpful.

Rik, what's the status of this, and which series do you consider more relevant:
the pure per-VMA lock change or the pure page-table walking + folio batching?

I'd assume the page-table walking would result in more speedup and the per-VMA in mostly less contention. So likely we'd want the page-table walking optimization first.

Signature · 3 lines
Cheers,

David
author reviewer
#34 lore raw
Quoted 17 lines · David Hildenbrand (Arm)
Quoted 10 lines · Rik van Riel

__access_remote_vm() holds mmap_read_lock() for the whole transfer. On
large machines, with large multi-threaded applications, the mmap_lock
is often contended, leading to things like reads of /proc/PID/cmdline
stalling.

This results in system monitoring tools getting stuck, right when system information would be the most helpful.

Rik, what's the status of this, and which series do you consider more relevant:
the pure per-VMA lock change or the pure page-table walking + folio batching?

Quoted 4 lines

I'd assume the page-table walking would result in more speedup and the per-VMA
in mostly less contention. So likely we'd want the page-table walking optimization first.

We can merge them in either order.

This series gives a speedup as well, by removing the double VMA lookup.

If you want to merge the page table walking and folio batching change first, I can easily rebase this per-VMA lock series on top of that.

Do you have additional feedback on changes you want me to make to get the page table walking and folio batching thing in a merge-ready shape?

Signature · 1 line
All Rights Reversed.
#35 lore raw
Quoted 28 lines · Rik van Riel
Quoted 17 lines · David Hildenbrand (Arm)
Quoted 10 lines · Rik van Riel

__access_remote_vm() holds mmap_read_lock() for the whole transfer. On
large machines, with large multi-threaded applications, the mmap_lock
is often contended, leading to things like reads of /proc/PID/cmdline
stalling.

This results in system monitoring tools getting stuck, right when system information would be the most helpful.

Rik, what's the status of this, and which series do you consider more relevant:
the pure per-VMA lock change or the pure page-table walking + folio batching?

Quoted 4 lines

I'd assume the page-table walking would result in more speedup and the per-VMA
in mostly less contention. So likely we'd want the page-table walking optimization first.

We can merge them in either order.

This series gives a speedup as well, by removing the double VMA lookup.

Quoted 7 lines

If you want to merge the page table walking and folio batching change first, I can easily rebase this per-VMA lock series on top of that.

Do you have additional feedback on changes you want me to make to get the page table walking and folio batching thing in a merge-ready shape?

I'll go through one patch set next week, I'm trying to figure out which one to prioritize :)

Signature · 3 lines
Cheers,

David