Skip to contentSkip to conversation

mm, uio: huge PFN mappings for /dev/mem and UIO

PATCH In review Eliav Farber 3 participants 8 messages Nov 12 – Nov 16

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v2 cover mm, uio: huge PFN mappings for /dev/mem and UIO

/dev/mem and UIO both hand large, well-aligned physical regions to userspace, and both populate them eagerly at mmap() time with remap_pfn_range(), that is with 4K PTEs however large and well aligned the region is. vfio-pci moved to fault-driven population in commit f9e54c3a2f5b ("vfio/pci: implement huge_fault support") and gets PMD and PUD mappings where alignment allows. This series does the same for /dev/mem and UIO, and factors out the logic vfio-pci open-codes so that a third driver does not have to repeat it.

Two things have to be solved to do this outside vfio-pci:

1. remap_pfn_range() reserves the range's memory type in the PAT memtype tree, via pfnmap_track() -> reserve_pfn_range(). The vmf_insert_pfn*() helpers only look the memory type up, they never reserve. A driver that populates lazily therefore loses the conflict detection that rejects a second mapping of the same physical range with a conflicting cache attribute. Patch 1 adds pfn_range_track(), which sets the VMA up as remap_pfn_range() would and reserves the range for the VMA's lifetime, but leaves population to the fault handler. For ->mmap_prepare drivers, where no VMA exists yet, it adds an MMAP_PFN_TRACK action.

2. The order dispatch itself: the window and alignment checks, and the pfn_modify_allowed() check that vmf_insert_pfn_pmd() and vmf_insert_pfn_pud() skip but the order-0 path performs. Patch 2 adds vmf_pfn_order_ok() and vmf_insert_pfn_order(), split in two so a driver can apply its own policy between the check and the insert.

Patches 3 and 4 convert /dev/mem and UIO to use them.

Dependency on the huge-pfnmap mm fixes
======================================

A VM_PFNMAP mapping that provides both ->fault and ->huge_fault can take faults of different orders on the same entry, because the mm's THP policy is read early in each fault and is not held across it. mm does not survive that race today. Patches 3 and 4 give /dev/mem and UIO exactly that shape, so they add two more reachable paths to it.

The fixes are posted separately, as they stand on their own - vfio-pci, mshv_vtl_main and nvgrace-gpu already have the same shape:

[PATCH v2 0/4] mm: fixes for huge PFN mappings
https://lore.kernel.org/all/[email protected]/

This series wants them underneath it. Without them, a losing fault hits the walk_to_pmd() assertion under CONFIG_DEBUG_VM, reports VM_FAULT_OOM for a healthy mapping, or trips the pfn-mismatch warning in insert_pmd() when a concurrent order-0 fault has installed a page table.

The write flag
==============

Patch 2 also passes the fault's write flag to vmf_insert_pfn_pmd() and vmf_insert_pfn_pud(), as the DAX and DRM ->huge_fault handlers do, rather than the literal false that vfio-pci passes. With write=false, insert_pmd() leaves a present read-only entry in place and returns VM_FAULT_NOPAGE, and wp_huge_pmd() splits the entry only on VM_FAULT_FALLBACK, so a store to a mapping that soft-dirty tracking write-protected faults forever. uffd-wp cannot apply here:
vma_can_userfault() rejects a VM_PFNMAP vma at its mm-backed test. vfio-pci has the same problem today and has no ->pfn_mkwrite either; that needs a separate patch, which I have not written.

The order-0 path takes the flag too, so the helper does not behave differently at different orders. A CPU store does not need it: the order-0 fault installs a read-only entry and the store faults again, where do_wp_page() -> wp_pfn_shared() upgrades it. A caller that faults a page in without retrying the access, fixup_user_fault() from an ->access handler, gets no second fault. I have not been able to observe that combination failing on /dev/mem or on either UIO memtype, so this is consistency rather than a fix for an observed failure.

Testing
=======

Tested on x86_64 at 7.4-rc2, and on arm64 at 7.3.4, which is what the test hardware runs. The four patches are identical in both trees.

On arm64 Cortex-A75: a local selftest over UIO map0, UIO map1 and /dev/mem reports 64 passed, 0 failed and 3 skipped of 67, the skips being the soft-dirty case, which needs CONFIG_MEM_SOFT_DIRTY. tools/testing/selftests/mm/pfnmap.c passes 6/6 on /dev/mem and 6/6 on /dev/uio0, and a COW-specific test passes 6/6. dmesg is free of warnings and call traces. The fault counts are unchanged from v1 and confirm the huge path: 4 faults for four PMD windows where a 4K mapping needs 2048, 2 for a two-window PMD-offset mapping, 1024 for a 4K-offset mapping of 1024 pages, 2048 for a deliberately misaligned VA, and 0 on a second pass.

x86_64 under QEMU, with a 4G ivshmem BAR as the test region and a small lab module exposing it through UIO:

  functional suite, 30 checks                   30/30
  the same plus MEM_SOFT_DIRTY, 31 checks       31/31
  PTE/PMD race suite, 6 checks                  6/6
  COW suite, 6 checks                           6/6

The soft-dirty config turns what the other configs skip into two real checks, a PUD and a PMD write-protected store, which exercise the huge-order write-flag handling described above. The race and COW suites ran on this series with the fixes series above applied under it.

The COW change in patches 3 and 4 was confirmed by building the same tree twice and changing nothing else. With ->fault left installed on the COW vma, eight threads racing a store and a read over 2000 rounds land

kernel BUG at mm/memory.c:2989!

  RIP: 0010:vmf_insert_pfn_prot+0xfb/0x100

and the guest then wedges, because BUG() in a fault path leaves mmap_lock held. With the separate COW vm_operations_struct the same run survives, 6/6. The intermediate shape - keeping ->fault but returning VM_FAULT_SIGBUS for a COW vma - is not enough either, and was observed raising a spurious SIGBUS on a healthy mapping at 200 rounds with four threads,
because installing ->fault at all skips the recheck in do_fault() of a pte found cleared under the page table lock.

Build-only checks, all with KCFLAGS=-Werror W=1:

  m68k ColdFire, CONFIG_MMU=n            vmlinux links
  riscv64, THP and UIO enabled           mm/ and drivers/uio/ build
  tools/testing/vma                      29/29

riscv64 is the only other architecture selecting HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD, and its defconfig leaves THP and UIO off, so both were turned on for that build.

Two things none of this reaches: 1G PUD mappings anywhere but x86, as the test hardware has no suitable 1G-aligned region, and the huge zero folio fallback in patch 2, which needs STRICT_DEVMEM=n and has no check targeting it.

Performance
===========

Unchanged from v1, where the numbers are tabulated; nothing in v2 touches the huge path. On arm64 the fault counts were re-measured and are
identical: 128 faults for a 256MB /dev/mem region against 65536 for a 4K mapping, and 4 against 2048 for an 8MB UIO region. Against today's eager mapping the series cut mmap() by 95x on the 256MB region and 5.6x on the 8MB one, total setup by about 12% on both, and the dTLB read-miss count by about 80x.

Changes since v1
================

- Patch 2: the order-0 path passes the fault's write flag too, so a caller that does not retry the access, fixup_user_fault() from an ->access handler, is not left with a read-only pte. vmf_insert_pfn_prot() keeps its signature; its body moved into a static __vmf_insert_pfn_prot() taking mkwrite.

- Patches 3 and 4: COW mappings get their own vm_operations_struct defining only .access, instead of a COW test inside the fault handler, which was not enough: installing ->fault at all skips do_fault()'s recheck of a pte cleared under the page table lock.

- Commit messages: corrected the L1TF claim in patches 3 and 4 - mmap(PROT_NONE) above l1tf_pfn_limit() now succeeds, and nothing fails later - and dropped patch 2's claim that uffd-wp can write-protect such a mapping.

- Patch 3: the commit message now also documents what the fault-time range_is_allowed() re-check changes without CONFIG_IO_STRICT_DEVMEM, where revoke_iomem() is a no-op.

The write-flag and COW changes came from sashiko-bot's review of v1.

v1: https://lore.kernel.org/all/[email protected]/

Eliav Farber (4):
mm: add MMAP_PFN_TRACK action and pfn_range_track() for lazy PFN maps
mm: add vmf_insert_pfn_order() and vmf_pfn_order_ok() helpers mm/char-mem: implement huge_fault support for /dev/mem
uio: implement huge_fault support for physical mappings

 drivers/uio/uio.c               | 118 ++++++++++++++++--
 include/linux/mm.h              |  47 ++++++++
 include/linux/mm_types.h        |   1 +
 mm/char-mem.c                   | 108 ++++++++++++++++-
 mm/internal.h                   |   3 +
 mm/memory.c                     | 208 ++++++++++++++++++++++++++++----
 mm/util.c                       |   7 ++
 tools/testing/vma/include/dup.h |   1 +
 8 files changed, 458 insertions(+), 35 deletions(-)

base-commit: 067c5c24cef23d913b67eaaf71ae67e7169a6620

Signature · 1 line
2.47.3
#1 lore raw

PATCH v2 1/4 mm: add MMAP_PFN_TRACK action and pfn_range_track() for lazy PFN maps

Drivers that expose a physical PFN range to userspace have traditionally used remap_pfn_range(), which both populates the page tables eagerly and, on x86, reserves the range's memory type in the PAT memtype tree via pfnmap_track() -> reserve_pfn_range() -> memtype_reserve(). That reservation is what lets the kernel detect and reject a second mapping of the same physical range with a conflicting cache attribute (e.g. WB vs UC/WC), which can otherwise cause data corruption or inconsistent cache state.

Drivers that instead populate a PFN mapping lazily from a fault handler (via vmf_insert_pfn()/vmf_insert_pfn_pmd()/vmf_insert_pfn_pud(), e.g. to obtain huge PFN mappings on fault) get no such reservation: those helpers only call pfnmap_setup_cachemode_pfn(), a lookup-only wrapper that never reserves. Such mappings are therefore never registered in the PAT memtype tree and conflicting mappings go undetected.

Add pfn_range_track(): it sets up the VMA exactly as remap_pfn_range() would (VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP flags), reserves and tracks the whole VMA's PFN range for its lifetime by attaching a pfnmap_track_ctx to the VMA (released automatically on VMA teardown), but does not populate any page tables - population is left to the caller's fault handler. COW mappings are rejected, as the fault-driven vmf_insert_pfn*() helpers cannot populate them.

On architectures without PFN tracking (!__HAVE_PFNMAP_TRACKING) the reservation is a no-op, matching remap_pfn_range()'s notrack path; the VMA flag setup still happens.

For drivers that use ->mmap_prepare, where no VMA exists yet, add an MMAP_PFN_TRACK mmap action and a mmap_action_pfn_track() helper. It sets up the VMA like MMAP_REMAP_PFN, and on completion tracks the range like pfn_range_track() instead of populating it. Mirror the new action type in the tools/testing/vma copy of the enum.

---
 include/linux/mm.h              | 19 +++++++
 include/linux/mm_types.h        |  1 +
 mm/internal.h                   |  3 ++
 mm/memory.c                     | 93 +++++++++++++++++++++++++++++++++
 mm/util.c                       |  7 +++
 tools/testing/vma/include/dup.h |  1 +
 6 files changed, 124 insertions(+)
include/linux/mm.h +19 -0
index 6af3b7008f1a..c11bdee1596a 100644
@@ -4752,6 +4752,23 @@ static inline void mmap_action_ioremap_full(struct vm_area_desc *desc,
 	mmap_action_ioremap(desc, desc->start, start_pfn, vma_desc_size(desc));
 }
 
+/**
+ * mmap_action_pfn_track - helper for mmap_prepare hook to specify that the PFN
+ * range backing the entirety of a VMA should be tracked, but not mapped.
+ * @desc: The VMA descriptor for the VMA.
+ * @start_pfn: The first PFN in the range.
+ *
+ * The VMA is set up as for mmap_action_remap_full(), but no page tables are
+ * populated: the mapping is populated on demand from the driver's fault
+ * handler. Must not be used for a COW mapping. See pfn_range_track().
+ */
+static inline void mmap_action_pfn_track(struct vm_area_desc *desc,
+					 unsigned long start_pfn)
+{
+	mmap_action_remap_full(desc, start_pfn);
+	desc->action.type = MMAP_PFN_TRACK;
+}
+
 /**
  * mmap_action_simple_ioremap - helper for mmap_prepare hook to specify that the
  * physical range in [start_phys_addr, start_phys_addr + size) should be I/O
@@ -4973,6 +4990,8 @@ struct vm_area_struct *find_extend_vma_locked(struct mm_struct *,
 		unsigned long addr);
 int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr,
 		    unsigned long pfn, unsigned long size, pgprot_t pgprot);
+int pfn_range_track(struct vm_area_struct *vma, unsigned long addr,
+		    unsigned long pfn, unsigned long size);
 
 int vm_insert_page(struct vm_area_struct *, unsigned long addr, struct page *);
 int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr,
include/linux/mm_types.h +1 -0
index 1ed35d258d2e..b4dfc1b12c2e 100644
@@ -821,6 +821,7 @@ enum mmap_action_type {
 	MMAP_SIMPLE_IO_REMAP,		/* I/O remap with guardrails. */
 	MMAP_KERNEL_PAGES,		/* Map kernel page range from array. */
 	MMAP_DISCONTIG_KERNEL_PAGES,	/* Map kernel discontig page range. */
+	MMAP_PFN_TRACK,			/* Track PFN range, map on fault. */
 };
 
 enum discontig_kernel_page_action {
mm/internal.h +3 -0
index e3f92883bf23..58a81f6041a9 100644
@@ -1546,6 +1546,9 @@ int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm);
 int remap_pfn_range_prepare(struct vm_area_desc *desc);
 int remap_pfn_range_complete(struct vm_area_struct *vma,
 			     struct mmap_action *action);
+int pfn_range_track_prepare(struct vm_area_desc *desc);
+int pfn_range_track_complete(struct vm_area_struct *vma,
+			     struct mmap_action *action);
 int simple_ioremap_prepare(struct vm_area_desc *desc);
 int map_kernel_pages_prepare(struct vm_area_desc *desc);
 int map_kernel_pages_complete(struct vm_area_struct *vma,
mm/memory.c +93 -0

99 lines not on this page · raw message

mm/util.c +7 -0
index 80f6c714844f..5332030ed370 100644
@@ -1546,6 +1546,8 @@ int mmap_action_prepare(struct vm_area_desc *desc)
 		return 0;
 	case MMAP_REMAP_PFN:
 		return remap_pfn_range_prepare(desc);
+	case MMAP_PFN_TRACK:
+		return pfn_range_track_prepare(desc);
 	case MMAP_IO_REMAP_PFN:
 		return io_remap_pfn_range_prepare(desc);
 	case MMAP_SIMPLE_IO_REMAP:
@@ -1583,6 +1585,9 @@ int mmap_action_complete(struct vm_area_struct *vma,
 	case MMAP_REMAP_PFN:
 		err = remap_pfn_range_complete(vma, action);
 		break;
+	case MMAP_PFN_TRACK:
+		err = pfn_range_track_complete(vma, action);
+		break;
 	case MMAP_KERNEL_PAGES:
 		err = map_kernel_pages_complete(vma, action);
 		break;
@@ -1611,6 +1616,7 @@ int mmap_action_prepare(struct vm_area_desc *desc)
 	case MMAP_SIMPLE_IO_REMAP:
 	case MMAP_KERNEL_PAGES:
 	case MMAP_DISCONTIG_KERNEL_PAGES:
+	case MMAP_PFN_TRACK:
 		WARN_ON_ONCE(1); /* nommu cannot handle these. */
 		break;
 	}
@@ -1633,6 +1639,7 @@ int mmap_action_complete(struct vm_area_struct *vma,
 	case MMAP_SIMPLE_IO_REMAP:
 	case MMAP_KERNEL_PAGES:
 	case MMAP_DISCONTIG_KERNEL_PAGES:
+	case MMAP_PFN_TRACK:
 		WARN_ON_ONCE(1); /* nommu cannot handle this. */
 
 		err = -EINVAL;
tools/testing/vma/include/dup.h +1 -0
index dc4927d26296..7bb7392c7fb5 100644
@@ -455,6 +455,7 @@ enum mmap_action_type {
 	MMAP_SIMPLE_IO_REMAP,		/* I/O remap with guardrails. */
 	MMAP_KERNEL_PAGES,		/* Map kernel page range from array. */
 	MMAP_DISCONTIG_KERNEL_PAGES,	/* Map kernel discontig page range. */
+	MMAP_PFN_TRACK,			/* Track PFN range, map on fault. */
 };
 
 /*

base-commit: 067c5c24cef23d913b67eaaf71ae67e7169a6620

Signature · 1 line
2.47.3
#2 lore raw

PATCH v2 2/4 mm: add vmf_insert_pfn_order() and vmf_pfn_order_ok() helpers

In reply to Eliav Farber

->fault/->huge_fault handlers of VM_PFNMAP mappings that want huge PFN mappings open-code the same steps: check that the order-aligned fault window lies within the VMA and that the pfn is aligned to the order, then call vmf_insert_pfn(), vmf_insert_pfn_pmd() or vmf_insert_pfn_pud() depending on the order, guarded by CONFIG_ARCH_SUPPORTS_{PMD,PUD}_PFNMAP, and fall back to a smaller order otherwise. vfio-pci has its own is_aligned_for_order() and vfio_pci_vmf_insert_pfn() for this.

Add vmf_pfn_order_ok() for the window and alignment check, and vmf_insert_pfn_order() for the order dispatch. The latter also checks pfn_modify_allowed() for huge orders: the order-0 vmf_insert_pfn() does this, but vmf_insert_pfn_pmd() and vmf_insert_pfn_pud() do not. The two are separate so that a driver can apply its own policy (access checks, device locking) between them.

vmf_insert_pfn_order() passes the fault's write flag to vmf_insert_pfn_pmd() and vmf_insert_pfn_pud(), as the DAX and DRM ->huge_fault handlers do. With write=false, insert_pmd() leaves a read-only entry in place and returns VM_FAULT_NOPAGE, so a store to a mapping that soft-dirty tracking write-protected faults forever; wp_huge_pmd() splits the entry only when the handler returns VM_FAULT_FALLBACK. Passing it marks the entry young, dirty and, where the VMA allows, writable, as wp_page_reuse() does at PTE level. uffd-wp cannot apply here: vma_can_userfault() rejects a VM_PFNMAP vma at its mm-backed test.

The order-0 path takes the flag too, so the helper does not behave differently at different orders. A CPU store does not need it: the order-0 fault installs a read-only entry and the store faults again, where do_wp_page() -> wp_pfn_shared() upgrades it. A caller that faults a page in without retrying the access, fixup_user_fault() from an ->access handler, gets no second fault, and generic_access_phys() takes its writability from pte_write(); I have not been able to observe that combination failing on /dev/mem or on either UIO memtype, so this is consistency rather than a fix for an observed failure. vmf_insert_pfn() cannot pass the flag, so vmf_insert_pfn_prot() keeps its signature and its body moves to a static helper that takes mkwrite.

The helper also declines the huge zero folio's pfn, which vmf_insert_pfn_pmd() refuses with SIGBUS, so that the fault is retried at PTE order where it maps.

They will be used by the /dev/mem and UIO huge_fault handlers added later in this series.

---
v2:
The order-0 path passes the fault's write flag too, so a caller that does not retry the access, fixup_user_fault() from an ->access handler, is not left with a read-only pte. vmf_insert_pfn_prot() keeps its signature; its body moved into a static __vmf_insert_pfn_prot() taking mkwrite. Dropped the commit message's claim that uffd-wp can write-protect such a mapping; it cannot. Reported by sashiko-bot.

 include/linux/mm.h |  28 +++++++++++
 mm/memory.c        | 115 +++++++++++++++++++++++++++++++++++----------
 2 files changed, 119 insertions(+), 24 deletions(-)
include/linux/mm.h +28 -0
index c11bdee1596a..57437b317709 100644
@@ -5010,6 +5010,34 @@ vm_fault_t vmf_insert_mixed(struct vm_area_struct *vma, unsigned long addr,
 			unsigned long pfn);
 vm_fault_t vmf_insert_mixed_mkwrite(struct vm_area_struct *vma,
 		unsigned long addr, unsigned long pfn);
+vm_fault_t vmf_insert_pfn_order(struct vm_fault *vmf, unsigned long pfn,
+				unsigned int order);
+
+/**
+ * vmf_pfn_order_ok - check that a pfn can be mapped at a given fault order
+ * @vmf: The fault being handled.
+ * @pfn: The pfn to map at the start of the order-aligned fault window.
+ * @order: The fault order (0 for a single page).
+ *
+ * For a huge (order > 0) fault, the order-aligned window around @vmf->address
+ * must lie within the VMA and @pfn must be aligned to @order. @pfn is only
+ * looked at once the window is known to lie within the VMA, so a caller may
+ * compute it before calling, even when the window starts below vm_start.
+ *
+ * Return: true if vmf_insert_pfn_order() may map @pfn at @order, false if the
+ * fault should fall back to a smaller order.
+ */
+static inline bool vmf_pfn_order_ok(const struct vm_fault *vmf,
+				    unsigned long pfn, unsigned int order)
+{
+	const struct vm_area_struct *vma = vmf->vma;
+	unsigned long addr = vmf->address & ~((PAGE_SIZE << order) - 1);
+
+	return !order || (addr >= vma->vm_start &&
+			  addr + (PAGE_SIZE << order) <= vma->vm_end &&
+			  IS_ALIGNED(pfn, 1UL << order));
+}
+
 int vm_iomap_memory(struct vm_area_struct *vma, phys_addr_t start, unsigned long len);
 
 static inline vm_fault_t vmf_insert_page(struct vm_area_struct *vma,
mm/memory.c +91 -24

133 lines not on this page · raw message

Signature · 1 line
2.47.3
#3 lore raw

PATCH v2 3/4 mm/char-mem: implement huge_fault support for /dev/mem

In reply to Eliav Farber

/dev/mem is commonly used to map large physical regions (reserved memory, device memory) into userspace. mmap_mem_prepare() maps the whole range at mmap() time with 4K PTEs, however large and well aligned it is, which costs many page-table entries and TLB misses.

Map non-COW mappings on fault instead, as vfio-pci does since commit f9e54c3a2f5b ("vfio/pci: implement huge_fault support"): request MMAP_PFN_TRACK rather than an eager remap, so the VMA is set up and its memory type tracked as before, and add ->fault/->huge_fault handlers that map the faulting window with vmf_insert_pfn_order(), using a PMD or PUD where alignment allows.

COW mappings cannot be populated by vmf_insert_pfn*(), which BUG_ON()s a COW VM_PFNMAP vma, so they stay eager and keep a vm_ops without fault handlers. do_fault() then rechecks a pte it found cleared under the page table lock, and refuses it with SIGBUS only once it is really gone, as after revoke_iomem(); a PMD-aligned COW mapping is also not reported as THP eligible in smaps.

The handler re-checks range_is_allowed(): with CONFIG_IO_STRICT_DEVMEM, revoke_iomem() zaps a range once a driver claims it, and a fault must not map it back in. With CONFIG_STRICT_DEVMEM this costs one devmem_is_allowed() call per page of the faulting window. Unlike the eager mapping, a revoke is not permanent: once the driver releases the range, a fault repopulates the mapping, as a fresh mmap() of it would.

Without CONFIG_IO_STRICT_DEVMEM the re-check still changes behaviour. A driver can take an IORESOURCE_EXCLUSIVE region at any time, which makes devmem_is_allowed() refuse those pages, and revoke_iomem() is a no-op in that configuration. The eager mapping therefore kept serving them, while a fault now returns SIGBUS for any page not already touched.

generic_access_phys() only follows present PTEs, so .access now faults the page in first; otherwise remote access (ptrace, /proc/pid/mem) to an untouched page would fail. A mapping that does not permit the fault, PROT_NONE for example, keeps that gap: fixup_user_fault() refuses it, where the eager mapping left a present PTE.

Validation at mmap() time is unchanged, except for the L1TF pfn check. remap_pfn_range() rejected a disallowed pfn at mmap() time, which error_override turned into -EAGAIN, so an unprivileged mmap(PROT_NONE) of a pfn above l1tf_pfn_limit() failed. It now succeeds. Nothing fails later instead: pfn_modify_allowed() only denies a prot that needs PTE inversion, i.e. a non-present one, and no pte is installed for an inaccessible vma.

vmf_insert_pfn() refuses the zero pfn, where remap_pfn_range() mapped it, so a page of RAM that happens to be the zero page now SIGBUSes. That needs /dev/mem access to RAM, i.e. a kernel without CONFIG_STRICT_DEVMEM.

Without huge PFN mappings, or for small or unaligned mappings, the range is populated one page per fault, paid only for pages that are actually touched.

---
v2:
COW mappings get their own vm_operations_struct defining only .access, instead of a COW test inside the fault handler, which was not enough:
installing ->fault at all skips do_fault()'s recheck of a pte cleared under the page table lock. Corrected the L1TF claim in the commit message - mmap(PROT_NONE) above l1tf_pfn_limit() now succeeds, and nothing fails later - and documented what the fault-time range_is_allowed() re-check changes without CONFIG_IO_STRICT_DEVMEM. Reported by sashiko-bot.

 mm/char-mem.c | 108 ++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 104 insertions(+), 4 deletions(-)
mm/char-mem.c +104 -4
index c5761ddedbd7..99f02d6a8b07 100644
@@ -318,9 +318,95 @@ static inline int private_mapping_ok(struct vm_area_desc *desc)
 }
 #endif
 
+#ifdef CONFIG_MMU
+/*
+ * Populate /dev/mem mappings on fault, using a PMD or PUD mapping where the
+ * architecture, the fault address and the pfn allow it.
+ */
+static vm_fault_t mmap_mem_huge_fault(struct vm_fault *vmf, unsigned int order)
+{
+	unsigned long addr = vmf->address & ~((PAGE_SIZE << order) - 1);
+	struct vm_area_struct *vma = vmf->vma;
+	unsigned long pfn;
+
+	/*
+	 * vm_pgoff is the base pfn of the mapping. vmf_pfn_order_ok() ignores
+	 * pfn if the window starts below vm_start.
+	 */
+	pfn = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT);
+	if (!vmf_pfn_order_ok(vmf, pfn, order))
+		return VM_FAULT_FALLBACK;
+
+	/*
+	 * Access was checked at mmap() time, but with CONFIG_IO_STRICT_DEVMEM
+	 * revoke_iomem() zaps the mapping once a driver claims part of the
+	 * range. Re-check here so a revoked page is not faulted back in. Like
+	 * the mmap()-time check, it is not serialised against revoke_iomem(),
+	 * so a fault that passed it can still install a page revoked right
+	 * after.
+	 */
+	if (!range_is_allowed(pfn, PAGE_SIZE << order))
+		return order ? VM_FAULT_FALLBACK : VM_FAULT_SIGBUS;
+
+	return vmf_insert_pfn_order(vmf, pfn, order);
+}
+
+static vm_fault_t mmap_mem_fault(struct vm_fault *vmf)
+{
+	return mmap_mem_huge_fault(vmf, 0);
+}
+#endif /* CONFIG_MMU */
+
+#ifdef CONFIG_HAVE_IOREMAP_PROT
+/*
+ * generic_access_phys() only follows present PTEs, and a lazily populated
+ * mapping has none for pages the target has not touched yet. Fault the page in
+ * first, so that remote access (ptrace, /proc/pid/mem) keeps working.
+ *
+ * fixup_user_fault() refuses a fault the vma does not permit, so an untouched
+ * page of a mapping without VM_READ stays unreadable, where the eager mapping
+ * left a present PTE.
+ */
+static int mmap_mem_access(struct vm_area_struct *vma, unsigned long addr,
+			   void *buf, int len, int write)
+{
+	unsigned int fault_flags = FAULT_FLAG_REMOTE;
+
+	if (write)
+		fault_flags |= FAULT_FLAG_WRITE;
+
+	/*
+	 * With unlocked == NULL, fixup_user_fault() keeps the mmap lock held
+	 * and the vma valid.
+	 */
+	fixup_user_fault(vma->vm_mm, addr, fault_flags, NULL);
+
+	return generic_access_phys(vma, addr, buf, len, write);
+}
+#endif
+
+/*
+ * COW mappings are mapped eagerly by remap_pfn_range() and cannot be populated
+ * on fault: vmf_insert_pfn*() BUG_ON() a COW VM_PFNMAP vma. Leave them without
+ * fault handlers, so that do_fault() rechecks a pte it found cleared under the
+ * page table lock, and fails only once the pte is really gone, as after
+ * revoke_iomem(), and so that they are not reported as THP eligible in smaps.
+ */
+static const struct vm_operations_struct mmap_mem_cow_vm_ops = {
+#ifdef CONFIG_HAVE_IOREMAP_PROT
+	.access = generic_access_phys,
+#endif
+};
+
 static const struct vm_operations_struct mmap_mem_ops = {
+#ifdef CONFIG_MMU
+	.fault = mmap_mem_fault,
+#endif
+#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
+	.huge_fault = mmap_mem_huge_fault,
+#endif
 #ifdef CONFIG_HAVE_IOREMAP_PROT
-	.access = generic_access_phys
+	.access = mmap_mem_access,
 #endif
 };
 
@@ -355,11 +441,25 @@ static int mmap_mem_prepare(struct vm_area_desc *desc)
 					       size,
 					       desc->page_prot);
 
+	desc->action.error_override = -EAGAIN;
+
+	if (vma_desc_is_cow_mapping(desc)) {
+		/*
+		 * vmf_insert_pfn*() cannot populate COW mappings: map them now.
+		 * Remap-pfn-range will mark the range with the I/O flag.
+		 */
+		desc->vm_ops = &mmap_mem_cow_vm_ops;
+		mmap_action_remap_full(desc, desc->pgoff);
+		return 0;
+	}
+
 	desc->vm_ops = &mmap_mem_ops;
 
-	/* Remap-pfn-range will mark the range with the I/O flag. */
-	mmap_action_remap_full(desc, desc->pgoff);
-	desc->action.error_override = -EAGAIN;
+	/*
+	 * Populate on fault (see mmap_mem_huge_fault()). Tracking the PFN range
+	 * keeps its memory type reserved, for PAT conflict detection.
+	 */
+	mmap_action_pfn_track(desc, desc->pgoff);
 
 	return 0;
 }
Signature · 1 line
2.47.3
#4 lore raw

Re: [PATCH v2 3/4] mm/char-mem: implement huge_fault support for /dev/mem

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] mm/char-mem: unchecked runtime PUD support in mmap_mem_huge_fault() --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] mm/char-mem: unchecked runtime PUD support in mmap_mem_huge_fault()
--

commit 231caef4edac4da87c99aec56a1e3e64b13e76e2
Author: Eliav Farber <[email protected]>

mm/char-mem: implement huge_fault support for /dev/mem

This commit populates non-COW /dev/mem mappings on fault via ->fault and ->huge_fault handlers using vmf_insert_pfn_order() rather than mapping the entire range eagerly with 4K PTEs at mmap() time.

Quoted 30 lines
mm/char-mem.c +22 -0
index c5761ddedbd7d..99f02d6a8b07d 100644
@@ -318,9 +318,95 @@ static inline int private_mapping_ok(struct vm_area_desc *desc)
 }
 #endif
 
+#ifdef CONFIG_MMU
+/*
+ * Populate /dev/mem mappings on fault, using a PMD or PUD mapping where the
+ * architecture, the fault address and the pfn allow it.
+ */
+static vm_fault_t mmap_mem_huge_fault(struct vm_fault *vmf, unsigned int order)
+{
+	unsigned long addr = vmf->address & ~((PAGE_SIZE << order) - 1);
+	struct vm_area_struct *vma = vmf->vma;
+	unsigned long pfn;
+
+	/*
+	 * vm_pgoff is the base pfn of the mapping. vmf_pfn_order_ok() ignores
+	 * pfn if the window starts below vm_start.
+	 */
+	pfn = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT);
+	if (!vmf_pfn_order_ok(vmf, pfn, order))
+		return VM_FAULT_FALLBACK;
+
+	/*
+	 * Access was checked at mmap() time, but with CONFIG_IO_STRICT_DEVMEM
+	 * revoke_iomem() zaps the mapping once a driver claims part of the

mm/char-mem.c static inline int private_mapping_ok(struct vm_area_desc *desc) · line 352

Quoted 10 lines
mm/char-mem.c +10 -0
+	 * range. Re-check here so a revoked page is not faulted back in. Like
+	 * the mmap()-time check, it is not serialised against revoke_iomem(),
+	 * so a fault that passed it can still install a page revoked right
+	 * after.
+	 */
+	if (!range_is_allowed(pfn, PAGE_SIZE << order))
+		return order ? VM_FAULT_FALLBACK : VM_FAULT_SIGBUS;
+
+	return vmf_insert_pfn_order(vmf, pfn, order);
+}

[Severity: Medium]
Can mmap_mem_huge_fault() attempt to install a PUD mapping when the hardware or MMU mode does not support PUD hugepages at runtime?

For VM_PFNMAP mappings, __thp_vma_allowable_orders() allows PUD_ORDER whenever PMD THP is enabled, and vmf_insert_pfn_order() only checks the compile-time CONFIG_ARCH_SUPPORTS_PUD_PFNMAP before calling vmf_insert_pfn_pud():

mm/memory.c:vmf_insert_pfn_order() {
    ...
	if (IS_ENABLED(CONFIG_ARCH_SUPPORTS_PUD_PFNMAP) && order == PUD_ORDER)
		return vmf_insert_pfn_pud(vmf, pfn, write);
    ...
}

Previously /dev/mem only created 4K PTE mappings via remap_pfn_range(), making this a regression when faulting on a PUD-aligned /dev/mem range. On PowerPC Book3S 64 running with the Hash MMU (!radix_enabled()), vmf_insert_pfn_pud() -> insert_pud() -> pfn_pud() -> __pud_mkhuge() hits an unconditional BUG() under the PUD lock. On x86_64 systems without X86_FEATURE_GBPAGES, pud_mkhuge() sets _PAGE_PSE in the PUD entry, causing a reserved-bit page fault and a kernel oops in pgtable_bad() on the retried access.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#5 lore raw

PATCH v2 4/4 uio: implement huge_fault support for physical mappings

In reply to Eliav Farber

UIO physical mappings (UIO_MEM_PHYS, UIO_MEM_IOVA) often expose large device memory regions to userspace. uio_mmap_physical() maps them at mmap() time with remap_pfn_range(), i.e. with 4K PTEs, even when the region and the user address are aligned for huge mappings.

Map non-COW mappings on fault instead, as vfio-pci does since commit f9e54c3a2f5b ("vfio/pci: implement huge_fault support"): call pfn_range_track() rather than remap_pfn_range(), so the VMA is set up and its memory type tracked as before, and add ->fault/->huge_fault handlers that map the faulting window with vmf_insert_pfn_order(), using a PMD or PUD where alignment allows. Like uio_vma_fault(), the handlers take idev->info_lock and fail with SIGBUS once the device is gone.

COW mappings cannot be populated by vmf_insert_pfn*(), which BUG_ON()s a COW VM_PFNMAP vma, so they stay eager and keep a vm_ops without fault handlers. do_fault() then rechecks a pte it found cleared under the page table lock, which is what a transient clearing by wp_page_copy() needs; a PMD-aligned COW mapping is also not reported as THP eligible in smaps.

vm_pgoff holds the UIO map index, which a fault handler cannot use: a VMA split (partial munmap(), mprotect(), ...) adds the split offset to vm_pgoff of the upper part, so the index no longer identifies the map. Store the base pfn in vm_pgoff instead, as remap_pfn_range() does for COW mappings; splits then keep it correct. As a side effect, /proc/<pid>/maps shows the physical address as the offset of such a mapping, rather than N * PAGE_SIZE.

generic_access_phys() only follows present PTEs, so .access now faults the page in first; otherwise remote access (ptrace, /proc/pid/mem) to an untouched page would fail. A mapping that does not permit the fault, PROT_NONE for example, keeps that gap: fixup_user_fault() refuses it, where the eager mapping left a present PTE.

Validation at mmap() time is unchanged, except for the L1TF pfn check. remap_pfn_range() rejected a disallowed pfn with -EACCES from mmap(), so an unprivileged mmap(PROT_NONE) of a pfn above l1tf_pfn_limit() failed. It now succeeds. Nothing fails later instead:
pfn_modify_allowed() only denies a prot that needs PTE inversion, i.e. a non-present one, and no pte is installed for an inaccessible vma.

Once the device is unregistered, a page with no present entry gets SIGBUS, where the eager mapping kept it mapped. Not only untouched
pages: splitting a huge entry with a partial munmap() or mprotect() zaps it, since __split_huge_pmd_locked() clears a PFN-mapped PMD rather than converting it to PTEs.

Without huge PFN mappings, or for small or unaligned mappings, the region is populated one page per fault, paid only for pages that are actually touched.

---
v2:
COW mappings get their own vm_operations_struct defining only .access; with ->fault installed, a COW vma could reach the BUG_ON in vmf_insert_pfn*(), reproduced as a kernel BUG at mm/memory.c:2989. Corrected the L1TF claim in the commit message: mmap(PROT_NONE) above l1tf_pfn_limit() now succeeds, and nothing fails later. Reported by sashiko-bot.

 drivers/uio/uio.c | 118 +++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 111 insertions(+), 7 deletions(-)
drivers/uio/uio.c +111 -7
index f8fa20522660..7c038e3b107c 100644
@@ -733,12 +733,94 @@ static int uio_mmap_logical(struct vm_area_struct *vma)
 	return 0;
 }
 
-static const struct vm_operations_struct uio_physical_vm_ops = {
+/*
+ * Populate UIO physical mappings on fault, using a PMD or PUD mapping where the
+ * architecture, the fault address and the pfn allow it.
+ */
+static vm_fault_t uio_physical_huge_fault(struct vm_fault *vmf,
+					  unsigned int order)
+{
+	struct vm_area_struct *vma = vmf->vma;
+	unsigned long addr = vmf->address & ~((PAGE_SIZE << order) - 1);
+	struct uio_device *idev = vma->vm_private_data;
+	vm_fault_t ret = VM_FAULT_SIGBUS;
+	unsigned long pfn;
+
+	/*
+	 * vm_pgoff is the base pfn of the mapping (see uio_mmap_physical()).
+	 * vmf_pfn_order_ok() ignores pfn if the window starts below vm_start.
+	 */
+	pfn = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT);
+	if (!vmf_pfn_order_ok(vmf, pfn, order))
+		return VM_FAULT_FALLBACK;
+
+	mutex_lock(&idev->info_lock);
+
+	if (idev->info)
+		ret = vmf_insert_pfn_order(vmf, pfn, order);
+
+	mutex_unlock(&idev->info_lock);
+
+	return ret;
+}
+
+static vm_fault_t uio_physical_fault(struct vm_fault *vmf)
+{
+	return uio_physical_huge_fault(vmf, 0);
+}
+
+#ifdef CONFIG_HAVE_IOREMAP_PROT
+/*
+ * generic_access_phys() only follows present PTEs, and a lazily populated
+ * mapping has none for pages the target has not touched yet. Fault the page in
+ * first (through the handlers above, so info_lock is honoured), so that remote
+ * access (ptrace, /proc/pid/mem) keeps working.
+ *
+ * fixup_user_fault() refuses a fault the vma does not permit, so an untouched
+ * page of a mapping without VM_READ stays unreadable, where the eager mapping
+ * left a present PTE.
+ */
+static int uio_physical_access(struct vm_area_struct *vma, unsigned long addr,
+			       void *buf, int len, int write)
+{
+	unsigned int fault_flags = FAULT_FLAG_REMOTE;
+
+	if (write)
+		fault_flags |= FAULT_FLAG_WRITE;
+
+	/*
+	 * With unlocked == NULL, fixup_user_fault() keeps the mmap lock held
+	 * and the vma valid.
+	 */
+	fixup_user_fault(vma->vm_mm, addr, fault_flags, NULL);
+
+	return generic_access_phys(vma, addr, buf, len, write);
+}
+#endif
+
+/*
+ * COW mappings are mapped eagerly by remap_pfn_range() and cannot be populated
+ * on fault: vmf_insert_pfn*() BUG_ON() a COW VM_PFNMAP vma. Leave them without
+ * fault handlers, so that do_fault() rechecks a pte it found cleared under the
+ * page table lock, and fails only once the pte is really gone, and so that they
+ * are not reported as THP eligible in smaps.
+ */
+static const struct vm_operations_struct uio_physical_cow_vm_ops = {
 #ifdef CONFIG_HAVE_IOREMAP_PROT
 	.access = generic_access_phys,
 #endif
 };
 
+static const struct vm_operations_struct uio_physical_vm_ops = {
+#ifdef CONFIG_HAVE_IOREMAP_PROT
+	.access = uio_physical_access,
+#endif
+	.fault = uio_physical_fault,
+#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
+	.huge_fault = uio_physical_huge_fault,
+#endif
+};
+
 static int uio_mmap_physical(struct vm_area_struct *vma)
 {
 	struct uio_device *idev = vma->vm_private_data;
@@ -754,7 +836,6 @@ static int uio_mmap_physical(struct vm_area_struct *vma)
 	if (vma->vm_end - vma->vm_start > mem->size)
 		return -EINVAL;
 
-	vma->vm_ops = &uio_physical_vm_ops;
 	if (idev->info->mem[mi].memtype == UIO_MEM_PHYS)
 		vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
 
@@ -767,11 +848,34 @@ static int uio_mmap_physical(struct vm_area_struct *vma)
 	 * So we just do the physical mmap without a page
 	 * offset.
 	 */
-	return remap_pfn_range(vma,
-			       vma->vm_start,
-			       mem->addr >> PAGE_SHIFT,
-			       vma->vm_end - vma->vm_start,
-			       vma->vm_page_prot);
+	if (vma_is_cow_mapping(vma)) {
+		/* COW mappings cannot be populated on fault: map them now. */
+		vma->vm_ops = &uio_physical_cow_vm_ops;
+
+		return remap_pfn_range(vma,
+				       vma->vm_start,
+				       mem->addr >> PAGE_SHIFT,
+				       vma->vm_end - vma->vm_start,
+				       vma->vm_page_prot);
+	}
+
+	vma->vm_ops = &uio_physical_vm_ops;
+
+	/*
+	 * Populate on fault (see uio_physical_huge_fault()). Tracking the PFN
+	 * range keeps its memory type reserved, for PAT conflict detection.
+	 *
+	 * vm_pgoff holds the map index, which the fault handler cannot use:
+	 * splitting the VMA adds the page offset of the split to vm_pgoff of
+	 * the upper part, so uio_find_mem_index() then returns the wrong map,
+	 * or none, and the start of that part would be mapped to mem->addr.
+	 * Store the base pfn instead, as remap_pfn_range() does for COW
+	 * mappings; VMA splits then keep vm_pgoff correct for each part.
+	 */
+	vma->vm_pgoff = mem->addr >> PAGE_SHIFT;
+
+	return pfn_range_track(vma, vma->vm_start, mem->addr >> PAGE_SHIFT,
+			       vma->vm_end - vma->vm_start);
 }
 
 static int uio_mmap_dma_coherent(struct vm_area_struct *vma)
Signature · 1 line
2.47.3
#6 lore raw
In reply to Eliav Farber

Err, there unanswered questions on v1 of this review, so why have you sent a respin?...

I also seem to remember David asking you to slow down on the respins earlier too.

AI might make it easy to send stuff out but you're outsourcing work to maintainers here...

Quoted 10 lines · Eliav Farber

/dev/mem and UIO both hand large, well-aligned physical regions to userspace, and both populate them eagerly at mmap() time with remap_pfn_range(), that is with 4K PTEs however large and well aligned the region is. vfio-pci moved to fault-driven population in commit f9e54c3a2f5b ("vfio/pci: implement huge_fault support") and gets PMD and PUD mappings where alignment allows. This series does the same for /dev/mem and UIO, and factors out the logic vfio-pci open-codes so that a third driver does not have to repeat it.

Two things have to be solved to do this outside vfio-pci:

[blah
blah
blah
blah
...]

Just way way way too many schloppy words. Write this yourself please.

Quoted 6 lines

Testing
=======

Tested on x86_64 at 7.4-rc2, and on arm64 at 7.3.4, which is what the test hardware runs. The four patches are identical in both trees.

[blah
blah
blah
blah
....]

Quoted 4 lines

Two things none of this reaches: 1G PUD mappings anywhere but x86, as the test hardware has no suitable 1G-aligned region, and the huge zero folio fallback in patch 2, which needs STRICT_DEVMEM=n and has no check targeting it.

This is so TOO MUCH noisy, schloppy information. Write this stuff yourself and small.

I hate to say it but it feels so so lazy to get an AI to do stuff like this without even bothering to reference _how things are done in mm_.

Be considerate please, we don't have time to read your novels.

And you've been told before...

Quoted 11 lines

Performance
===========

Unchanged from v1, where the numbers are tabulated; nothing in v2 touches the huge path. On arm64 the fault counts were re-measured and are
identical: 128 faults for a 256MB /dev/mem region against 65536 for a 4K mapping, and 4 against 2048 for an 8MB UIO region. Against today's eager mapping the series cut mmap() by 95x on the 256MB region and 5.6x on the 8MB one, total setup by about 12% on both, and the dTLB read-miss count by about 80x.

We don't need a (schloppy) 'perf since last-version' update thanks.

Quoted 15 lines

Changes since v1
================

- Patch 2: the order-0 path passes the fault's write flag too, so a caller that does not retry the access, fixup_user_fault() from an ->access handler, is not left with a read-only pte. vmf_insert_pfn_prot() keeps its signature; its body moved into a static __vmf_insert_pfn_prot() taking mkwrite.

- Patches 3 and 4: COW mappings get their own vm_operations_struct defining only .access, instead of a COW test inside the fault handler, which was not enough: installing ->fault at all skips do_fault()'s recheck of a pte cleared under the page table lock.

Quoted 8 lines

- Commit messages: corrected the L1TF claim in patches 3 and 4 - mmap(PROT_NONE) above l1tf_pfn_limit() now succeeds, and nothing fails later - and dropped patch 2's claim that uffd-wp can write-protect such a mapping.

- Patch 3: the commit message now also documents what the fault-time range_is_allowed() re-check changes without CONFIG_IO_STRICT_DEVMEM, where revoke_iomem() is a no-op.

This is so schloppy and WAY TOO MANY WORDS. Please just write these things yourself...

Quoted 2 lines

The write-flag and COW changes came from sashiko-bot's review of v1.

You credit sashiko but not David... Please follow how _other established mm people_ do these things please.

See [0] for an example. Note the brevity and actually crediting people who asked, so they know that their feedback was addressed.

Quoted 2 lines

v1: https://lore.kernel.org/all/[email protected]/

See https://lore.kernel.org/all/[email protected]/

You were told to rewrite your answer, instead you sent a v2. This isn't how to interact on-list.

--
Cheers, Lorenzo

[0]: https://lore.kernel.org/linux-mm/[email protected]/

#7 lore raw
Quoted 8 lines · Lorenzo Stoakes (ARM)

Err, there unanswered questions on v1 of this review, so why have you sent a respin?...

I also seem to remember David asking you to slow down on the respins earlier too.

AI might make it easy to send stuff out but you're outsourcing work to maintainers here...

OK on this one I see that this got sent out before David's mail, disregard this bit, but please regard the rest :)

--
Cheers, Lorenzo