Skip to content

mm, uio: huge PFN mappings for /dev/mem and UIO

Eliav Farber linux-mm lkml 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest4 patches 2 repliesIn review
  2. v14 patches 8 repliesSuperseded

Changes in v2 from the cover letter

================

- Patch 2: the order-0 path passes the fault's write flag too, so a caller that does not retry the access, fixup_user_fault() from an ->access handler, is not left with a read-only pte. vmf_insert_pfn_prot() keeps its signature; its body moved into a static __vmf_insert_pfn_prot() taking mkwrite.

- Patches 3 and 4: COW mappings get their own vm_operations_struct defining only .access, instead of a COW test inside the fault handler, which was not enough: installing ->fault at all skips do_fault()'s recheck of a pte cleared under the page table lock.

- Commit messages: corrected the L1TF claim in patches 3 and 4 - mmap(PROT_NONE) above l1tf_pfn_limit() now succeeds, and nothing fails later - and dropped patch 2's claim that uffd-wp can write-protect such a mapping.

- Patch 3: the commit message now also documents what the fault-time range_is_allowed() re-check changes without CONFIG_IO_STRICT_DEVMEM, where revoke_iomem() is a no-op.

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4mm: add MMAP_PFN_TRACK action and pfn_range_track() for lazy PFN mapsinclude/linux/mm.h include/linux/mm_types.h mm/internal.h +3 more+124 -00 replies
2/4mm: add vmf_insert_pfn_order() and vmf_pfn_order_ok() helpersinclude/linux/mm.h mm/memory.c+119 -240 replies
3/4mm/char-mem: implement huge_fault support for /dev/memmm/char-mem.c+104 -40 replies
4/4uio: implement huge_fault support for physical mappingsdrivers/uio/uio.c+111 -70 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 0/4] mm, uio: huge PFN mappings for /dev/mem and UIO

/dev/mem and UIO both hand large, well-aligned physical regions to userspace, and both populate them eagerly at mmap() time with remap_pfn_range(), that is with 4K PTEs however large and well aligned the region is. vfio-pci moved to fault-driven population in commit f9e54c3a2f5b ("vfio/pci: implement huge_fault support") and gets PMD and PUD mappings where alignment allows. This series does the same for /dev/mem and UIO, and factors out the logic vfio-pci open-codes so that a third driver does not have to repeat it.

Two things have to be solved to do this outside vfio-pci:

1. remap_pfn_range() reserves the range's memory type in the PAT memtype tree, via pfnmap_track() -> reserve_pfn_range(). The vmf_insert_pfn*() helpers only look the memory type up, they never reserve. A driver that populates lazily therefore loses the conflict detection that rejects a second mapping of the same physical range with a conflicting cache attribute. Patch 1 adds pfn_range_track(), which sets the VMA up as remap_pfn_range() would and reserves the range for the VMA's lifetime, but leaves population to the fault handler. For ->mmap_prepare drivers, where no VMA exists yet, it adds an MMAP_PFN_TRACK action.

2. The order dispatch itself: the window and alignment checks, and the pfn_modify_allowed() check that vmf_insert_pfn_pmd() and vmf_insert_pfn_pud() skip but the order-0 path performs. Patch 2 adds vmf_pfn_order_ok() and vmf_insert_pfn_order(), split in two so a driver can apply its own policy between the check and the insert.

Patches 3 and 4 convert /dev/mem and UIO to use them.

Dependency on the huge-pfnmap mm fixes
======================================

A VM_PFNMAP mapping that provides both ->fault and ->huge_fault can take faults of different orders on the same entry, because the mm's THP policy is read early in each fault and is not held across it. mm does not survive that race today. Patches 3 and 4 give /dev/mem and UIO exactly that shape, so they add two more reachable paths to it.

The fixes are posted separately, as they stand on their own - vfio-pci, mshv_vtl_main and nvgrace-gpu already have the same shape:

[PATCH v2 0/4] mm: fixes for huge PFN mappings
https://lore.kernel.org/all/[email protected]/

This series wants them underneath it. Without them, a losing fault hits the walk_to_pmd() assertion under CONFIG_DEBUG_VM, reports VM_FAULT_OOM for a healthy mapping, or trips the pfn-mismatch warning in insert_pmd() when a concurrent order-0 fault has installed a page table.

The write flag
==============

Patch 2 also passes the fault's write flag to vmf_insert_pfn_pmd() and vmf_insert_pfn_pud(), as the DAX and DRM ->huge_fault handlers do, rather than the literal false that vfio-pci passes. With write=false, insert_pmd() leaves a present read-only entry in place and returns VM_FAULT_NOPAGE, and wp_huge_pmd() splits the entry only on VM_FAULT_FALLBACK, so a store to a mapping that soft-dirty tracking write-protected faults forever. uffd-wp cannot apply here:
vma_can_userfault() rejects a VM_PFNMAP vma at its mm-backed test. vfio-pci has the same problem today and has no ->pfn_mkwrite either; that needs a separate patch, which I have not written.

The order-0 path takes the flag too, so the helper does not behave differently at different orders. A CPU store does not need it: the order-0 fault installs a read-only entry and the store faults again, where do_wp_page() -> wp_pfn_shared() upgrades it. A caller that faults a page in without retrying the access, fixup_user_fault() from an ->access handler, gets no second fault. I have not been able to observe that combination failing on /dev/mem or on either UIO memtype, so this is consistency rather than a fix for an observed failure.

Testing
=======

Tested on x86_64 at 7.4-rc2, and on arm64 at 7.3.4, which is what the test hardware runs. The four patches are identical in both trees.

On arm64 Cortex-A75: a local selftest over UIO map0, UIO map1 and /dev/mem reports 64 passed, 0 failed and 3 skipped of 67, the skips being the soft-dirty case, which needs CONFIG_MEM_SOFT_DIRTY. tools/testing/selftests/mm/pfnmap.c passes 6/6 on /dev/mem and 6/6 on /dev/uio0, and a COW-specific test passes 6/6. dmesg is free of warnings and call traces. The fault counts are unchanged from v1 and confirm the huge path: 4 faults for four PMD windows where a 4K mapping needs 2048, 2 for a two-window PMD-offset mapping, 1024 for a 4K-offset mapping of 1024 pages, 2048 for a deliberately misaligned VA, and 0 on a second pass.

x86_64 under QEMU, with a 4G ivshmem BAR as the test region and a small lab module exposing it through UIO:

  functional suite, 30 checks                   30/30
  the same plus MEM_SOFT_DIRTY, 31 checks       31/31
  PTE/PMD race suite, 6 checks                  6/6
  COW suite, 6 checks                           6/6

The soft-dirty config turns what the other configs skip into two real checks, a PUD and a PMD write-protected store, which exercise the huge-order write-flag handling described above. The race and COW suites ran on this series with the fixes series above applied under it.

The COW change in patches 3 and 4 was confirmed by building the same tree twice and changing nothing else. With ->fault left installed on the COW vma, eight threads racing a store and a read over 2000 rounds land

kernel BUG at mm/memory.c:2989!

  RIP: 0010:vmf_insert_pfn_prot+0xfb/0x100

and the guest then wedges, because BUG() in a fault path leaves mmap_lock held. With the separate COW vm_operations_struct the same run survives, 6/6. The intermediate shape - keeping ->fault but returning VM_FAULT_SIGBUS for a COW vma - is not enough either, and was observed raising a spurious SIGBUS on a healthy mapping at 200 rounds with four threads,
because installing ->fault at all skips the recheck in do_fault() of a pte found cleared under the page table lock.

Build-only checks, all with KCFLAGS=-Werror W=1:

  m68k ColdFire, CONFIG_MMU=n            vmlinux links
  riscv64, THP and UIO enabled           mm/ and drivers/uio/ build
  tools/testing/vma                      29/29

riscv64 is the only other architecture selecting HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD, and its defconfig leaves THP and UIO off, so both were turned on for that build.

Two things none of this reaches: 1G PUD mappings anywhere but x86, as the test hardware has no suitable 1G-aligned region, and the huge zero folio fallback in patch 2, which needs STRICT_DEVMEM=n and has no check targeting it.

Performance
===========

Unchanged from v1, where the numbers are tabulated; nothing in v2 touches the huge path. On arm64 the fault counts were re-measured and are
identical: 128 faults for a 256MB /dev/mem region against 65536 for a 4K mapping, and 4 against 2048 for an 8MB UIO region. Against today's eager mapping the series cut mmap() by 95x on the 256MB region and 5.6x on the 8MB one, total setup by about 12% on both, and the dTLB read-miss count by about 80x.

Changes since v1
================

- Patch 2: the order-0 path passes the fault's write flag too, so a caller that does not retry the access, fixup_user_fault() from an ->access handler, is not left with a read-only pte. vmf_insert_pfn_prot() keeps its signature; its body moved into a static __vmf_insert_pfn_prot() taking mkwrite.

- Patches 3 and 4: COW mappings get their own vm_operations_struct defining only .access, instead of a COW test inside the fault handler, which was not enough: installing ->fault at all skips do_fault()'s recheck of a pte cleared under the page table lock.

- Commit messages: corrected the L1TF claim in patches 3 and 4 - mmap(PROT_NONE) above l1tf_pfn_limit() now succeeds, and nothing fails later - and dropped patch 2's claim that uffd-wp can write-protect such a mapping.

- Patch 3: the commit message now also documents what the fault-time range_is_allowed() re-check changes without CONFIG_IO_STRICT_DEVMEM, where revoke_iomem() is a no-op.

The write-flag and COW changes came from sashiko-bot's review of v1.

v1: https://lore.kernel.org/all/[email protected]/

Eliav Farber (4):
mm: add MMAP_PFN_TRACK action and pfn_range_track() for lazy PFN maps
mm: add vmf_insert_pfn_order() and vmf_pfn_order_ok() helpers mm/char-mem: implement huge_fault support for /dev/mem
uio: implement huge_fault support for physical mappings

 drivers/uio/uio.c               | 118 ++++++++++++++++--
 include/linux/mm.h              |  47 ++++++++
 include/linux/mm_types.h        |   1 +
 mm/char-mem.c                   | 108 ++++++++++++++++-
 mm/internal.h                   |   3 +
 mm/memory.c                     | 208 ++++++++++++++++++++++++++++----
 mm/util.c                       |   7 ++
 tools/testing/vma/include/dup.h |   1 +
 8 files changed, 458 insertions(+), 35 deletions(-)

base-commit: 067c5c24cef23d913b67eaaf71ae67e7169a6620

Signature · 1 line
2.47.3