Skip to content

RFC per-VMA locks proposal

Suren Baghdasaryan linux-arm-kernel linux-mm linuxppc-dev lkml 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest28 patches 0 repliesNew

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/28mm: introduce CONFIG_PER_VMA_LOCKmm/Kconfig+13 -00 replies
2/28mm: rcu safe VMA freeinginclude/linux/mm_types.h kernel/fork.c+24 -50 replies
3/28mm: introduce __find_vma to be used without mmap_lock protectiondrivers/gpu/drm/i915/i915_gpu_error.c include/linux/mm.h mm/mmap.c+12 -70 replies
4/28mm: move mmap_lock assert function definitionsinclude/linux/mmap_lock.h+12 -120 replies
5/28mm: add per-VMA lock and helper functions to control itinclude/linux/mm.h include/linux/mm_types.h include/linux/mmap_lock.h +2 more+105 -00 replies
6/28mm: mark VMA as locked whenever vma->vm_flags are modifiedfs/proc/task_mmu.c fs/userfaultfd.c mm/madvise.c +3 more+12 -00 replies
7/28kernel/fork: mark VMAs as locked before copying pages during forkkernel/fork.c+3 -10 replies
8/28mm/khugepaged: mark VMA as locked while collapsing a hugepagemm/khugepaged.c+1 -00 replies
9/28mm/mempolicy: mark VMA as locked when changing protection policymm/mempolicy.c+5 -10 replies
10/28mm/mmap: mark VMAs as locked in vma_adjustmm/mmap.c+10 -10 replies
11/28mm/mmap: mark VMAs as locked before merging or splitting themmm/mmap.c+9 -10 replies
12/28mm/mremap: mark VMA as locked while remapping it to a new address rangemm/mmap.c mm/mremap.c+2 -00 replies
13/28mm: conditionally mark VMA as locked in free_pgtables and unmap_page_rangeinclude/linux/mm.h mm/internal.h mm/memory.c +2 more+35 -230 replies
14/28mm: mark VMAs as locked before isolating themmm/mmap.c mm/nommu.c+4 -00 replies
15/28mm/mmap: mark adjacent VMAs as locked if they can grow into unmapped areamm/mmap.c+6 -20 replies
16/28kernel/fork: assert no VMA readers during its destructioninclude/linux/mm.h kernel/fork.c+10 -00 replies
17/28mm/mmap: prevent pagefault handler from racing with mmu_notifier registrationmm/mmap.c+3 -00 replies
18/28mm: add FAULT_FLAG_VMA_LOCK flaginclude/linux/mm.h include/linux/mm_types.h+3 -10 replies
19/28mm: disallow do_swap_page to handle page faults under VMA lockmm/memory.c+5 -00 replies
20/28mm: introduce per-VMA lock statisticsinclude/linux/vm_event_item.h include/linux/vmstat.h mm/Kconfig.debug +1 more+26 -00 replies
21/28mm: introduce find_and_lock_anon_vma to be used from arch-specific codeinclude/linux/mm.h mm/memory.c+48 -00 replies
22/28x86/mm: try VMA lock-based page fault handling firstarch/x86/mm/fault.c+36 -00 replies
23/28x86/mm: define ARCH_SUPPORTS_PER_VMA_LOCKarch/x86/Kconfig+1 -00 replies
24/28arm64/mm: try VMA lock-based page fault handling firstarch/arm64/mm/fault.c+36 -00 replies
25/28arm64/mm: define ARCH_SUPPORTS_PER_VMA_LOCKarch/arm64/Kconfig+1 -00 replies
26/28powerc/mm: try VMA lock-based page fault handling firstarch/powerpc/mm/fault.c+41 -00 replies
27/28powerpc/mm: define ARCH_SUPPORTS_PER_VMA_LOCKarch/powerpc/platforms/powernv/Kconfig arch/powerpc/platforms/pseries/Kconfig+2 -00 replies
28/28kernel/fork: throttle call_rcu() calls in vm_area_freeinclude/linux/mm.h include/linux/mm_types.h kernel/fork.c +2 more+75 -90 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 00/28] per-VMA locks proposal

This is a proof of concept for per-vma locks idea that was discussed during SPF [1] discussion at LSF/MM this year [2], which concluded with suggestion that “a reader/writer semaphore could be put into the VMA itself; that would have the effect of using the VMA as a sort of range lock. There would still be contention at the VMA level, but it would be an improvement.” This patchset implements this suggested approach.

When handling page faults we lookup the VMA that contains the faulting page under RCU protection and try to acquire its lock. If that fails we fall back to using mmap_lock, similar to how SPF handled this situation.

One notable way the implementation deviates from the proposal is the way VMAs are marked as locked. Because during some of mm updates multiple VMAs need to be locked until the end of the update (e.g. vma_merge, split_vma, etc). Tracking all the locked VMAs, avoiding recursive locks and other complications would make the code more complex. Therefore we provide a way to "mark" VMAs as locked and then unmark all locked VMAs all at once. This is done using two sequence numbers - one in the vm_area_struct and one in the mm_struct. VMA is considered locked when these sequence numbers are equal. To mark a VMA as locked we set the sequence number in vm_area_struct to be equal to the sequence number in mm_struct. To unlock all VMAs we increment mm_struct's seq number. This allows for an efficient way to track locked VMAs and to drop the locks on all VMAs at the end of the update.

The patchset implements per-VMA locking only for anonymous pages which are not in swap. If the initial proposal is considered acceptable, then support for swapped and file-backed page faults will be added.

Performance benchmarks show similar although slightly smaller benefits as with SPF patchset (~75% of SPF benefits). Still, with lower complexity this approach might be more desirable.

The patchset applies cleanly over 6.0-rc3
The tree for testing is posted at [3]

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lwn.net/Articles/893906/
[3] https://github.com/surenbaghdasaryan/linux/tree/per_vma_lock_rfc

Laurent Dufour (2):
powerc/mm: try VMA lock-based page fault handling first powerpc/mm: define ARCH_SUPPORTS_PER_VMA_LOCK

Michel Lespinasse (1):
mm: rcu safe VMA freeing

Suren Baghdasaryan (25):
mm: introduce CONFIG_PER_VMA_LOCK
mm: introduce __find_vma to be used without mmap_lock protection
mm: move mmap_lock assert function definitions
mm: add per-VMA lock and helper functions to control it
mm: mark VMA as locked whenever vma->vm_flags are modified kernel/fork: mark VMAs as locked before copying pages during fork mm/khugepaged: mark VMA as locked while collapsing a hugepage mm/mempolicy: mark VMA as locked when changing protection policy mm/mmap: mark VMAs as locked in vma_adjust
mm/mmap: mark VMAs as locked before merging or splitting them mm/mremap: mark VMA as locked while remapping it to a new address
range
mm: conditionally mark VMA as locked in free_pgtables and
unmap_page_range
mm: mark VMAs as locked before isolating them
mm/mmap: mark adjacent VMAs as locked if they can grow into unmapped
area
kernel/fork: assert no VMA readers during its destruction mm/mmap: prevent pagefault handler from racing with mmu_notifier
registration
mm: add FAULT_FLAG_VMA_LOCK flag
mm: disallow do_swap_page to handle page faults under VMA lock
mm: introduce per-VMA lock statistics
mm: introduce find_and_lock_anon_vma to be used from arch-specific
code
x86/mm: try VMA lock-based page fault handling first x86/mm: define ARCH_SUPPORTS_PER_VMA_LOCK
arm64/mm: try VMA lock-based page fault handling first arm64/mm: define ARCH_SUPPORTS_PER_VMA_LOCK
kernel/fork: throttle call_rcu() calls in vm_area_free

 arch/arm64/Kconfig                     |   1 +
 arch/arm64/mm/fault.c                  |  36 +++++++++
 arch/powerpc/mm/fault.c                |  41 ++++++++++
 arch/powerpc/platforms/powernv/Kconfig |   1 +
 arch/powerpc/platforms/pseries/Kconfig |   1 +
 arch/x86/Kconfig                       |   1 +
 arch/x86/mm/fault.c                    |  36 +++++++++
 drivers/gpu/drm/i915/i915_gpu_error.c  |   4 +-
 fs/proc/task_mmu.c                     |   1 +
 fs/userfaultfd.c                       |   6 ++
 include/linux/mm.h                     | 104 ++++++++++++++++++++++++-
 include/linux/mm_types.h               |  33 ++++++--
 include/linux/mmap_lock.h              |  37 ++++++---
 include/linux/vm_event_item.h          |   6 ++
 include/linux/vmstat.h                 |   6 ++
 kernel/fork.c                          |  75 +++++++++++++++++-
 mm/Kconfig                             |  13 ++++
 mm/Kconfig.debug                       |   8 ++
 mm/init-mm.c                           |   6 ++
 mm/internal.h                          |   4 +-
 mm/khugepaged.c                        |   1 +
 mm/madvise.c                           |   1 +
 mm/memory.c                            |  82 ++++++++++++++++---
 mm/mempolicy.c                         |   6 +-
 mm/mlock.c                             |   2 +
 mm/mmap.c                              |  60 ++++++++++----
 mm/mprotect.c                          |   1 +
 mm/mremap.c                            |   1 +
 mm/nommu.c                             |   2 +
 mm/oom_kill.c                          |   3 +-
 mm/vmstat.c                            |   6 ++
 31 files changed, 531 insertions(+), 54 deletions(-)
Signature · 1 line
2.37.2.672.g94769d06f0-goog