Skip to content

RFC Fine grained MM locking

Michel Lespinasse linux-mm 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest24 patches 3 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/24MM locking API: initial implementation as rwsem wrapperslinux/mm.h linux/mm_lock.h+60 -00 replies
2/24MM locking API: use coccinelle to convert mmap_sem rwsem call sitesacct.c aio.c alpha/kernel/traps.c +145 more+653 -6520 replies
3/24MM locking API: manual conversion of mmap_sem call sites missed by coccinelleandroid/binder_alloc.c mips/mm/fault.c proc/base.c +1 more+14 -140 replies
4/24MM locking API: add range argumentsKconfig Makefile bpf/stackmap.c +7 more+322 -220 replies
5/24MM locking API: allow for sleeping during unlockbpf/stackmap.c exit.c mm_lock_rwsem_checked.c+9 -20 replies
6/24MM locking API: implement fine grained range locksKconfig Makefile firmware/efi/efi.c +6 more+827 -120 replies
7/24mm/memory: add range field to struct vm_faulthugetlb.c khugepaged.c linux/mm.h +1 more+4 -00 replies
8/24mm/memory: allow specifying MM lock range to handle_mm_fault()hugetlb.c linux/hugetlb.h linux/mm.h +1 more+30 -160 replies
9/24do_swap_page: use the vmf->range field when dropping mmap_semfilemap.c linux/pagemap.h memory.c+8 -70 replies
10/24handle_userfault: use the vmf->range field when dropping mmap_semuserfaultfd.c+2 -20 replies
11/24x86 fault handler: merge bad_area() functionsx86/mm/fault.c+24 -300 replies
12/24x86 fault handler: use an explicit MM lock rangex86/mm/fault.c+14 -100 replies
13/24mm/memory: add prepare_mm_fault() functionlinux/mm.h memory.c+40 -00 replies
14/24mm/swap_state: disable swap vma readaheadswap_state.c+6 -00 replies
15/24x86 fault handler: use a pseudo-vma when operating on anonymous vmas.x86/mm/fault.c+19 -10 replies
16/24MM locking API: add vma locking APIlinux/mm_lock.h linux/mm_types.h+22 -40 replies
17/24x86 fault handler: implement range lockingx86/mm/fault.c+32 -80 replies
18/24shared file mappings: use the vmf->range field when dropping mmap_semfilemap.c internal.h+3 -20 replies
19/24mm: add field to annotate vm_operations that support range lockingext4/file.c filemap.c linux/mm.h +1 more+19 -00 replies
20/24x86 fault handler: extend range locking to supported file vmasx86/mm/fault.c+2 -20 replies
21/24do_mmap: add locked argumentaio.c linux/mm.h mmap.c +4 more+16 -220 replies
22/24do_mmap: implement locked argumentmmap.c nommu.c+89 -360 replies
23/24do_mmap: use locked=false in vm_mmap_pgoff() and aio_setup_ring()aio.c util.c+5 -150 replies
24/24do_mmap: implement easiest cases of fine grained lockingmmap.c+29 -70 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 00/24] Fine grained MM locking

Hi,

This is the first version of my work towards fine grained MM locking. This is still early work - I am happy with my page fault changes, but want to expand on the mmap/munmap side of things before I send the next version. I have previously shared this with some of the copied folks (for those who received that, there are no additional changes in this public resend). Please expect a v2 within a few weeks, with further changes for fine grained range locking in the mmap and munmap paths.

This work originated in discussions at LSF/MM 2019; it is intended to address the latency issues that are caused by false conflicts between threads working on separate parts of their address space. The priorities are to keep things as simple as possible, and to allow for progressive conversion of the code base to finer grained MM locks.

The general approach is to replace the mmap_sem rwsem with a range lock. Initially all lock/unlock sites are automatically converted to lock the entire address space through a new API. Then, the API is extended to support range locking. Locking sites can then be progressively converted to use range locking, while leaving unconverted sites working with no code changes.

When using a range lock (as opposed to a coarse lock), the following rules apply:
- Some structures (notably the vma rbtree and associated statistics) are per-mm. They need to be locked separately using a new mm_vma_lock. The entire point of this patch set is to reduce false sharing latencies, so the mm_vma_lock must be held only for short times. We expect to do O(log N) operations holding the lock (for example, walking or updating the vma rbtree) but no O(N) operations (such as iterating on all vmas within a range or all mapped pages within a range).
- Code holding the mm_vma_lock should only update vma attributes for the range it has a write lock for. However, range locks only protects the vma's attributes, not the vmas themselves - vmas can still be split or merged with their neighbors if they have compatible attributes.
- Code holding a range lock but not the mm_vma_lock must be prepared for the vmas at both ends of the locked range to be merged with their neighbors outside of the locked range. The easiest way to do that is to copy the vma of record into a pseudo-vma before releasing the mm_vma_lock (this is a bit kludgy and I would prefer to copy only the necessary VMA attributes, but using a pseudo-vma makes it easier to maintain this patchset out of mainline for the moment).

Call sites that take a range lock usualy immediately take the mm_vma_lock next - it would probably be more efficient to collapse mm_vma_lock with the mutex that protects the range lock structures. This isn't done yet as I tried to simplify the initial implementation.

In the future I would also like to remove the various workarounds we have been doing to limit mmap_sem hold times (i.e. FAULT_FLAG_ALLOW_RETRY, vm_populate and munmap downgrading to a read lock, ...) which shouldn't be necessary if the locking was only effective on the memory ranges affected by each operation.

The included changes apply on top of upstream kernel v5.5. Please apply with git am -p0 - I'm not sure why my git format-patch setup requires that.

Commits 1 to 6 implement a range locking API:
- 1 implements coarse locking as wrappers around rwsem;
- 2 converts most mmap_sem locking sites to use the new coarse locking API (using coccinelle to automate the conversion);
- 3 converts remaining mmap_sem locking sites which were missed by coccinelle;
- 4 extends the API to support range locking. The initial implementation still uses coarse locking (ignoring the range); but it validates that the callers use matching ranges in lock and unlock calls;
- 5 prepares callers to allow for sleeping during unlock;
- 6 actually implements the range locking functions.

Commits 7 to 12 allow the x86 fault handler to specify a range that may be released while handling the fault:
- 7 adds a range field to struct mm_fault;
- 8 makes handle_mm_fault() populate that field;
- 9 and 10 honor it when dropping mmap_sem during fault handling;
- 11 is a cleanup to the x86 fault handler to prepare for 12;
- 12 changes the x86 fault handler to use an explicit lock range.

Commits 13 to 15 prepare for operating on a pseudo-vma during faults:
- 13 adds a prepare_vma_fault which may update the vma of record (specifically, allocate an anon_vma) before creating the pseudo-vma;
- 14 disables swap vma readahead as its implementation keeps stats in the vma;
- 15 changes the x86 fault handler to use pseudo-vmas when handling anon vmas.

Commits 16 and 17 implement range locking in x86 anonymous vma faults:
- Commit 16 adds the vma locking API to be used to manipulate vmas when holding a fine grained ranged lock;
- Commit 17 converts the x86 fault handler to use a pmd sized range lock when operating on anon vmas.

Commits 18 to 20 extend the above to also work on filemap based files:
- Commit 18 makes sure we release the correct range when dropping mmap_sem during filemap file access;
- Commit 19 tags vm_operations that support range locking;
- Commit 20 makes the x86 fault handler use fine grained ranges when faulting the supported files.

Commits 21 to 24 implement range locking for the most basic mmap() case:
- 21 adds a locked argument to do_mmap();
- 22 makes do_mmap acquire the mmap_sem if locked is false;
- 23 converts soem easy call sites to pass locked=false;
- 24 changes do_mmap to acquire a fine grained lock in the easiest case (anonymous mapping, known address, no prior existing mapping).

Michel Lespinasse (24):
MM locking API: initial implementation as rwsem wrappers MM locking API: use coccinelle to convert mmap_sem rwsem call sites MM locking API: manual conversion of mmap_sem call sites missed by
coccinelle
MM locking API: add range arguments
MM locking API: allow for sleeping during unlock
MM locking API: implement fine grained range locks mm/memory: add range field to struct vm_fault
mm/memory: allow specifying MM lock range to handle_mm_fault()
do_swap_page: use the vmf->range field when dropping mmap_sem
handle_userfault: use the vmf->range field when dropping mmap_sem x86 fault handler: merge bad_area() functions
x86 fault handler: use an explicit MM lock range mm/memory: add prepare_mm_fault() function
mm/swap_state: disable swap vma readahead
x86 fault handler: use a pseudo-vma when operating on anonymous vmas. MM locking API: add vma locking API
x86 fault handler: implement range locking
shared file mappings: use the vmf->range field when dropping mmap_sem
mm: add field to annotate vm_operations that support range locking x86 fault handler: extend range locking to supported file vmas
do_mmap: add locked argument
do_mmap: implement locked argument
do_mmap: use locked=false in vm_mmap_pgoff() and aio_setup_ring()
do_mmap: implement easiest cases of fine grained locking

 arch/alpha/kernel/traps.c                     |   4 +-
 arch/alpha/mm/fault.c                         |  10 +-
 arch/arc/kernel/process.c                     |   4 +-
 arch/arc/kernel/troubleshoot.c                |   4 +-
 arch/arc/mm/fault.c                           |   4 +-
 arch/arm/kernel/process.c                     |   4 +-
 arch/arm/kernel/swp_emulate.c                 |   4 +-
 arch/arm/lib/uaccess_with_memcpy.c            |  16 +-
 arch/arm/mm/fault.c                           |   6 +-
 arch/arm64/kernel/traps.c                     |   4 +-
 arch/arm64/kernel/vdso.c                      |   8 +-
 arch/arm64/mm/fault.c                         |   8 +-
 arch/csky/kernel/vdso.c                       |   4 +-
 arch/csky/mm/fault.c                          |   8 +-
 arch/hexagon/kernel/vdso.c                    |   4 +-
 arch/hexagon/mm/vm_fault.c                    |   8 +-
 arch/ia64/kernel/perfmon.c                    |   8 +-
 arch/ia64/mm/fault.c                          |   8 +-
 arch/ia64/mm/init.c                           |  12 +-
 arch/m68k/kernel/sys_m68k.c                   |  14 +-
 arch/m68k/mm/fault.c                          |   8 +-
 arch/microblaze/mm/fault.c                    |  12 +-
 arch/mips/kernel/traps.c                      |   4 +-
 arch/mips/kernel/vdso.c                       |   4 +-
 arch/mips/mm/fault.c                          |  10 +-
 arch/nds32/kernel/vdso.c                      |   6 +-
 arch/nds32/mm/fault.c                         |  12 +-
 arch/nios2/mm/fault.c                         |  12 +-
 arch/nios2/mm/init.c                          |   4 +-
 arch/openrisc/mm/fault.c                      |  10 +-
 arch/parisc/kernel/traps.c                    |   6 +-
 arch/parisc/mm/fault.c                        |   8 +-
 arch/powerpc/kernel/vdso.c                    |   6 +-
 arch/powerpc/kvm/book3s_64_mmu_hv.c           |   4 +-
 arch/powerpc/kvm/book3s_hv.c                  |   6 +-
 arch/powerpc/kvm/book3s_hv_uvmem.c            |  12 +-
 arch/powerpc/kvm/e500_mmu_host.c              |   4 +-
 arch/powerpc/mm/book3s64/iommu_api.c          |   4 +-
 arch/powerpc/mm/book3s64/subpage_prot.c       |  12 +-
 arch/powerpc/mm/copro_fault.c                 |   4 +-
 arch/powerpc/mm/fault.c                       |  12 +-
 arch/powerpc/oprofile/cell/spu_task_sync.c    |   6 +-
 arch/powerpc/platforms/cell/spufs/file.c      |   4 +-
 arch/riscv/kernel/vdso.c                      |   4 +-
 arch/riscv/mm/fault.c                         |  10 +-
 arch/s390/kernel/vdso.c                       |   4 +-
 arch/s390/kvm/gaccess.c                       |   4 +-
 arch/s390/kvm/kvm-s390.c                      |  24 +-
 arch/s390/kvm/priv.c                          |  32 +-
 arch/s390/mm/fault.c                          |   6 +-
 arch/s390/mm/gmap.c                           |  40 +-
 arch/s390/pci/pci_mmio.c                      |   4 +-
 arch/sh/kernel/sys_sh.c                       |   6 +-
 arch/sh/kernel/vsyscall/vsyscall.c            |   4 +-
 arch/sh/mm/fault.c                            |  14 +-
 arch/sparc/mm/fault_32.c                      |  18 +-
 arch/sparc/mm/fault_64.c                      |  12 +-
 arch/sparc/vdso/vma.c                         |   4 +-
 arch/um/include/asm/mmu_context.h             |   6 +-
 arch/um/kernel/tlb.c                          |   2 +-
 arch/um/kernel/trap.c                         |   6 +-
 arch/unicore32/mm/fault.c                     |   6 +-
 arch/x86/entry/vdso/vma.c                     |  10 +-
 arch/x86/kernel/tboot.c                       |   2 +-
 arch/x86/kernel/vm86_32.c                     |   4 +-
 arch/x86/kvm/mmu/paging_tmpl.h                |   8 +-
 arch/x86/mm/debug_pagetables.c                |   8 +-
 arch/x86/mm/fault.c                           | 110 ++-
 arch/x86/mm/mpx.c                             |  15 +-
 arch/x86/um/vdso/vma.c                        |   4 +-
 arch/xtensa/mm/fault.c                        |  10 +-
 drivers/android/binder_alloc.c                |  10 +-
 drivers/firmware/efi/efi.c                    |   2 +-
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |   4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |  10 +-
 drivers/gpu/drm/amd/amdkfd/kfd_events.c       |   4 +-
 drivers/gpu/drm/i915/gem/i915_gem_mman.c      |   4 +-
 drivers/gpu/drm/i915/gem/i915_gem_userptr.c   |   8 +-
 drivers/gpu/drm/nouveau/nouveau_svm.c         |  20 +-
 drivers/gpu/drm/radeon/radeon_cs.c            |   4 +-
 drivers/gpu/drm/radeon/radeon_gem.c           |   6 +-
 drivers/gpu/drm/ttm/ttm_bo_vm.c               |   4 +-
 drivers/infiniband/core/umem.c                |   6 +-
 drivers/infiniband/core/umem_odp.c            |  10 +-
 drivers/infiniband/core/uverbs_main.c         |   4 +-
 drivers/infiniband/hw/mlx4/mr.c               |   4 +-
 drivers/infiniband/hw/qib/qib_user_pages.c    |   6 +-
 drivers/infiniband/hw/usnic/usnic_uiom.c      |   4 +-
 drivers/infiniband/sw/siw/siw_mem.c           |   4 +-
 drivers/iommu/amd_iommu_v2.c                  |   4 +-
 drivers/iommu/intel-svm.c                     |   4 +-
 drivers/media/v4l2-core/videobuf-core.c       |   4 +-
 drivers/media/v4l2-core/videobuf-dma-contig.c |   4 +-
 drivers/media/v4l2-core/videobuf-dma-sg.c     |   4 +-
 drivers/misc/cxl/cxllib.c                     |   4 +-
 drivers/misc/cxl/fault.c                      |   4 +-
 drivers/misc/sgi-gru/grufault.c               |  16 +-
 drivers/misc/sgi-gru/grufile.c                |   4 +-
 drivers/oprofile/buffer_sync.c                |  10 +-
 drivers/staging/kpc2000/kpc_dma/fileops.c     |   4 +-
 drivers/tee/optee/call.c                      |   4 +-
 drivers/vfio/vfio_iommu_type1.c               |  12 +-
 drivers/xen/gntdev.c                          |   4 +-
 drivers/xen/privcmd.c                         |  14 +-
 fs/aio.c                                      |  16 +-
 fs/coredump.c                                 |   4 +-
 fs/exec.c                                     |  16 +-
 fs/ext4/file.c                                |   1 +
 fs/io_uring.c                                 |   4 +-
 fs/proc/base.c                                |  18 +-
 fs/proc/task_mmu.c                            |  28 +-
 fs/proc/task_nommu.c                          |  18 +-
 fs/userfaultfd.c                              |  28 +-
 include/linux/hugetlb.h                       |   5 +-
 include/linux/mm.h                            |  56 +-
 include/linux/mm_lock.h                       | 285 ++++++++
 include/linux/mm_types.h                      |  22 +
 include/linux/mm_types_task.h                 |  21 +
 include/linux/mmu_notifier.h                  |   5 +-
 include/linux/pagemap.h                       |   7 +-
 include/linux/sched.h                         |   2 +
 init/init_task.c                              |   1 +
 ipc/shm.c                                     |  11 +-
 kernel/acct.c                                 |   4 +-
 kernel/bpf/stackmap.c                         |  32 +-
 kernel/events/core.c                          |   4 +-
 kernel/events/uprobes.c                       |  16 +-
 kernel/exit.c                                 |   8 +-
 kernel/fork.c                                 |  17 +-
 kernel/futex.c                                |   4 +-
 kernel/sched/fair.c                           |   4 +-
 kernel/sys.c                                  |  18 +-
 kernel/trace/trace_output.c                   |   4 +-
 mm/Kconfig                                    |  25 +
 mm/Makefile                                   |   2 +
 mm/filemap.c                                  |  10 +-
 mm/frame_vector.c                             |   4 +-
 mm/gup.c                                      |  20 +-
 mm/hugetlb.c                                  |  13 +-
 mm/init-mm.c                                  |   3 +-
 mm/internal.h                                 |   2 +-
 mm/khugepaged.c                               |  37 +-
 mm/ksm.c                                      |  34 +-
 mm/madvise.c                                  |  18 +-
 mm/memcontrol.c                               |   8 +-
 mm/memory.c                                   |  55 +-
 mm/mempolicy.c                                |  22 +-
 mm/migrate.c                                  |   8 +-
 mm/mincore.c                                  |   4 +-
 mm/mlock.c                                    |  16 +-
 mm/mm_lock_range.c                            | 691 ++++++++++++++++++
 mm/mm_lock_rwsem_checked.c                    | 134 ++++
 mm/mmap.c                                     | 170 +++--
 mm/mmu_notifier.c                             |   4 +-
 mm/mprotect.c                                 |  12 +-
 mm/mremap.c                                   |   6 +-
 mm/msync.c                                    |   8 +-
 mm/nommu.c                                    |  36 +-
 mm/oom_kill.c                                 |   4 +-
 mm/process_vm_access.c                        |   4 +-
 mm/shmem.c                                    |   1 +
 mm/swap_state.c                               |   6 +
 mm/swapfile.c                                 |   4 +-
 mm/userfaultfd.c                              |  14 +-
 mm/util.c                                     |  14 +-
 net/ipv4/tcp.c                                |   4 +-
 net/xdp/xdp_umem.c                            |   4 +-
 virt/kvm/arm/mmu.c                            |  14 +-
 virt/kvm/async_pf.c                           |   4 +-
 virt/kvm/kvm_main.c                           |   8 +-
 170 files changed, 2183 insertions(+), 798 deletions(-)
 create mode 100644 include/linux/mm_lock.h
 create mode 100644 mm/mm_lock_range.c
 create mode 100644 mm/mm_lock_rwsem_checked.c
Signature · 1 line
2.25.0.341.g760bfbb309-goog