Fine grained MM locking
Subsystems (4)
cover Fine grained MM locking
Hi,
This is the first version of my work towards fine grained MM locking. This is still early work - I am happy with my page fault changes, but want to expand on the mmap/munmap side of things before I send the next version. I have previously shared this with some of the copied folks (for those who received that, there are no additional changes in this public resend). Please expect a v2 within a few weeks, with further changes for fine grained range locking in the mmap and munmap paths.
This work originated in discussions at LSF/MM 2019; it is intended to address the latency issues that are caused by false conflicts between threads working on separate parts of their address space. The priorities are to keep things as simple as possible, and to allow for progressive conversion of the code base to finer grained MM locks.
The general approach is to replace the mmap_sem rwsem with a range lock. Initially all lock/unlock sites are automatically converted to lock the entire address space through a new API. Then, the API is extended to support range locking. Locking sites can then be progressively converted to use range locking, while leaving unconverted sites working with no code changes.
When using a range lock (as opposed to a coarse lock), the following rules apply:
- Some structures (notably the vma rbtree and associated statistics) are per-mm. They need to be locked separately using a new mm_vma_lock. The entire point of this patch set is to reduce false sharing latencies, so the mm_vma_lock must be held only for short times. We expect to do O(log N) operations holding the lock (for example, walking or updating the vma rbtree) but no O(N) operations (such as iterating on all vmas within a range or all mapped pages within a range).
- Code holding the mm_vma_lock should only update vma attributes for the range it has a write lock for. However, range locks only protects the vma's attributes, not the vmas themselves - vmas can still be split or merged with their neighbors if they have compatible attributes.
- Code holding a range lock but not the mm_vma_lock must be prepared for the vmas at both ends of the locked range to be merged with their neighbors outside of the locked range. The easiest way to do that is to copy the vma of record into a pseudo-vma before releasing the mm_vma_lock (this is a bit kludgy and I would prefer to copy only the necessary VMA attributes, but using a pseudo-vma makes it easier to maintain this patchset out of mainline for the moment).
Call sites that take a range lock usualy immediately take the mm_vma_lock next - it would probably be more efficient to collapse mm_vma_lock with the mutex that protects the range lock structures. This isn't done yet as I tried to simplify the initial implementation.
In the future I would also like to remove the various workarounds we have been doing to limit mmap_sem hold times (i.e. FAULT_FLAG_ALLOW_RETRY, vm_populate and munmap downgrading to a read lock, ...) which shouldn't be necessary if the locking was only effective on the memory ranges affected by each operation.
The included changes apply on top of upstream kernel v5.5. Please apply with git am -p0 - I'm not sure why my git format-patch setup requires that.
Commits 1 to 6 implement a range locking API:
- 1 implements coarse locking as wrappers around rwsem;
- 2 converts most mmap_sem locking sites to use the new coarse locking API (using coccinelle to automate the conversion);
- 3 converts remaining mmap_sem locking sites which were missed by coccinelle;
- 4 extends the API to support range locking. The initial implementation still uses coarse locking (ignoring the range); but it validates that the callers use matching ranges in lock and unlock calls;
- 5 prepares callers to allow for sleeping during unlock;
- 6 actually implements the range locking functions.
Commits 7 to 12 allow the x86 fault handler to specify a range that may be released while handling the fault:
- 7 adds a range field to struct mm_fault;
- 8 makes handle_mm_fault() populate that field;
- 9 and 10 honor it when dropping mmap_sem during fault handling;
- 11 is a cleanup to the x86 fault handler to prepare for 12;
- 12 changes the x86 fault handler to use an explicit lock range.
Commits 13 to 15 prepare for operating on a pseudo-vma during faults:
- 13 adds a prepare_vma_fault which may update the vma of record (specifically, allocate an anon_vma) before creating the pseudo-vma;
- 14 disables swap vma readahead as its implementation keeps stats in the vma;
- 15 changes the x86 fault handler to use pseudo-vmas when handling anon vmas.
Commits 16 and 17 implement range locking in x86 anonymous vma faults:
- Commit 16 adds the vma locking API to be used to manipulate vmas when holding a fine grained ranged lock;
- Commit 17 converts the x86 fault handler to use a pmd sized range lock when operating on anon vmas.
Commits 18 to 20 extend the above to also work on filemap based files:
- Commit 18 makes sure we release the correct range when dropping mmap_sem during filemap file access;
- Commit 19 tags vm_operations that support range locking;
- Commit 20 makes the x86 fault handler use fine grained ranges when faulting the supported files.
Commits 21 to 24 implement range locking for the most basic mmap() case:
- 21 adds a locked argument to do_mmap();
- 22 makes do_mmap acquire the mmap_sem if locked is false;
- 23 converts soem easy call sites to pass locked=false;
- 24 changes do_mmap to acquire a fine grained lock in the easiest case (anonymous mapping, known address, no prior existing mapping).
Michel Lespinasse (24):
MM locking API: initial implementation as rwsem wrappers MM locking API: use coccinelle to convert mmap_sem rwsem call sites MM locking API: manual conversion of mmap_sem call sites missed by
coccinelle
MM locking API: add range arguments
MM locking API: allow for sleeping during unlock
MM locking API: implement fine grained range locks mm/memory: add range field to struct vm_fault
mm/memory: allow specifying MM lock range to handle_mm_fault()
do_swap_page: use the vmf->range field when dropping mmap_sem
handle_userfault: use the vmf->range field when dropping mmap_sem x86 fault handler: merge bad_area() functions
x86 fault handler: use an explicit MM lock range mm/memory: add prepare_mm_fault() function
mm/swap_state: disable swap vma readahead
x86 fault handler: use a pseudo-vma when operating on anonymous vmas. MM locking API: add vma locking API
x86 fault handler: implement range locking
shared file mappings: use the vmf->range field when dropping mmap_sem
mm: add field to annotate vm_operations that support range locking x86 fault handler: extend range locking to supported file vmas
do_mmap: add locked argument
do_mmap: implement locked argument
do_mmap: use locked=false in vm_mmap_pgoff() and aio_setup_ring()
do_mmap: implement easiest cases of fine grained locking
arch/alpha/kernel/traps.c | 4 +- arch/alpha/mm/fault.c | 10 +- arch/arc/kernel/process.c | 4 +- arch/arc/kernel/troubleshoot.c | 4 +- arch/arc/mm/fault.c | 4 +- arch/arm/kernel/process.c | 4 +- arch/arm/kernel/swp_emulate.c | 4 +- arch/arm/lib/uaccess_with_memcpy.c | 16 +- arch/arm/mm/fault.c | 6 +- arch/arm64/kernel/traps.c | 4 +- arch/arm64/kernel/vdso.c | 8 +- arch/arm64/mm/fault.c | 8 +- arch/csky/kernel/vdso.c | 4 +- arch/csky/mm/fault.c | 8 +- arch/hexagon/kernel/vdso.c | 4 +- arch/hexagon/mm/vm_fault.c | 8 +- arch/ia64/kernel/perfmon.c | 8 +- arch/ia64/mm/fault.c | 8 +- arch/ia64/mm/init.c | 12 +- arch/m68k/kernel/sys_m68k.c | 14 +- arch/m68k/mm/fault.c | 8 +- arch/microblaze/mm/fault.c | 12 +- arch/mips/kernel/traps.c | 4 +- arch/mips/kernel/vdso.c | 4 +- arch/mips/mm/fault.c | 10 +- arch/nds32/kernel/vdso.c | 6 +- arch/nds32/mm/fault.c | 12 +- arch/nios2/mm/fault.c | 12 +- arch/nios2/mm/init.c | 4 +- arch/openrisc/mm/fault.c | 10 +- arch/parisc/kernel/traps.c | 6 +- arch/parisc/mm/fault.c | 8 +- arch/powerpc/kernel/vdso.c | 6 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_hv.c | 6 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 12 +- arch/powerpc/kvm/e500_mmu_host.c | 4 +- arch/powerpc/mm/book3s64/iommu_api.c | 4 +- arch/powerpc/mm/book3s64/subpage_prot.c | 12 +- arch/powerpc/mm/copro_fault.c | 4 +- arch/powerpc/mm/fault.c | 12 +- arch/powerpc/oprofile/cell/spu_task_sync.c | 6 +- arch/powerpc/platforms/cell/spufs/file.c | 4 +- arch/riscv/kernel/vdso.c | 4 +- arch/riscv/mm/fault.c | 10 +- arch/s390/kernel/vdso.c | 4 +- arch/s390/kvm/gaccess.c | 4 +- arch/s390/kvm/kvm-s390.c | 24 +- arch/s390/kvm/priv.c | 32 +- arch/s390/mm/fault.c | 6 +- arch/s390/mm/gmap.c | 40 +- arch/s390/pci/pci_mmio.c | 4 +- arch/sh/kernel/sys_sh.c | 6 +- arch/sh/kernel/vsyscall/vsyscall.c | 4 +- arch/sh/mm/fault.c | 14 +- arch/sparc/mm/fault_32.c | 18 +- arch/sparc/mm/fault_64.c | 12 +- arch/sparc/vdso/vma.c | 4 +- arch/um/include/asm/mmu_context.h | 6 +- arch/um/kernel/tlb.c | 2 +- arch/um/kernel/trap.c | 6 +- arch/unicore32/mm/fault.c | 6 +- arch/x86/entry/vdso/vma.c | 10 +- arch/x86/kernel/tboot.c | 2 +- arch/x86/kernel/vm86_32.c | 4 +- arch/x86/kvm/mmu/paging_tmpl.h | 8 +- arch/x86/mm/debug_pagetables.c | 8 +- arch/x86/mm/fault.c | 110 ++- arch/x86/mm/mpx.c | 15 +- arch/x86/um/vdso/vma.c | 4 +- arch/xtensa/mm/fault.c | 10 +- drivers/android/binder_alloc.c | 10 +- drivers/firmware/efi/efi.c | 2 +- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 10 +- drivers/gpu/drm/amd/amdkfd/kfd_events.c | 4 +- drivers/gpu/drm/i915/gem/i915_gem_mman.c | 4 +- drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 8 +- drivers/gpu/drm/nouveau/nouveau_svm.c | 20 +- drivers/gpu/drm/radeon/radeon_cs.c | 4 +- drivers/gpu/drm/radeon/radeon_gem.c | 6 +- drivers/gpu/drm/ttm/ttm_bo_vm.c | 4 +- drivers/infiniband/core/umem.c | 6 +- drivers/infiniband/core/umem_odp.c | 10 +- drivers/infiniband/core/uverbs_main.c | 4 +- drivers/infiniband/hw/mlx4/mr.c | 4 +- drivers/infiniband/hw/qib/qib_user_pages.c | 6 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 4 +- drivers/infiniband/sw/siw/siw_mem.c | 4 +- drivers/iommu/amd_iommu_v2.c | 4 +- drivers/iommu/intel-svm.c | 4 +- drivers/media/v4l2-core/videobuf-core.c | 4 +- drivers/media/v4l2-core/videobuf-dma-contig.c | 4 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 4 +- drivers/misc/cxl/cxllib.c | 4 +- drivers/misc/cxl/fault.c | 4 +- drivers/misc/sgi-gru/grufault.c | 16 +- drivers/misc/sgi-gru/grufile.c | 4 +- drivers/oprofile/buffer_sync.c | 10 +- drivers/staging/kpc2000/kpc_dma/fileops.c | 4 +- drivers/tee/optee/call.c | 4 +- drivers/vfio/vfio_iommu_type1.c | 12 +- drivers/xen/gntdev.c | 4 +- drivers/xen/privcmd.c | 14 +- fs/aio.c | 16 +- fs/coredump.c | 4 +- fs/exec.c | 16 +- fs/ext4/file.c | 1 + fs/io_uring.c | 4 +- fs/proc/base.c | 18 +- fs/proc/task_mmu.c | 28 +- fs/proc/task_nommu.c | 18 +- fs/userfaultfd.c | 28 +- include/linux/hugetlb.h | 5 +- include/linux/mm.h | 56 +- include/linux/mm_lock.h | 285 ++++++++ include/linux/mm_types.h | 22 + include/linux/mm_types_task.h | 21 + include/linux/mmu_notifier.h | 5 +- include/linux/pagemap.h | 7 +- include/linux/sched.h | 2 + init/init_task.c | 1 + ipc/shm.c | 11 +- kernel/acct.c | 4 +- kernel/bpf/stackmap.c | 32 +- kernel/events/core.c | 4 +- kernel/events/uprobes.c | 16 +- kernel/exit.c | 8 +- kernel/fork.c | 17 +- kernel/futex.c | 4 +- kernel/sched/fair.c | 4 +- kernel/sys.c | 18 +- kernel/trace/trace_output.c | 4 +- mm/Kconfig | 25 + mm/Makefile | 2 + mm/filemap.c | 10 +- mm/frame_vector.c | 4 +- mm/gup.c | 20 +- mm/hugetlb.c | 13 +- mm/init-mm.c | 3 +- mm/internal.h | 2 +- mm/khugepaged.c | 37 +- mm/ksm.c | 34 +- mm/madvise.c | 18 +- mm/memcontrol.c | 8 +- mm/memory.c | 55 +- mm/mempolicy.c | 22 +- mm/migrate.c | 8 +- mm/mincore.c | 4 +- mm/mlock.c | 16 +- mm/mm_lock_range.c | 691 ++++++++++++++++++ mm/mm_lock_rwsem_checked.c | 134 ++++ mm/mmap.c | 170 +++-- mm/mmu_notifier.c | 4 +- mm/mprotect.c | 12 +- mm/mremap.c | 6 +- mm/msync.c | 8 +- mm/nommu.c | 36 +- mm/oom_kill.c | 4 +- mm/process_vm_access.c | 4 +- mm/shmem.c | 1 + mm/swap_state.c | 6 + mm/swapfile.c | 4 +- mm/userfaultfd.c | 14 +- mm/util.c | 14 +- net/ipv4/tcp.c | 4 +- net/xdp/xdp_umem.c | 4 +- virt/kvm/arm/mmu.c | 14 +- virt/kvm/async_pf.c | 4 +- virt/kvm/kvm_main.c | 8 +- 170 files changed, 2183 insertions(+), 798 deletions(-) create mode 100644 include/linux/mm_lock.h create mode 100644 mm/mm_lock_range.c create mode 100644 mm/mm_lock_rwsem_checked.c
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 1/24 MM locking API: initial implementation as rwsem wrappers
This change wraps the existing mmap_sem related rwsem calls into a new MM locking API. This is in preparation to extending that API to support locking fine grained memory ranges.
--- include/linux/mm.h | 1 + include/linux/mm_lock.h | 59 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 include/linux/mm_lock.h
linux/mm.h +1 -0
@@ -15,6 +15,7 @@#include <linux/atomic.h> #include <linux/debug_locks.h> #include <linux/mm_types.h> +#include <linux/mm_lock.h> #include <linux/range.h> #include <linux/pfn.h> #include <linux/percpu-refcount.h>
linux/mm_lock.h new file +59 -0
59 lines not on this page · raw message
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 3/24 MM locking API: manual conversion of mmap_sem call sites missed by coccinelle
In reply to Michel LespinasseConvert the last few remaining mmap_sem rwsem calls to use the new MM locking API. These were missed by coccinelle for some reason (I think coccinelle does not support some of the preprocessor constructs in these files ?)
--- arch/mips/mm/fault.c | 10 +++++----- arch/x86/kvm/mmu/paging_tmpl.h | 8 ++++---- drivers/android/binder_alloc.c | 4 ++-- fs/proc/base.c | 6 +++--- 4 files changed, 14 insertions(+), 14 deletions(-)
mips/mm/fault.c +5 -5
@@ -97,7 +97,7 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,if (user_mode (regs)) flags |=FAULT_FLAG_USER ; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -191,7 +191,7 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write, } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ;/*
@@ -199,7 +199,7 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm); bad_area_nosemaphore:/* User mode accesses just cause a SIGSEGV */
@@ -251,14 +251,14 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,* We ran out of memory, call the OOM killer, and return the userspace * (which will retry the fault, or kill us if we got oom-killed). */ -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!user_mode (regs)) goto no_context;pagefault_out_of_memory ();return ; do_sigbus: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* Kernel mode? Handle exceptions or die */ if (!user_mode (regs))
x86/kvm/mmu/paging_tmpl.h +4 -4
@@ -150,22 +150,22 @@ static int FNAME(cmpxchg_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,unsigned long pfn;unsigned long paddr; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); vma =find_vma_intersection (current->mm, vaddr, vaddr +PAGE_SIZE );if (!vma || !(vma->vm_flags &VM_PFNMAP )) { -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);return -EFAULT ; } pfn = ((vaddr - vma->vm_start) >>PAGE_SHIFT ) + vma->vm_pgoff; paddr = pfn <<PAGE_SHIFT ; table =memremap (paddr,PAGE_SIZE ,MEMREMAP_WB );if (!table) { -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);return -EFAULT ; } ret =CMPXCHG (&table[index], orig_pte, new_pte);memunmap (table); -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); }return (ret != orig_pte);
android/binder_alloc.c +2 -2
@@ -932,7 +932,7 @@ enum lru_status binder_alloc_free_page(struct list_head *item, mm = alloc->vma_vm_mm;if (!mmget_not_zero (mm)) goto err_mmget; -if (!down_read_trylock (&mm->mmap_sem)) +if (!mm_read_trylock (mm)) goto err_down_read_mmap_sem_failed; vma =binder_alloc_get_vma (alloc);
@@ -946,7 +946,7 @@ enum lru_status binder_alloc_free_page(struct list_head *item,trace_binder_unmap_user_end (alloc, index); } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);mmput (mm);trace_binder_unmap_kernel_start (alloc, index);
proc/base.c +3 -3
@@ -2189,7 +2189,7 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx)if (!mm) goto out_put_task; - ret =down_read_killable (&mm->mmap_sem); + ret =mm_read_lock_killable (mm);if (ret) {mmput (mm); goto out_put_task;
@@ -2216,7 +2216,7 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx) p =genradix_ptr_alloc (&fa, nr_files++,GFP_KERNEL );if (!p) { ret = -ENOMEM ; -up_read (&mm->mmap_sem); +mm_read_unlock (mm);mmput (mm); goto out_put_task; }
@@ -2225,7 +2225,7 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx) p->end = vma->vm_end; p->mode = vma->vm_file->f_mode; } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);mmput (mm);for (i =0 ; i < nr_files; i++) {
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 2/24 MM locking API: use coccinelle to convert mmap_sem rwsem call sites
In reply to Michel LespinasseThis change converts the existing mmap_sem rwsem calls to use the new MM locking API instead.
The change is generated using coccinelle with the following rules:
// spatch --sp-file foo.cocci --in-place --include-headers --dir .
@@
expression mm;
@@
-init_rwsem(&mm->mmap_sem)
+mm_init_lock(mm)
@@
expression mm;
@@
-down_write(&mm->mmap_sem)
+mm_write_lock(mm)
@@
expression mm;
@@
-down_write_killable(&mm->mmap_sem)
+mm_write_lock_killable(mm)
@@
expression mm;
@@
-down_write_trylock(&mm->mmap_sem)
+mm_write_trylock(mm)
@@
expression mm;
@@
-up_write(&mm->mmap_sem)
+mm_write_unlock(mm)
@@
expression mm;
@@
-downgrade_write(&mm->mmap_sem)
+mm_downgrade_write_lock(mm)
@@
expression mm;
@@
-down_read(&mm->mmap_sem)
+mm_read_lock(mm)
@@
expression mm;
@@
-down_read_killable(&mm->mmap_sem)
+mm_read_lock_killable(mm)
@@
expression mm;
@@
-down_read_trylock(&mm->mmap_sem)
+mm_read_trylock(mm)
@@
expression mm;
@@
-up_read(&mm->mmap_sem)
+mm_read_unlock(mm)
@@
expression mm;
@@
-rwsem_is_locked(&mm->mmap_sem)
+mm_is_locked(mm)
--- arch/alpha/kernel/traps.c | 4 +- arch/alpha/mm/fault.c | 10 ++--- arch/arc/kernel/process.c | 4 +- arch/arc/kernel/troubleshoot.c | 4 +- arch/arc/mm/fault.c | 4 +- arch/arm/kernel/process.c | 4 +- arch/arm/kernel/swp_emulate.c | 4 +- arch/arm/lib/uaccess_with_memcpy.c | 16 ++++---- arch/arm/mm/fault.c | 6 +-- arch/arm64/kernel/traps.c | 4 +- arch/arm64/kernel/vdso.c | 8 ++-- arch/arm64/mm/fault.c | 8 ++-- arch/csky/kernel/vdso.c | 4 +- arch/csky/mm/fault.c | 8 ++-- arch/hexagon/kernel/vdso.c | 4 +- arch/hexagon/mm/vm_fault.c | 8 ++-- arch/ia64/kernel/perfmon.c | 8 ++-- arch/ia64/mm/fault.c | 8 ++-- arch/ia64/mm/init.c | 12 +++--- arch/m68k/kernel/sys_m68k.c | 14 +++---- arch/m68k/mm/fault.c | 8 ++-- arch/microblaze/mm/fault.c | 12 +++--- arch/mips/kernel/traps.c | 4 +- arch/mips/kernel/vdso.c | 4 +- arch/nds32/kernel/vdso.c | 6 +-- arch/nds32/mm/fault.c | 12 +++--- arch/nios2/mm/fault.c | 12 +++--- arch/nios2/mm/init.c | 4 +- arch/openrisc/mm/fault.c | 10 ++--- arch/parisc/kernel/traps.c | 6 +-- arch/parisc/mm/fault.c | 8 ++-- arch/powerpc/kernel/vdso.c | 6 +-- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_hv.c | 6 +-- arch/powerpc/kvm/book3s_hv_uvmem.c | 12 +++--- arch/powerpc/kvm/e500_mmu_host.c | 4 +- arch/powerpc/mm/book3s64/iommu_api.c | 4 +- arch/powerpc/mm/book3s64/subpage_prot.c | 12 +++--- arch/powerpc/mm/copro_fault.c | 4 +- arch/powerpc/mm/fault.c | 12 +++--- arch/powerpc/oprofile/cell/spu_task_sync.c | 6 +-- arch/powerpc/platforms/cell/spufs/file.c | 4 +- arch/riscv/kernel/vdso.c | 4 +- arch/riscv/mm/fault.c | 10 ++--- arch/s390/kernel/vdso.c | 4 +- arch/s390/kvm/gaccess.c | 4 +- arch/s390/kvm/kvm-s390.c | 24 +++++------ arch/s390/kvm/priv.c | 32 +++++++-------- arch/s390/mm/fault.c | 6 +-- arch/s390/mm/gmap.c | 40 +++++++++---------- arch/s390/pci/pci_mmio.c | 4 +- arch/sh/kernel/sys_sh.c | 6 +-- arch/sh/kernel/vsyscall/vsyscall.c | 4 +- arch/sh/mm/fault.c | 14 +++---- arch/sparc/mm/fault_32.c | 18 ++++----- arch/sparc/mm/fault_64.c | 12 +++--- arch/sparc/vdso/vma.c | 4 +- arch/um/include/asm/mmu_context.h | 2 +- arch/um/kernel/tlb.c | 2 +- arch/um/kernel/trap.c | 6 +-- arch/unicore32/mm/fault.c | 6 +-- arch/x86/entry/vdso/vma.c | 10 ++--- arch/x86/kernel/vm86_32.c | 4 +- arch/x86/mm/debug_pagetables.c | 8 ++-- arch/x86/mm/fault.c | 8 ++-- arch/x86/mm/mpx.c | 12 +++--- arch/x86/um/vdso/vma.c | 4 +- arch/xtensa/mm/fault.c | 10 ++--- drivers/android/binder_alloc.c | 6 +-- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 10 ++--- drivers/gpu/drm/amd/amdkfd/kfd_events.c | 4 +- drivers/gpu/drm/i915/gem/i915_gem_mman.c | 4 +- drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 8 ++-- drivers/gpu/drm/nouveau/nouveau_svm.c | 20 +++++----- drivers/gpu/drm/radeon/radeon_cs.c | 4 +- drivers/gpu/drm/radeon/radeon_gem.c | 6 +-- drivers/gpu/drm/ttm/ttm_bo_vm.c | 4 +- drivers/infiniband/core/umem.c | 6 +-- drivers/infiniband/core/umem_odp.c | 10 ++--- drivers/infiniband/core/uverbs_main.c | 4 +- drivers/infiniband/hw/mlx4/mr.c | 4 +- drivers/infiniband/hw/qib/qib_user_pages.c | 6 +-- drivers/infiniband/hw/usnic/usnic_uiom.c | 4 +- drivers/infiniband/sw/siw/siw_mem.c | 4 +- drivers/iommu/amd_iommu_v2.c | 4 +- drivers/iommu/intel-svm.c | 4 +- drivers/media/v4l2-core/videobuf-core.c | 4 +- drivers/media/v4l2-core/videobuf-dma-contig.c | 4 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 4 +- drivers/misc/cxl/cxllib.c | 4 +- drivers/misc/cxl/fault.c | 4 +- drivers/misc/sgi-gru/grufault.c | 16 ++++---- drivers/misc/sgi-gru/grufile.c | 4 +- drivers/oprofile/buffer_sync.c | 10 ++--- drivers/staging/kpc2000/kpc_dma/fileops.c | 4 +- drivers/tee/optee/call.c | 4 +- drivers/vfio/vfio_iommu_type1.c | 12 +++--- drivers/xen/gntdev.c | 4 +- drivers/xen/privcmd.c | 14 +++---- fs/aio.c | 4 +- fs/coredump.c | 4 +- fs/exec.c | 16 ++++---- fs/io_uring.c | 4 +- fs/proc/base.c | 12 +++--- fs/proc/task_mmu.c | 28 ++++++------- fs/proc/task_nommu.c | 18 ++++----- fs/userfaultfd.c | 28 ++++++------- include/linux/mmu_notifier.h | 5 ++- ipc/shm.c | 8 ++-- kernel/acct.c | 4 +- kernel/bpf/stackmap.c | 4 +- kernel/events/core.c | 4 +- kernel/events/uprobes.c | 16 ++++---- kernel/exit.c | 8 ++-- kernel/fork.c | 12 +++--- kernel/futex.c | 4 +- kernel/sched/fair.c | 4 +- kernel/sys.c | 18 ++++----- kernel/trace/trace_output.c | 4 +- mm/filemap.c | 6 +-- mm/frame_vector.c | 4 +- mm/gup.c | 20 +++++----- mm/internal.h | 2 +- mm/khugepaged.c | 36 ++++++++--------- mm/ksm.c | 34 ++++++++-------- mm/madvise.c | 18 ++++----- mm/memcontrol.c | 8 ++-- mm/memory.c | 12 +++--- mm/mempolicy.c | 22 +++++----- mm/migrate.c | 8 ++-- mm/mincore.c | 4 +- mm/mlock.c | 16 ++++---- mm/mmap.c | 32 +++++++-------- mm/mmu_notifier.c | 4 +- mm/mprotect.c | 12 +++--- mm/mremap.c | 6 +-- mm/msync.c | 8 ++-- mm/nommu.c | 16 ++++---- mm/oom_kill.c | 4 +- mm/process_vm_access.c | 4 +- mm/swapfile.c | 4 +- mm/userfaultfd.c | 14 +++---- mm/util.c | 8 ++-- net/ipv4/tcp.c | 4 +- net/xdp/xdp_umem.c | 4 +- virt/kvm/arm/mmu.c | 14 +++---- virt/kvm/async_pf.c | 4 +- virt/kvm/kvm_main.c | 8 ++-- 149 files changed, 653 insertions(+), 652 deletions(-)
alpha/kernel/traps.c +2 -2
@@ -957,12 +957,12 @@ do_entUnaUser(void __user * va, unsigned long opcode, si_code =SEGV_ACCERR ; else {struct mm_struct *mm = current->mm; -down_read (&mm->mmap_sem); +mm_read_lock (mm);if (find_vma (mm, (unsigned long )va)) si_code =SEGV_ACCERR ;else si_code =SEGV_MAPERR ; -up_read (&mm->mmap_sem); +mm_read_unlock (mm); }send_sig_fault (SIGSEGV , si_code, va,0 , current);return ;
alpha/mm/fault.c +5 -5
@@ -117,7 +117,7 @@ do_page_fault(unsigned long address, unsigned long mmcsr,if (user_mode (regs)) flags |=FAULT_FLAG_USER ; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -180,14 +180,14 @@ do_page_fault(unsigned long address, unsigned long mmcsr, } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ;/* Something tried to access memory that isn't in our memory map. Fix it, but check if it's kernel or user first. */ bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (user_mode (regs)) goto do_sigsegv;
@@ -211,14 +211,14 @@ do_page_fault(unsigned long address, unsigned long mmcsr,/* We ran out of memory, or some other thing happened to us that made us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!user_mode (regs)) goto no_context;pagefault_out_of_memory ();return ; do_sigbus: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* Send a sigbus, regardless of whether we were in kernel or user mode. */ force_sig_fault (SIGBUS ,BUS_ADRERR , (void __user *) address,0 );
arc/kernel/process.c +2 -2
@@ -88,10 +88,10 @@ SYSCALL_DEFINE3(arc_usr_cmpxchg, int *, uaddr, int, expected, int, new)if (unlikely (ret != -EFAULT )) goto fail; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); ret =fixup_user_fault (current, current->mm, (unsigned long ) uaddr,FAULT_FLAG_WRITE ,NULL ); -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (likely (!ret)) goto again;
arc/kernel/troubleshoot.c +2 -2
@@ -89,7 +89,7 @@ static void show_faulting_vma(unsigned long address)/* can't use print_vma_addr() yet as it doesn't check for * non-inclusive vma */ -down_read (&active_mm->mmap_sem); +mm_read_lock (active_mm); vma =find_vma (active_mm, address);/* check against the find_vma( ) behaviour which returns the next VMA
@@ -112,7 +112,7 @@ static void show_faulting_vma(unsigned long address) }else pr_info (" @No matching VMA found\n" ); -up_read (&active_mm->mmap_sem); +mm_read_unlock (active_mm); }static void show_ecr_verbose (struct pt_regs *regs)
arc/mm/fault.c +2 -2
@@ -107,7 +107,7 @@ void do_page_fault(unsigned long address, struct pt_regs *regs) flags |=FAULT_FLAG_WRITE ; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, address);if (!vma)
@@ -159,7 +159,7 @@ void do_page_fault(unsigned long address, struct pt_regs *regs) } bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* * Major/minor page fault accounting
arm/kernel/process.c +2 -2
@@ -431,7 +431,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) npages =1 ;/* for sigpage */ npages += vdso_total_pages; -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ; hint =sigpage_addr (mm, npages); addr =get_unmapped_area (NULL , hint, npages <<PAGE_SHIFT ,0 ,0 );
@@ -458,7 +458,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)arm_install_vdso (mm, addr +PAGE_SIZE ); up_fail: -up_write (&mm->mmap_sem); +mm_write_unlock (mm); return ret; }#endif
arm/kernel/swp_emulate.c +2 -2
@@ -97,12 +97,12 @@ static void set_segfault(struct pt_regs *regs, unsigned long addr) {int si_code; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm);if (find_vma (current->mm, addr) ==NULL ) si_code =SEGV_MAPERR ;else si_code =SEGV_ACCERR ; -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);pr_debug ("SWP{B} emulation: access caused memory abort!\n" ); arm_notify_die("Illegal memory access", regs,
arm/lib/uaccess_with_memcpy.c +8 -8
@@ -96,7 +96,7 @@ __copy_to_user_memcpy(void __user *to, const void *from, unsigned long n) atomic =faulthandler_disabled ();if (!atomic) -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm);while (n) {pte_t *pte;spinlock_t *ptl;
@@ -104,11 +104,11 @@ __copy_to_user_memcpy(void __user *to, const void *from, unsigned long n)while (!pin_page_for_write (to, &pte, &ptl)) {if (!atomic) -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (__put_user (0 , (char __user *)to)) goto out;if (!atomic) -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); } tocopy = (~(unsigned long )to & ~PAGE_MASK ) +1 ;
@@ -128,7 +128,7 @@ __copy_to_user_memcpy(void __user *to, const void *from, unsigned long n)spin_unlock (ptl); }if (!atomic) -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); out:return n;
@@ -165,17 +165,17 @@ __clear_user_memset(void __user *addr, unsigned long n)return 0 ; } -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm);while (n) {pte_t *pte;spinlock_t *ptl;int tocopy;while (!pin_page_for_write (addr, &pte, &ptl)) { -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (__put_user (0 , (char __user *)addr)) goto out; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); } tocopy = (~(unsigned long )addr & ~PAGE_MASK ) +1 ;
@@ -193,7 +193,7 @@ __clear_user_memset(void __user *addr, unsigned long n)else spin_unlock (ptl); } -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); out:return n;
arm/mm/fault.c +3 -3
@@ -270,11 +270,11 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)* validly references user space from well defined areas of the code, * we can bug out early if this is from code which shouldn't. */ -if (!down_read_trylock (&mm->mmap_sem)) { +if (!mm_read_trylock (mm)) {if (!user_mode (regs) && !search_exception_tables (regs->ARM_pc)) goto no_context; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); }else {/* * The above down_read_trylock() might have succeeded in
@@ -327,7 +327,7 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* * Handle the "normal" case first - VM_FAULT_MAJOR
arm64/kernel/traps.c +2 -2
@@ -384,12 +384,12 @@ void arm64_notify_segfault(unsigned long addr) {int code; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm);if (find_vma (current->mm, addr) ==NULL ) code =SEGV_MAPERR ;else code =SEGV_ACCERR ; -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);force_signal_inject (SIGSEGV , code, addr); }
arm64/kernel/vdso.c +4 -4
@@ -357,7 +357,7 @@ int aarch32_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)struct mm_struct *mm = current->mm;int ret; -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ; ret =aarch32_kuser_helpers_setup (mm);
@@ -374,7 +374,7 @@ int aarch32_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)#endif /* CONFIG_COMPAT_VDSO */ out: -up_write (&mm->mmap_sem); +mm_write_unlock (mm); return ret; }#endif /* CONFIG_COMPAT */
@@ -418,7 +418,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm,struct mm_struct *mm = current->mm;int ret; -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ; ret = __setup_additional_pages(ARM64_VDSO,
@@ -426,7 +426,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, bprm, uses_interp); -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return ret; }
arm64/mm/fault.c +4 -4
@@ -491,11 +491,11 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr,* validly references user space from well defined areas of the code, * we can bug out early if this is from code which shouldn't. */ -if (!down_read_trylock (&mm->mmap_sem)) { +if (!mm_read_trylock (mm)) {if (!user_mode (regs) && !search_exception_tables (regs->pc)) goto no_context; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); }else {/* * The above down_read_trylock() might have succeeded in which
@@ -504,7 +504,7 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr,might_sleep ();#ifdef CONFIG_DEBUG_VM if (!user_mode (regs) && !search_exception_tables (regs->pc)) { -up_read (&mm->mmap_sem); +mm_read_unlock (mm); goto no_context; }#endif
@@ -536,7 +536,7 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, goto retry; } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* * Handle the "normal" (no error) case first.
csky/kernel/vdso.c +2 -2
@@ -50,7 +50,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)unsigned long addr;struct mm_struct *mm = current->mm; -down_write (&mm->mmap_sem); +mm_write_lock (mm); addr =get_unmapped_area (NULL ,STACK_TOP ,PAGE_SIZE ,0 ,0 );if (IS_ERR_VALUE (addr)) {
@@ -70,7 +70,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) mm->context.vdso = (void *)addr; up_fail: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return ret; }
csky/mm/fault.c +4 -4
@@ -116,7 +116,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long write,if (in_atomic () || !mm) goto bad_area_nosemaphore; -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -166,7 +166,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long write, address); } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ;/*
@@ -174,7 +174,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long write,* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm); bad_area_nosemaphore:/* User mode accesses just cause a SIGSEGV */
@@ -206,7 +206,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long write,return ; do_sigbus: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* Kernel mode? Handle exceptions or die */ if (!user_mode (regs))
hexagon/kernel/vdso.c +2 -2
@@ -52,7 +52,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)unsigned long vdso_base;struct mm_struct *mm = current->mm; -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ;/* Try to get it loaded right near ld.so/glibc. */
@@ -76,7 +76,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) mm->context.vdso = (void *)vdso_base; up_fail: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return ret; }
hexagon/mm/vm_fault.c +4 -4
@@ -55,7 +55,7 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)if (user_mode (regs)) flags |=FAULT_FLAG_USER ; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -108,11 +108,11 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs) } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ; } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* Handle copyin/out exception cases */ if (!user_mode (regs))
@@ -139,7 +139,7 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)return ; bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (user_mode (regs)) {force_sig_fault (SIGSEGV , si_code, (void __user *)address);
ia64/kernel/perfmon.c +4 -4
@@ -2258,13 +2258,13 @@ pfm_smpl_buffer_alloc(struct task_struct *task, struct file *filp, pfm_context_t* now we atomically find some area in the address space and * remap the buffer in it. */ -down_write (&task->mm->mmap_sem); +mm_write_lock (task->mm);/* find some free area in address space, must have mmap sem held */ vma->vm_start =get_unmapped_area (NULL ,0 , size,0 ,MAP_PRIVATE |MAP_ANONYMOUS );if (IS_ERR_VALUE (vma->vm_start)) {DPRINT (("Cannot find unmapped area for size %ld\n" , size)); -up_write (&task->mm->mmap_sem); +mm_write_unlock (task->mm); goto error; } vma->vm_end = vma->vm_start + size;
@@ -2275,7 +2275,7 @@ pfm_smpl_buffer_alloc(struct task_struct *task, struct file *filp, pfm_context_t/* can only be applied to current task, need to have the mm semaphore held when called */ if (pfm_remap_buffer (vma, (unsigned long )smpl_buf, vma->vm_start, size)) {DPRINT (("Can't remap buffer\n" )); -up_write (&task->mm->mmap_sem); +mm_write_unlock (task->mm); goto error; }
@@ -2286,7 +2286,7 @@ pfm_smpl_buffer_alloc(struct task_struct *task, struct file *filp, pfm_context_tinsert_vm_struct (mm, vma);vm_stat_account (vma->vm_mm, vma->vm_flags,vma_pages (vma)); -up_write (&task->mm->mmap_sem); +mm_write_unlock (task->mm);/* * keep track of user level virtual address
ia64/mm/fault.c +4 -4
@@ -102,7 +102,7 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *reif (mask &VM_WRITE ) flags |=FAULT_FLAG_WRITE ; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma_prev (mm, address, &prev_vma);if (!vma && !prev_vma )
@@ -179,7 +179,7 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ; check_expansion:
@@ -210,7 +210,7 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re goto good_area; bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);#ifdef CONFIG_VIRTUAL_MEM_MAP bad_area_no_up:#endif
@@ -276,7 +276,7 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *rereturn ; out_of_memory: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!user_mode (regs)) goto no_context;pagefault_out_of_memory ();
ia64/mm/init.c +6 -6
@@ -118,13 +118,13 @@ ia64_init_addr_space (void) vma->vm_end = vma->vm_start +PAGE_SIZE ; vma->vm_flags =VM_DATA_DEFAULT_FLAGS |VM_GROWSUP |VM_ACCOUNT ; vma->vm_page_prot =vm_get_page_prot (vma->vm_flags); -down_write (¤t->mm->mmap_sem); +mm_write_lock (current->mm);if (insert_vm_struct (current->mm, vma)) { -up_write (¤t->mm->mmap_sem); +mm_write_unlock (current->mm);vm_area_free (vma);return ; } -up_write (¤t->mm->mmap_sem); +mm_write_unlock (current->mm); }/* map NaT-page at address zero to speed up speculative dereferencing of NULL: */
@@ -136,13 +136,13 @@ ia64_init_addr_space (void) vma->vm_page_prot =__pgprot (pgprot_val (PAGE_READONLY ) | _PAGE_MA_NAT); vma->vm_flags =VM_READ |VM_MAYREAD |VM_IO |VM_DONTEXPAND |VM_DONTDUMP ; -down_write (¤t->mm->mmap_sem); +mm_write_lock (current->mm);if (insert_vm_struct (current->mm, vma)) { -up_write (¤t->mm->mmap_sem); +mm_write_unlock (current->mm);vm_area_free (vma);return ; } -up_write (¤t->mm->mmap_sem); +mm_write_unlock (current->mm); } } }
m68k/kernel/sys_m68k.c +7 -7
@@ -399,7 +399,7 @@ sys_cacheflush (unsigned long addr, int scope, int cache, unsigned long len)* Verify that the specified address region actually belongs * to this process. */ -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); vma =find_vma (current->mm, addr);if (!vma || addr < vma->vm_start || addr + len > vma->vm_end) goto out_unlock;
@@ -450,7 +450,7 @@ sys_cacheflush (unsigned long addr, int scope, int cache, unsigned long len) } } out_unlock: -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); out:return ret; }
@@ -472,7 +472,7 @@ sys_atomic_cmpxchg_32(unsigned long newval, int oldval, int d3, int d4, int d5,spinlock_t *ptl;unsigned long mem_value; -down_read (&mm->mmap_sem); +mm_read_lock (mm); pgd =pgd_offset (mm, (unsigned long )mem);if (!pgd_present (*pgd)) goto bad_access;
@@ -501,11 +501,11 @@ sys_atomic_cmpxchg_32(unsigned long newval, int oldval, int d3, int d4, int d5,__put_user (newval, mem);pte_unmap_unlock (pte, ptl); -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return mem_value; bad_access: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* This is not necessarily a bad access, we can get here if a memory we're trying to write to should be copied-on-write. Make the kernel do the necessary page stuff, then re-iterate.
@@ -545,13 +545,13 @@ sys_atomic_cmpxchg_32(unsigned long newval, int oldval, int d3, int d4, int d5,struct mm_struct *mm = current->mm;unsigned long mem_value; -down_read (&mm->mmap_sem); +mm_read_lock (mm); mem_value = *mem;if (mem_value == oldval) *mem = newval; -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return mem_value; }
m68k/mm/fault.c +4 -4
@@ -86,7 +86,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,if (user_mode (regs)) flags |=FAULT_FLAG_USER ; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, address);if (!vma)
@@ -177,7 +177,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address, } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return 0 ;/*
@@ -185,7 +185,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!user_mode (regs)) goto no_context;pagefault_out_of_memory ();
@@ -214,6 +214,6 @@ int do_page_fault(struct pt_regs *regs, unsigned long address, current->thread.faddr = address; send_sig: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return send_fault_sig (regs); }
microblaze/mm/fault.c +6 -6
@@ -137,12 +137,12 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,* source. If this is invalid we can skip the address space check, * thus avoiding the deadlock. */ -if (unlikely (!down_read_trylock (&mm->mmap_sem))) { +if (unlikely (!mm_read_trylock (mm))) {if (kernel_mode (regs) && !search_exception_tables (regs->pc)) goto bad_area_nosemaphore; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); } vma =find_vma (mm, address);
@@ -249,7 +249,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long address, } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* * keep track of tlb+htab misses that are good addrs but
@@ -260,7 +260,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,return ; bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm); bad_area_nosemaphore: pte_errors++;
@@ -279,7 +279,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!user_mode (regs))bad_page_fault (regs, address,SIGKILL );else
@@ -287,7 +287,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,return ; do_sigbus: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (user_mode (regs)) {force_sig_fault (SIGBUS ,BUS_ADRERR , (void __user *)address);return ;
mips/kernel/traps.c +2 -2
@@ -754,13 +754,13 @@ int process_fpemu_return(int sig, void __user *fault_addr, unsigned long fcr31)return 1 ;case SIGSEGV : -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); vma =find_vma (current->mm, (unsigned long )fault_addr);if (vma && (vma->vm_start <= (unsigned long )fault_addr)) si_code =SEGV_ACCERR ;else si_code =SEGV_MAPERR ; -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);force_sig_fault (SIGSEGV , si_code, fault_addr);return 1 ;
mips/kernel/vdso.c +2 -2
@@ -92,7 +92,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)struct vm_area_struct *vma;int ret; -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ;/* Map delay slot emulation page */
@@ -183,6 +183,6 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) ret =0 ; out: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return ret; }
nds32/kernel/vdso.c +3 -3
@@ -130,7 +130,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) vdso_mapping_len += L1_cache_info[DCACHE ].aliasing_num -1 ;#endif -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ; addr =vdso_random_addr (vdso_mapping_len);
@@ -185,12 +185,12 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) goto up_fail; } -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return 0 ; up_fail: mm->context.vdso =NULL ; -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return ret; }
nds32/mm/fault.c +6 -6
@@ -127,12 +127,12 @@ void do_page_fault(unsigned long entry, unsigned long addr,* validly references user space from well defined areas of the code, * we can bug out early if this is from code which shouldn't. */ -if (unlikely (!down_read_trylock (&mm->mmap_sem))) { +if (unlikely (!mm_read_trylock (mm))) {if (!user_mode (regs) && !search_exception_tables (instruction_pointer (regs))) goto no_context; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); }else {/* * The above down_read_trylock() might have succeeded in which
@@ -257,7 +257,7 @@ void do_page_fault(unsigned long entry, unsigned long addr, } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ;/*
@@ -265,7 +265,7 @@ void do_page_fault(unsigned long entry, unsigned long addr,* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm); bad_area_nosemaphore:
@@ -325,14 +325,14 @@ void do_page_fault(unsigned long entry, unsigned long addr,*/ out_of_memory: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!user_mode (regs)) goto no_context;pagefault_out_of_memory ();return ; do_sigbus: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* Kernel mode? Handle exceptions or die */ if (!user_mode (regs))
nios2/mm/fault.c +6 -6
@@ -83,11 +83,11 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,if (user_mode (regs)) flags |=FAULT_FLAG_USER ; -if (!down_read_trylock (&mm->mmap_sem)) { +if (!mm_read_trylock (mm)) {if (!user_mode (regs) && !search_exception_tables (regs->ea)) goto bad_area_nosemaphore; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); } vma =find_vma (mm, address);
@@ -172,7 +172,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause, } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ;/*
@@ -180,7 +180,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm); bad_area_nosemaphore:/* User mode accesses just cause a SIGSEGV */
@@ -218,14 +218,14 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!user_mode (regs)) goto no_context;pagefault_out_of_memory ();return ; do_sigbus: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* Kernel mode? Handle exceptions or die */ if (!user_mode (regs))
nios2/mm/init.c +2 -2
@@ -112,14 +112,14 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)struct mm_struct *mm = current->mm;int ret; -down_write (&mm->mmap_sem); +mm_write_lock (mm);/* Map kuser helpers to user space address */ ret =install_special_mapping (mm,KUSER_BASE ,KUSER_SIZE ,VM_READ |VM_EXEC |VM_MAYREAD |VM_MAYEXEC , kuser_page); -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return ret; }
openrisc/mm/fault.c +5 -5
@@ -104,7 +104,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address, goto no_context; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, address);if (!vma)
@@ -193,7 +193,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address, } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ;/*
@@ -202,7 +202,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,*/ bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm); bad_area_nosemaphore:
@@ -261,14 +261,14 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address, __asm__ __volatile__("l.nop 42" ); __asm__ __volatile__("l.nop 1" ); -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!user_mode (regs)) goto no_context;pagefault_out_of_memory ();return ; do_sigbus: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* * Send a sigbus, regardless of whether we were in kernel
parisc/kernel/traps.c +3 -3
@@ -717,7 +717,7 @@ void notrace handle_interruption(int code, struct pt_regs *regs)if (user_mode (regs)) {struct vm_area_struct *vma; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); vma =find_vma (current->mm,regs->iaoq[0 ]);if (vma && (regs->iaoq[0 ] >= vma->vm_start) && (vma->vm_flags &VM_EXEC )) {
@@ -725,10 +725,10 @@ void notrace handle_interruption(int code, struct pt_regs *regs) fault_address = regs->iaoq[0 ]; fault_space = regs->iasq[0 ]; -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);break ;/* call do_page_fault() */ } -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); }/* Fall Through */ case 27 :
parisc/mm/fault.c +4 -4
@@ -282,7 +282,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,if (acc_type &VM_WRITE ) flags |=FAULT_FLAG_WRITE ; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma_prev (mm, address, &prev_vma);if (!vma || address < vma->vm_start) goto check_expansion;
@@ -339,7 +339,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long code, goto retry; } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ; check_expansion:
@@ -351,7 +351,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,* Something tried to access memory that isn't in our memory map.. */ bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (user_mode (regs)) {int signo, si_code;
@@ -423,7 +423,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,parisc_terminate ("Bad Address (null pointer deref?)" , regs, code, address); out_of_memory: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!user_mode (regs)) goto no_context;pagefault_out_of_memory ();
powerpc/kernel/vdso.c +3 -3
@@ -171,7 +171,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)* and end up putting it elsewhere. * Add enough to the size so that the result can be aligned. */ -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ; vdso_base = get_unmapped_area(NULL, vdso_base, (vdso_pages << PAGE_SHIFT) +
@@ -211,11 +211,11 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) goto fail_mmapsem; } -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return 0 ; fail_mmapsem: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return rc; }
powerpc/kvm/book3s_64_mmu_hv.c +2 -2
@@ -582,7 +582,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, npages =get_user_pages_fast (hva,1 , writing ?FOLL_WRITE :0 , pages);if (npages <1 ) {/* Check if it's an I/O mapping */ -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, hva);if (vma && vma->vm_start <= hva && hva + psize <= vma->vm_end && (vma->vm_flags &VM_PFNMAP )) {
@@ -592,7 +592,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, is_ci =pte_ci (__pte ((pgprot_val (vma->vm_page_prot)))); write_ok = vma->vm_flags &VM_WRITE ; } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!pfn) goto out_put; }else {
powerpc/kvm/book3s_hv.c +3 -3
@@ -4640,14 +4640,14 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu)/* Look up the VMA for the start of this memory slot */ hva = memslot->userspace_addr; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); vma =find_vma (current->mm, hva);if (!vma || vma->vm_start > hva || (vma->vm_flags &VM_IO )) goto up_out; psize =vma_kernel_pagesize (vma); -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);/* We can handle 4k, 64k or 16M pages in the VRMA */ if (psize >=0x1000000 )
@@ -4680,7 +4680,7 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu)return err; up_out: -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); goto out_srcu; }
powerpc/kvm/book3s_hv_uvmem.c +6 -6
@@ -366,7 +366,7 @@ kvmppc_svm_page_in(struct vm_area_struct *vma, unsigned long start,*/ ret =ksm_madvise (vma, vma->vm_start, vma->vm_end,MADV_UNMERGEABLE , &vma->vm_flags); -downgrade_write (&kvm->mm->mmap_sem); +mm_downgrade_write_lock (kvm->mm); *downgrade = true;if (ret)return ret;
@@ -483,7 +483,7 @@ kvmppc_h_svm_page_in(struct kvm *kvm, unsigned long gpa, ret =H_PARAMETER ; srcu_idx =srcu_read_lock (&kvm->srcu); -down_write (&kvm->mm->mmap_sem); +mm_write_lock (kvm->mm); start =gfn_to_hva (kvm, gfn);if (kvm_is_error_hva (start))
@@ -506,9 +506,9 @@ kvmppc_h_svm_page_in(struct kvm *kvm, unsigned long gpa,mutex_unlock (&kvm->arch.uvmem_lock); out:if (downgrade) -up_read (&kvm->mm->mmap_sem); +mm_read_unlock (kvm->mm);else -up_write (&kvm->mm->mmap_sem); +mm_write_unlock (kvm->mm);srcu_read_unlock (&kvm->srcu, srcu_idx);return ret; }
@@ -660,7 +660,7 @@ kvmppc_h_svm_page_out(struct kvm *kvm, unsigned long gpa, ret =H_PARAMETER ; srcu_idx =srcu_read_lock (&kvm->srcu); -down_read (&kvm->mm->mmap_sem); +mm_read_lock (kvm->mm); start =gfn_to_hva (kvm, gfn);if (kvm_is_error_hva (start)) goto out;
@@ -673,7 +673,7 @@ kvmppc_h_svm_page_out(struct kvm *kvm, unsigned long gpa,if (!kvmppc_svm_page_out (vma, start, end, page_shift, kvm, gpa)) ret =H_SUCCESS ; out: -up_read (&kvm->mm->mmap_sem); +mm_read_unlock (kvm->mm);srcu_read_unlock (&kvm->srcu, srcu_idx);return ret; }
powerpc/kvm/e500_mmu_host.c +2 -2
@@ -355,7 +355,7 @@ static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500,if (tlbsel ==1 ) {struct vm_area_struct *vma; -down_read (&kvm->mm->mmap_sem); +mm_read_lock (kvm->mm); vma =find_vma (kvm->mm, hva); if (vma && hva >= vma->vm_start &&
@@ -441,7 +441,7 @@ static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500, tsize =max (BOOK3E_PAGESZ_4K , tsize & ~1 ); } -up_read (&kvm->mm->mmap_sem); +mm_read_unlock (kvm->mm); }if (likely (!pfnmap)) {
powerpc/mm/book3s64/iommu_api.c +2 -2
@@ -96,7 +96,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua, goto unlock_exit; } -down_read (&mm->mmap_sem); +mm_read_lock (mm); chunk = (1UL << (PAGE_SHIFT +MAX_ORDER -1 )) /sizeof (struct vm_area_struct *); chunk =min (chunk, entries);
@@ -114,7 +114,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua, pinned += ret;break ; } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (pinned != entries) {if (!ret) ret = -EFAULT ;
powerpc/mm/book3s64/subpage_prot.c +6 -6
@@ -92,7 +92,7 @@ static void subpage_prot_clear(unsigned long addr, unsigned long len)size_t nw;unsigned long next, limit; -down_write (&mm->mmap_sem); +mm_write_lock (mm); spt =mm_ctx_subpage_prot (&mm->context);if (!spt)
@@ -127,7 +127,7 @@ static void subpage_prot_clear(unsigned long addr, unsigned long len) } err_out: -up_write (&mm->mmap_sem); +mm_write_unlock (mm); }#ifdef CONFIG_TRANSPARENT_HUGEPAGE
@@ -217,7 +217,7 @@ SYSCALL_DEFINE3(subpage_prot, unsigned long, addr,if (!access_ok (map, (len >>PAGE_SHIFT ) *sizeof (u32)))return -EFAULT ; -down_write (&mm->mmap_sem); +mm_write_lock (mm); spt =mm_ctx_subpage_prot (&mm->context);if (!spt) {
@@ -267,11 +267,11 @@ SYSCALL_DEFINE3(subpage_prot, unsigned long, addr,if (addr + (nw <<PAGE_SHIFT ) > next) nw = (next - addr) >>PAGE_SHIFT ; -up_write (&mm->mmap_sem); +mm_write_unlock (mm);if (__copy_from_user (spp, map, nw *sizeof (u32)))return -EFAULT ; map += nw; -down_write (&mm->mmap_sem); +mm_write_lock (mm);/* now flush any existing HPTEs for the range */ hpte_flush_range (mm, addr, nw);
@@ -280,6 +280,6 @@ SYSCALL_DEFINE3(subpage_prot, unsigned long, addr, spt->maxaddr = limit; err =0 ; out: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return err; }
powerpc/mm/copro_fault.c +2 -2
@@ -33,7 +33,7 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,if (mm->pgd ==NULL )return -EFAULT ; -down_read (&mm->mmap_sem); +mm_read_lock (mm); ret = -EFAULT ; vma =find_vma (mm, ea);if (!vma)
@@ -82,7 +82,7 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea, current->min_flt++; out_unlock: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ret; }EXPORT_SYMBOL_GPL (copro_handle_mm_fault);
powerpc/mm/fault.c +6 -6
@@ -108,7 +108,7 @@ static int __bad_area(struct pt_regs *regs, unsigned long address, int si_code)* Something tried to access memory that isn't in our memory map.. * Fix it, but check if it's kernel or user first.. */ -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return __bad_area_nosemaphore (regs, address, si_code); }
@@ -515,12 +515,12 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address,* source. If this is invalid we can skip the address space check, * thus avoiding the deadlock. */ -if (unlikely (!down_read_trylock (&mm->mmap_sem))) { +if (unlikely (!mm_read_trylock (mm))) {if (!is_user && !search_exception_tables (regs->nip))return bad_area_nosemaphore (regs, address); retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); }else {/* * The above down_read_trylock() might have succeeded in
@@ -544,7 +544,7 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address,if (!must_retry)return bad_area (regs, address); -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (fault_in_pages_readable ((const char __user *)regs->nip,sizeof (unsigned int )))return bad_area_nosemaphore (regs, address);
@@ -576,7 +576,7 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address,int pkey =vma_pkey (vma); -up_read (&mm->mmap_sem); +mm_read_unlock (mm); return bad_key_fault_exception(regs, address, pkey); }#endif /* CONFIG_PPC_MEM_KEYS */
@@ -607,7 +607,7 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address,return is_user ?0 :SIGBUS ; } -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (unlikely (fault &VM_FAULT_ERROR ))return mm_fault_error (regs, address, fault);
powerpc/oprofile/cell/spu_task_sync.c +3 -3
@@ -332,7 +332,7 @@ get_exec_dcookie_and_offset(struct spu *spu, unsigned int *offsetp,fput (exe_file); } -down_read (&mm->mmap_sem); +mm_read_lock (mm);for (vma = mm->mmap; vma; vma = vma->vm_next) {if (vma->vm_start > spu_ref || vma->vm_end <= spu_ref)continue ;
@@ -349,13 +349,13 @@ get_exec_dcookie_and_offset(struct spu *spu, unsigned int *offsetp, *spu_bin_dcookie =fast_get_dcookie (&vma->vm_file->f_path);pr_debug ("got dcookie for %pD\n" , vma->vm_file); -up_read (&mm->mmap_sem); +mm_read_unlock (mm); out:return app_cookie; fail_no_image_cookie: -up_read (&mm->mmap_sem); +mm_read_unlock (mm); printk(KERN_ERR "SPU_PROF: " "%s, line %d: Cannot find dcookie for SPU binary\n",
powerpc/platforms/cell/spufs/file.c +2 -2
@@ -336,11 +336,11 @@ static vm_fault_t spufs_ps_fault(struct vm_fault *vmf, goto refault;if (ctx->state ==SPU_STATE_SAVED ) { -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);spu_context_nospu_trace (spufs_ps_fault__sleep, ctx); err =spufs_wait (ctx->run_wq, ctx->state ==SPU_STATE_RUNNABLE );spu_context_trace (spufs_ps_fault__wake, ctx, ctx->spu); -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); }else { area = ctx->spu->problem_phys + ps_offs; ret = vmf_insert_pfn(vmf->vma, vmf->address,
riscv/kernel/vdso.c +2 -2
@@ -61,7 +61,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, vdso_len = (vdso_pages +1 ) <<PAGE_SHIFT ; -down_write (&mm->mmap_sem); +mm_write_lock (mm); vdso_base =get_unmapped_area (NULL ,0 , vdso_len,0 ,0 );if (IS_ERR_VALUE (vdso_base)) { ret = vdso_base;
@@ -83,7 +83,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, mm->context.vdso =NULL ; end: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return ret; }
riscv/mm/fault.c +5 -5
@@ -69,7 +69,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, addr); retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, addr);if (unlikely (!vma)) goto bad_area;
@@ -160,7 +160,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs) } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ;/*
@@ -168,7 +168,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)* Fix it, but check if it's kernel or user first. */ bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* User mode accesses just cause a SIGSEGV */ if (user_mode (regs)) {do_trap (regs,SIGSEGV , code, addr);
@@ -196,14 +196,14 @@ asmlinkage void do_page_fault(struct pt_regs *regs)* (which will retry the fault, or kill us if we got oom-killed). */ out_of_memory: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!user_mode (regs)) goto no_context;pagefault_out_of_memory ();return ; do_sigbus: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* Kernel mode? Handle exceptions or die */ if (!user_mode (regs)) goto no_context;
s390/kernel/vdso.c +2 -2
@@ -208,7 +208,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)* it at vdso_base which is the "natural" base for it, but we might * fail and end up putting it elsewhere. */ -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ; vdso_base =get_unmapped_area (NULL ,0 , vdso_pages <<PAGE_SHIFT ,0 ,0 );if (IS_ERR_VALUE (vdso_base)) {
@@ -239,7 +239,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) rc =0 ; out_up: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return rc; }
s390/kvm/gaccess.c +2 -2
@@ -1170,7 +1170,7 @@ int kvm_s390_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg,int dat_protection, fake;int rc; -down_read (&sg->mm->mmap_sem); +mm_read_lock (sg->mm);/* * We don't want any guest-2 tables to change - so the parent * tables/pointers we read stay valid - unshadowing is however
@@ -1199,6 +1199,6 @@ int kvm_s390_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg,if (!rc) rc =gmap_shadow_page (sg, saddr,__pte (pte.val));ipte_unlock (vcpu); -up_read (&sg->mm->mmap_sem); +mm_read_unlock (sg->mm);return rc; }
s390/kvm/kvm-s390.c +12 -12
@@ -753,9 +753,9 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap) r = -EINVAL ; else { r =0 ; -down_write (&kvm->mm->mmap_sem); +mm_write_lock (kvm->mm); kvm->mm->context.allow_gmap_hpage_1m =1 ; -up_write (&kvm->mm->mmap_sem); +mm_write_unlock (kvm->mm);/* * We might have to create fake 4k page * tables. To avoid that the hardware works on
@@ -1805,7 +1805,7 @@ static long kvm_s390_get_skeys(struct kvm *kvm, struct kvm_s390_skeys *args)if (!keys)return -ENOMEM ; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); srcu_idx =srcu_read_lock (&kvm->srcu);for (i =0 ; i < args->count; i++) { hva =gfn_to_hva (kvm, args->start_gfn + i);
@@ -1819,7 +1819,7 @@ static long kvm_s390_get_skeys(struct kvm *kvm, struct kvm_s390_skeys *args)break ; }srcu_read_unlock (&kvm->srcu, srcu_idx); -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (!r) { r = copy_to_user((uint8_t __user *)args->skeydata_addr, keys,
@@ -1863,7 +1863,7 @@ static long kvm_s390_set_skeys(struct kvm *kvm, struct kvm_s390_skeys *args) goto out; i =0 ; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); srcu_idx =srcu_read_lock (&kvm->srcu);while (i < args->count) { unlocked = false;
@@ -1890,7 +1890,7 @@ static long kvm_s390_set_skeys(struct kvm *kvm, struct kvm_s390_skeys *args) i++; }srcu_read_unlock (&kvm->srcu, srcu_idx); -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); out:kvfree (keys);return r;
@@ -2073,14 +2073,14 @@ static int kvm_s390_get_cmma_bits(struct kvm *kvm,if (!values)return -ENOMEM ; -down_read (&kvm->mm->mmap_sem); +mm_read_lock (kvm->mm); srcu_idx =srcu_read_lock (&kvm->srcu);if (peek) ret =kvm_s390_peek_cmma (kvm, args, values, bufsize);else ret =kvm_s390_get_cmma (kvm, args, values, bufsize);srcu_read_unlock (&kvm->srcu, srcu_idx); -up_read (&kvm->mm->mmap_sem); +mm_read_unlock (kvm->mm);if (kvm->arch.migration_mode) args->remaining =atomic64_read (&kvm->arch.cmma_dirty_pages);
@@ -2130,7 +2130,7 @@ static int kvm_s390_set_cmma_bits(struct kvm *kvm, goto out; } -down_read (&kvm->mm->mmap_sem); +mm_read_lock (kvm->mm); srcu_idx =srcu_read_lock (&kvm->srcu);for (i =0 ; i < args->count; i++) { hva =gfn_to_hva (kvm, args->start_gfn + i);
@@ -2145,12 +2145,12 @@ static int kvm_s390_set_cmma_bits(struct kvm *kvm,set_pgste_bits (kvm->mm, hva, mask, pgstev); }srcu_read_unlock (&kvm->srcu, srcu_idx); -up_read (&kvm->mm->mmap_sem); +mm_read_unlock (kvm->mm);if (!kvm->mm->context.uses_cmm) { -down_write (&kvm->mm->mmap_sem); +mm_write_lock (kvm->mm); kvm->mm->context.uses_cmm =1 ; -up_write (&kvm->mm->mmap_sem); +mm_write_unlock (kvm->mm); } out:vfree (bits);
s390/kvm/priv.c +16 -16
@@ -270,18 +270,18 @@ static int handle_iske(struct kvm_vcpu *vcpu)return kvm_s390_inject_program_int (vcpu,PGM_ADDRESSING ); retry: unlocked = false; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); rc =get_guest_storage_key (current->mm, vmaddr, &key);if (rc) { rc =fixup_user_fault (current, current->mm, vmaddr,FAULT_FLAG_WRITE , &unlocked);if (!rc) { -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); goto retry; } } -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (rc == -EFAULT )return kvm_s390_inject_program_int (vcpu,PGM_ADDRESSING );if (rc <0 )
@@ -317,17 +317,17 @@ static int handle_rrbe(struct kvm_vcpu *vcpu)return kvm_s390_inject_program_int (vcpu,PGM_ADDRESSING ); retry: unlocked = false; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); rc =reset_guest_reference_bit (current->mm, vmaddr);if (rc <0 ) { rc =fixup_user_fault (current, current->mm, vmaddr,FAULT_FLAG_WRITE , &unlocked);if (!rc) { -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); goto retry; } } -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (rc == -EFAULT )return kvm_s390_inject_program_int (vcpu,PGM_ADDRESSING );if (rc <0 )
@@ -385,7 +385,7 @@ static int handle_sske(struct kvm_vcpu *vcpu)if (kvm_is_error_hva (vmaddr))return kvm_s390_inject_program_int (vcpu,PGM_ADDRESSING ); -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); rc =cond_set_guest_storage_key (current->mm, vmaddr, key, &oldkey, m3 &SSKE_NQ , m3 &SSKE_MR , m3 &SSKE_MC );
@@ -395,7 +395,7 @@ static int handle_sske(struct kvm_vcpu *vcpu) FAULT_FLAG_WRITE, &unlocked); rc = !rc ? -EAGAIN : rc; } -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (rc == -EFAULT )return kvm_s390_inject_program_int (vcpu,PGM_ADDRESSING );if (rc <0 )
@@ -1084,7 +1084,7 @@ static int handle_pfmf(struct kvm_vcpu *vcpu)if (rc)return rc; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); rc =cond_set_guest_storage_key (current->mm, vmaddr, key,NULL , nq, mr, mc);if (rc <0 ) {
@@ -1092,7 +1092,7 @@ static int handle_pfmf(struct kvm_vcpu *vcpu) FAULT_FLAG_WRITE, &unlocked); rc = !rc ? -EAGAIN : rc; } -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (rc == -EFAULT )return kvm_s390_inject_program_int (vcpu,PGM_ADDRESSING );if (rc == -EAGAIN )
@@ -1213,9 +1213,9 @@ static int handle_essa(struct kvm_vcpu *vcpu)* already correct, we do nothing and avoid the lock. */ if (vcpu->kvm->mm->context.uses_cmm ==0 ) { -down_write (&vcpu->kvm->mm->mmap_sem); +mm_write_lock (vcpu->kvm->mm); vcpu->kvm->mm->context.uses_cmm =1 ; -up_write (&vcpu->kvm->mm->mmap_sem); +mm_write_unlock (vcpu->kvm->mm); }/* * If we are here, we are supposed to have CMMA enabled in
@@ -1232,11 +1232,11 @@ static int handle_essa(struct kvm_vcpu *vcpu) }else {int srcu_idx; -down_read (&vcpu->kvm->mm->mmap_sem); +mm_read_lock (vcpu->kvm->mm); srcu_idx =srcu_read_lock (&vcpu->kvm->srcu); i =__do_essa (vcpu, orc);srcu_read_unlock (&vcpu->kvm->srcu, srcu_idx); -up_read (&vcpu->kvm->mm->mmap_sem); +mm_read_unlock (vcpu->kvm->mm);if (i <0 )return i;/* Account for the possible extra cbrl entry */
@@ -1244,10 +1244,10 @@ static int handle_essa(struct kvm_vcpu *vcpu) } vcpu->arch.sie_block->cbrlo &=PAGE_MASK ;/* reset nceo */ cbrlo =phys_to_virt (vcpu->arch.sie_block->cbrlo); -down_read (&gmap->mm->mmap_sem); +mm_read_lock (gmap->mm);for (i =0 ; i < entries; ++i)__gmap_zap (gmap, cbrlo[i]); -up_read (&gmap->mm->mmap_sem); +mm_read_unlock (gmap->mm);return 0 ; }
s390/mm/fault.c +3 -3
@@ -434,7 +434,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) flags |=FAULT_FLAG_USER ;if (access ==VM_WRITE || (trans_exc_code & store_indication) ==0x400 ) flags |=FAULT_FLAG_WRITE ; -down_read (&mm->mmap_sem); +mm_read_lock (mm); gmap =NULL ;if (IS_ENABLED (CONFIG_PGSTE ) && type ==GMAP_FAULT ) {
@@ -519,7 +519,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) flags &= ~(FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_RETRY_NOWAIT ); flags |=FAULT_FLAG_TRIED ; -down_read (&mm->mmap_sem); +mm_read_lock (mm); goto retry; } }
@@ -537,7 +537,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) } fault =0 ; out_up: -up_read (&mm->mmap_sem); +mm_read_unlock (mm); out:return fault; }
s390/mm/gmap.c +20 -20
@@ -405,10 +405,10 @@ int gmap_unmap_segment(struct gmap *gmap, unsigned long to, unsigned long len)return -EINVAL ; flush =0 ; -down_write (&gmap->mm->mmap_sem); +mm_write_lock (gmap->mm);for (off =0 ; off < len; off +=PMD_SIZE ) flush |=__gmap_unmap_by_gaddr (gmap, to + off); -up_write (&gmap->mm->mmap_sem); +mm_write_unlock (gmap->mm);if (flush)gmap_flush_tlb (gmap);return 0 ;
@@ -438,7 +438,7 @@ int gmap_map_segment(struct gmap *gmap, unsigned long from,return -EINVAL ; flush =0 ; -down_write (&gmap->mm->mmap_sem); +mm_write_lock (gmap->mm);for (off =0 ; off < len; off +=PMD_SIZE ) {/* Remove old translation */ flush |=__gmap_unmap_by_gaddr (gmap, to + off);
@@ -448,7 +448,7 @@ int gmap_map_segment(struct gmap *gmap, unsigned long from, (void *) from + off))break ; } -up_write (&gmap->mm->mmap_sem); +mm_write_unlock (gmap->mm);if (flush)gmap_flush_tlb (gmap);if (off >= len)
@@ -495,9 +495,9 @@ unsigned long gmap_translate(struct gmap *gmap, unsigned long gaddr) {unsigned long rc; -down_read (&gmap->mm->mmap_sem); +mm_read_lock (gmap->mm); rc =__gmap_translate (gmap, gaddr); -up_read (&gmap->mm->mmap_sem); +mm_read_unlock (gmap->mm);return rc; }EXPORT_SYMBOL_GPL (gmap_translate);
@@ -640,7 +640,7 @@ int gmap_fault(struct gmap *gmap, unsigned long gaddr,int rc;bool unlocked; -down_read (&gmap->mm->mmap_sem); +mm_read_lock (gmap->mm); retry: unlocked = false;
@@ -663,7 +663,7 @@ int gmap_fault(struct gmap *gmap, unsigned long gaddr, rc =__gmap_link (gmap, gaddr, vmaddr); out_up: -up_read (&gmap->mm->mmap_sem); +mm_read_unlock (gmap->mm);return rc; }EXPORT_SYMBOL_GPL (gmap_fault);
@@ -696,7 +696,7 @@ void gmap_discard(struct gmap *gmap, unsigned long from, unsigned long to)unsigned long gaddr, vmaddr, size;struct vm_area_struct *vma; -down_read (&gmap->mm->mmap_sem); +mm_read_lock (gmap->mm);for (gaddr = from; gaddr < to; gaddr = (gaddr +PMD_SIZE ) &PMD_MASK ) {/* Find the vm address for the guest address */
@@ -719,7 +719,7 @@ void gmap_discard(struct gmap *gmap, unsigned long from, unsigned long to) size =min (to - gaddr,PMD_SIZE - (gaddr & ~PMD_MASK ));zap_page_range (vma, vmaddr, size); } -up_read (&gmap->mm->mmap_sem); +mm_read_unlock (gmap->mm); }EXPORT_SYMBOL_GPL (gmap_discard);
@@ -1102,9 +1102,9 @@ int gmap_mprotect_notify(struct gmap *gmap, unsigned long gaddr,return -EINVAL ;if (!MACHINE_HAS_ESOP && prot ==PROT_READ )return -EINVAL ; -down_read (&gmap->mm->mmap_sem); +mm_read_lock (gmap->mm); rc =gmap_protect_range (gmap, gaddr, len, prot,GMAP_NOTIFY_MPROT ); -up_read (&gmap->mm->mmap_sem); +mm_read_unlock (gmap->mm);return rc; }EXPORT_SYMBOL_GPL (gmap_mprotect_notify);
@@ -1692,11 +1692,11 @@ struct gmap *gmap_shadow(struct gmap *parent, unsigned long asce, }spin_unlock (&parent->shadow_lock);/* protect after insertion, so it will get properly invalidated */ -down_read (&parent->mm->mmap_sem); +mm_read_lock (parent->mm); rc =gmap_protect_range (parent, asce & _ASCE_ORIGIN, ((asce & _ASCE_TABLE_LENGTH) +1 ) *PAGE_SIZE ,PROT_READ ,GMAP_NOTIFY_SHADOW ); -up_read (&parent->mm->mmap_sem); +mm_read_unlock (parent->mm);spin_lock (&parent->shadow_lock); new->initialized = true;if (rc) {
@@ -2538,12 +2538,12 @@ int s390_enable_sie(void)/* Fail if the page tables are 2K */ if (!mm_alloc_pgste (mm))return -EINVAL ; -down_write (&mm->mmap_sem); +mm_write_lock (mm); mm->context.has_pgste =1 ;/* split thp mappings and disable thp for future mappings */ thp_split_mm (mm);walk_page_range (mm,0 ,TASK_SIZE , &zap_zero_walk_ops,NULL ); -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return 0 ; }EXPORT_SYMBOL_GPL (s390_enable_sie);
@@ -2596,7 +2596,7 @@ int s390_enable_skey(void)struct vm_area_struct *vma;int rc =0 ; -down_write (&mm->mmap_sem); +mm_write_lock (mm);if (mm_uses_skeys (mm)) goto out_up;
@@ -2614,7 +2614,7 @@ int s390_enable_skey(void)walk_page_range (mm,0 ,TASK_SIZE , &enable_skey_walk_ops,NULL ); out_up: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return rc; }EXPORT_SYMBOL_GPL (s390_enable_skey);
@@ -2635,8 +2635,8 @@ static const struct mm_walk_ops reset_cmma_walk_ops = {void s390_reset_cmma (struct mm_struct *mm) { -down_write (&mm->mmap_sem); +mm_write_lock (mm);walk_page_range (mm,0 ,TASK_SIZE , &reset_cmma_walk_ops,NULL ); -up_write (&mm->mmap_sem); +mm_write_unlock (mm); }EXPORT_SYMBOL_GPL (s390_reset_cmma);
s390/pci/pci_mmio.c +2 -2
@@ -18,7 +18,7 @@ static long get_pfn(unsigned long user_addr, unsigned long access,struct vm_area_struct *vma;long ret; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); ret = -EINVAL ; vma =find_vma (current->mm, user_addr);if (!vma)
@@ -28,7 +28,7 @@ static long get_pfn(unsigned long user_addr, unsigned long access, goto out; ret =follow_pfn (vma, user_addr, pfn); out: -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);return ret; }
sh/kernel/sys_sh.c +3 -3
@@ -69,10 +69,10 @@ asmlinkage int sys_cacheflush(unsigned long addr, unsigned long len, int op)if (addr + len < addr)return -EFAULT ; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); vma =find_vma (current->mm, addr);if (vma ==NULL || addr < vma->vm_start || addr + len > vma->vm_end) { -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);return -EFAULT ; }
@@ -91,6 +91,6 @@ asmlinkage int sys_cacheflush(unsigned long addr, unsigned long len, int op)if (op &CACHEFLUSH_I )flush_icache_range (addr, addr+len); -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);return 0 ; }
sh/kernel/vsyscall/vsyscall.c +2 -2
@@ -61,7 +61,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)unsigned long addr;int ret; -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ; addr =get_unmapped_area (NULL ,0 ,PAGE_SIZE ,0 ,0 );
@@ -80,7 +80,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) current->mm->context.vdso = (void *)addr; up_fail: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return ret; }
sh/mm/fault.c +7 -7
@@ -261,7 +261,7 @@ __bad_area(struct pt_regs *regs, unsigned long error_code,* Something tried to access memory that isn't in our memory map.. * Fix it, but check if it's kernel or user first.. */ -up_read (&mm->mmap_sem); +mm_read_unlock (mm);__bad_area_nosemaphore (regs, error_code, address, si_code); }
@@ -285,7 +285,7 @@ do_sigbus(struct pt_regs *regs, unsigned long error_code, unsigned long address)struct task_struct *tsk = current;struct mm_struct *mm = tsk->mm; -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* Kernel mode? Handle exceptions or die: */ if (!user_mode (regs))
@@ -304,7 +304,7 @@ mm_fault_error(struct pt_regs *regs, unsigned long error_code,*/ if (fatal_signal_pending (current)) {if (!(fault &VM_FAULT_RETRY )) -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (!user_mode (regs))no_context (regs, error_code, address);return 1 ;
@@ -316,11 +316,11 @@ mm_fault_error(struct pt_regs *regs, unsigned long error_code,if (fault &VM_FAULT_OOM ) {/* Kernel mode? Handle exceptions or die: */ if (!user_mode (regs)) { -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);no_context (regs, error_code, address);return 1 ; } -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);/* * We ran out of memory, call the OOM killer, and return the
@@ -424,7 +424,7 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs, } retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, address);if (unlikely (!vma)) {
@@ -493,5 +493,5 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs, } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm); }
sparc/mm/fault_32.c +9 -9
@@ -196,7 +196,7 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm);if (!from_user && address >=PAGE_OFFSET ) goto bad_area;
@@ -273,7 +273,7 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write, } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ;/*
@@ -281,7 +281,7 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm); bad_area_nosemaphore:/* User mode accesses just cause a SIGSEGV */
@@ -330,7 +330,7 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (from_user) {pagefault_out_of_memory ();return ;
@@ -338,7 +338,7 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write, goto no_context; do_sigbus: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);do_fault_siginfo (BUS_ADRERR ,SIGBUS , regs, text_fault);if (!from_user) goto no_context;
@@ -392,7 +392,7 @@ static void force_user_fault(unsigned long address, int write) code =SEGV_MAPERR ; -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, address);if (!vma) goto bad_area;
@@ -417,15 +417,15 @@ static void force_user_fault(unsigned long address, int write)case VM_FAULT_OOM : goto do_sigbus; } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ; bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);__do_fault_siginfo (code,SIGSEGV , tsk->thread.kregs, address);return ; do_sigbus: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);__do_fault_siginfo (BUS_ADRERR ,SIGBUS , tsk->thread.kregs, address); }
sparc/mm/fault_64.c +6 -6
@@ -315,7 +315,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); -if (!down_read_trylock (&mm->mmap_sem)) { +if (!mm_read_trylock (mm)) {if ((regs->tstate &TSTATE_PRIV ) && !search_exception_tables (regs->tpc)) { insn =get_fault_insn (regs, insn);
@@ -323,7 +323,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs) } retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); }if (fault_code &FAULT_CODE_BAD_RA )
@@ -456,7 +456,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs) goto retry; } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm); mm_rss =get_mm_rss (mm);#if defined(CONFIG_TRANSPARENT_HUGEPAGE )
@@ -487,7 +487,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)*/ bad_area: insn =get_fault_insn (regs, insn); -up_read (&mm->mmap_sem); +mm_read_unlock (mm); handle_kernel_fault:do_kernel_fault (regs, si_code, fault_code, insn, address);
@@ -499,7 +499,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)*/ out_of_memory: insn =get_fault_insn (regs, insn); -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!(regs->tstate &TSTATE_PRIV )) {pagefault_out_of_memory (); goto exit_exception;
@@ -512,7 +512,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs) do_sigbus: insn =get_fault_insn (regs, insn); -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* * Send a sigbus, regardless of whether we were in kernel
sparc/vdso/vma.c +2 -2
@@ -366,7 +366,7 @@ static int map_vdso(const struct vdso_image *image,unsigned long text_start, addr =0 ;int ret =0 ; -down_write (&mm->mmap_sem); +mm_write_lock (mm);/* * First, get an unmapped region: then randomize it, and make sure that
@@ -422,7 +422,7 @@ static int map_vdso(const struct vdso_image *image,if (ret) current->mm->context.vdso =NULL ; -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return ret; }
um/include/asm/mmu_context.h +1 -1
@@ -54,7 +54,7 @@ static inline void activate_mm(struct mm_struct *old, struct mm_struct *new)__switch_mm (&new->context.id);down_write_nested (&new->mmap_sem,1 );uml_setup_stubs (new); -up_write (&new->mmap_sem); +mm_write_unlock (new); } static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,
um/kernel/tlb.c +1 -1
@@ -350,7 +350,7 @@ void fix_range_common(struct mm_struct *mm, unsigned long start_addr,printk (KERN_ERR "fix_range_common: failed, killing current " "process: %d\n" ,task_tgid_vnr (current));/* We are under mmap_sem, release it such that current can terminate */ -up_write (¤t->mm->mmap_sem); +mm_write_unlock (current->mm);force_sig (SIGKILL );do_signal (¤t->thread.regs); }
um/kernel/trap.c +3 -3
@@ -47,7 +47,7 @@ int handle_page_fault(unsigned long address, unsigned long ip,if (is_user) flags |=FAULT_FLAG_USER ; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, address);if (!vma) goto out;
@@ -124,7 +124,7 @@ int handle_page_fault(unsigned long address, unsigned long ip,#endif flush_tlb_page (vma, address); out: -up_read (&mm->mmap_sem); +mm_read_unlock (mm); out_nosemaphore:return err;
@@ -133,7 +133,7 @@ int handle_page_fault(unsigned long address, unsigned long ip,* We ran out of memory, call the OOM killer, and return the userspace * (which will retry the fault, or kill us if we got oom-killed). */ -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!is_user) goto out_nosemaphore;pagefault_out_of_memory ();
unicore32/mm/fault.c +3 -3
@@ -224,12 +224,12 @@ static int do_pf(unsigned long addr, unsigned int fsr, struct pt_regs *regs)* validly references user space from well defined areas of the code, * we can bug out early if this is from code which shouldn't. */ -if (!down_read_trylock (&mm->mmap_sem)) { +if (!mm_read_trylock (mm)) {if (!user_mode (regs) && !search_exception_tables (regs->UCreg_pc)) goto no_context; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); }else {/* * The above down_read_trylock() might have succeeded in
@@ -266,7 +266,7 @@ static int do_pf(unsigned long addr, unsigned int fsr, struct pt_regs *regs) } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* * Handle the "normal" case first - VM_FAULT_MAJOR
x86/entry/vdso/vma.c +5 -5
@@ -150,7 +150,7 @@ static int map_vdso(const struct vdso_image *image, unsigned long addr)unsigned long text_start;int ret =0 ; -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ; addr = get_unmapped_area(NULL, addr,
@@ -193,7 +193,7 @@ static int map_vdso(const struct vdso_image *image, unsigned long addr) } up_fail: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return ret; }
@@ -255,7 +255,7 @@ int map_vdso_once(const struct vdso_image *image, unsigned long addr)struct mm_struct *mm = current->mm;struct vm_area_struct *vma; -down_write (&mm->mmap_sem); +mm_write_lock (mm);/* * Check if we have already mapped vdso blob - fail to prevent * abusing from userspace install_speciall_mapping, which may
@@ -266,11 +266,11 @@ int map_vdso_once(const struct vdso_image *image, unsigned long addr)for (vma = mm->mmap; vma; vma = vma->vm_next) {if (vma_is_special_mapping (vma, &vdso_mapping) ||vma_is_special_mapping (vma, &vvar_mapping)) { -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return -EEXIST ; } } -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return map_vdso (image, addr); }
x86/kernel/vm86_32.c +2 -2
@@ -172,7 +172,7 @@ static void mark_screen_rdonly(struct mm_struct *mm)pte_t *pte;int i; -down_write (&mm->mmap_sem); +mm_write_lock (mm); pgd =pgd_offset (mm,0xA0000 );if (pgd_none_or_clear_bad (pgd)) goto out;
@@ -198,7 +198,7 @@ static void mark_screen_rdonly(struct mm_struct *mm) }pte_unmap_unlock (pte, ptl); out: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);flush_tlb_mm_range (mm,0xA0000 ,0xA0000 +32 *PAGE_SIZE ,PAGE_SHIFT , false); }
x86/mm/debug_pagetables.c +4 -4
@@ -16,9 +16,9 @@ DEFINE_SHOW_ATTRIBUTE(ptdump);static int ptdump_curknl_show (struct seq_file *m,void *v) {if (current->mm->pgd) { -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm);ptdump_walk_pgd_level_debugfs (m, current->mm->pgd, false); -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); }return 0 ; }
@@ -29,9 +29,9 @@ DEFINE_SHOW_ATTRIBUTE(ptdump_curknl);static int ptdump_curusr_show (struct seq_file *m,void *v) {if (current->mm->pgd) { -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm);ptdump_walk_pgd_level_debugfs (m, current->mm->pgd, true); -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); }return 0 ; }
x86/mm/fault.c +4 -4
@@ -928,7 +928,7 @@ __bad_area(struct pt_regs *regs, unsigned long error_code,* Something tried to access memory that isn't in our memory map.. * Fix it, but check if it's kernel or user first.. */ -up_read (&mm->mmap_sem); +mm_read_unlock (mm);__bad_area_nosemaphore (regs, error_code, address, pkey, si_code); }
@@ -1379,7 +1379,7 @@ void do_user_addr_fault(struct pt_regs *regs,* 1. Failed to acquire mmap_sem, and * 2. The access did not originate in userspace. */ -if (unlikely (!down_read_trylock (&mm->mmap_sem))) { +if (unlikely (!mm_read_trylock (mm))) {if (!user_mode (regs) && !search_exception_tables (regs->ip)) {/* * Fault from code in kernel from
@@ -1389,7 +1389,7 @@ void do_user_addr_fault(struct pt_regs *regs,return ; } retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); }else {/* * The above down_read_trylock() might have succeeded in
@@ -1464,7 +1464,7 @@ void do_user_addr_fault(struct pt_regs *regs,return ; } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (unlikely (fault &VM_FAULT_ERROR )) {mm_fault_error (regs, hw_error_code, address, fault);return ;
x86/mm/mpx.c +6 -6
@@ -52,10 +52,10 @@ static unsigned long mpx_mmap(unsigned long len)if (len !=mpx_bt_size_bytes (mm))return -EINVAL ; -down_write (&mm->mmap_sem); +mm_write_lock (mm); addr =do_mmap (NULL ,0 , len,PROT_READ |PROT_WRITE ,MAP_ANONYMOUS |MAP_PRIVATE ,VM_MPX ,0 , &populate,NULL ); -up_write (&mm->mmap_sem); +mm_write_unlock (mm);if (populate)mm_populate (addr, populate);
@@ -227,7 +227,7 @@ int mpx_enable_management(void)* unmap path; we can just use mm->context.bd_addr instead. */ bd_base =mpx_get_bounds_dir (); -down_write (&mm->mmap_sem); +mm_write_lock (mm);/* MPX doesn't support addresses above 47 bits yet. */ if (find_vma (mm,DEFAULT_MAP_WINDOW )) {
@@ -241,7 +241,7 @@ int mpx_enable_management(void)if (mm->context.bd_addr ==MPX_INVALID_BOUNDS_DIR ) ret = -ENXIO ; out: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return ret; }
@@ -252,9 +252,9 @@ int mpx_disable_management(void)if (!cpu_feature_enabled (X86_FEATURE_MPX ))return -ENXIO ; -down_write (&mm->mmap_sem); +mm_write_lock (mm); mm->context.bd_addr =MPX_INVALID_BOUNDS_DIR ; -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return 0 ; }
x86/um/vdso/vma.c +2 -2
@@ -58,7 +58,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)if (!vdso_enabled)return 0 ; -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ; err = install_special_mapping(mm, um_vdso_addr, PAGE_SIZE,
@@ -66,7 +66,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)VM_MAYREAD |VM_MAYWRITE |VM_MAYEXEC , vdsop); -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return err; }
xtensa/mm/fault.c +5 -5
@@ -74,7 +74,7 @@ void do_page_fault(struct pt_regs *regs)if (user_mode (regs)) flags |=FAULT_FLAG_USER ; retry: -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, address);if (!vma)
@@ -140,7 +140,7 @@ void do_page_fault(struct pt_regs *regs) } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address);if (flags &VM_FAULT_MAJOR )perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , regs, address);
@@ -153,7 +153,7 @@ void do_page_fault(struct pt_regs *regs)* Fix it, but check if it's kernel or user first.. */ bad_area: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (user_mode (regs)) { current->thread.bad_vaddr = address; current->thread.error_code = is_write;
@@ -168,7 +168,7 @@ void do_page_fault(struct pt_regs *regs)* us unable to handle the page fault gracefully. */ out_of_memory: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (!user_mode (regs))bad_page_fault (regs, address,SIGKILL );else
@@ -176,7 +176,7 @@ void do_page_fault(struct pt_regs *regs)return ; do_sigbus: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* Send a sigbus, regardless of whether we were in kernel * or user mode.
android/binder_alloc.c +3 -3
@@ -212,7 +212,7 @@ static int binder_update_page_range(struct binder_alloc *alloc, int allocate, mm = alloc->vma_vm_mm;if (mm) { -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma = alloc->vma; }
@@ -270,7 +270,7 @@ static int binder_update_page_range(struct binder_alloc *alloc, int allocate,trace_binder_alloc_page_end (alloc, index); }if (mm) { -up_read (&mm->mmap_sem); +mm_read_unlock (mm);mmput (mm); }return 0 ;
@@ -303,7 +303,7 @@ static int binder_update_page_range(struct binder_alloc *alloc, int allocate, } err_no_vma:if (mm) { -up_read (&mm->mmap_sem); +mm_read_unlock (mm);mmput (mm); }return vma ? -ENOMEM : -ESRCH ;
gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +2 -2
@@ -1343,9 +1343,9 @@ int amdgpu_amdkfd_gpuvm_map_memory_to_gpu(* concurrently and the queues are actually stopped */ if (amdgpu_ttm_tt_get_usermm (bo->tbo.ttm)) { -down_write (¤t->mm->mmap_sem); +mm_write_lock (current->mm); is_invalid_userptr =atomic_read (&mem->invalid); -up_write (¤t->mm->mmap_sem); +mm_write_unlock (current->mm); }mutex_lock (&mem->lock);
gpu/drm/amd/amdgpu/amdgpu_ttm.c +5 -5
@@ -838,7 +838,7 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) goto out_free_ranges; } -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, start);if (unlikely (!vma || start < vma->vm_start)) { r = -EFAULT ;
@@ -849,15 +849,15 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) r = -EPERM ; goto out_unlock; } -up_read (&mm->mmap_sem); +mm_read_unlock (mm); timeout = jiffies +msecs_to_jiffies (HMM_RANGE_DEFAULT_TIMEOUT ); retry: range->notifier_seq =mmu_interval_read_begin (&bo->notifier); -down_read (&mm->mmap_sem); +mm_read_lock (mm); r =hmm_range_fault (range,0 ); -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (unlikely (r <=0 )) {/* * FIXME: This timeout should encompass the retry from
@@ -886,7 +886,7 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)return 0 ; out_unlock: -up_read (&mm->mmap_sem); +mm_read_unlock (mm); out_free_pfns:kvfree (range->pfns); out_free_ranges:
gpu/drm/amd/amdkfd/kfd_events.c +2 -2
@@ -902,7 +902,7 @@ void kfd_signal_iommu_event(struct kfd_dev *dev, unsigned int pasid,memset (&memory_exception_data,0 ,sizeof (memory_exception_data)); -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, address); memory_exception_data.gpu_id = dev->id;
@@ -925,7 +925,7 @@ void kfd_signal_iommu_event(struct kfd_dev *dev, unsigned int pasid, memory_exception_data.failure.NoExecute =0 ; } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);mmput (mm); pr_debug("notpresent %d, noexecute %d, readonly %d\n",
gpu/drm/i915/gem/i915_gem_mman.c +2 -2
@@ -89,7 +89,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,struct mm_struct *mm = current->mm;struct vm_area_struct *vma; -if (down_write_killable (&mm->mmap_sem)) { +if (mm_write_lock_killable (mm)) { addr = -EINTR ; goto err; }
@@ -99,7 +99,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,pgprot_writecombine (vm_get_page_prot (vma->vm_flags)); else addr = -ENOMEM ; -up_write (&mm->mmap_sem); +mm_write_unlock (mm);if (IS_ERR_VALUE (addr)) goto err; }
gpu/drm/i915/gem/i915_gem_userptr.c +4 -4
@@ -201,7 +201,7 @@ i915_mmu_notifier_find(struct i915_mm_struct *mm)if (IS_ERR (mn)) err =PTR_ERR (mn); -down_write (&mm->mm->mmap_sem); +mm_write_lock (mm->mm);mutex_lock (&mm->i915->mm_lock);if (mm->mn ==NULL && !err) {/* Protected by mmap_sem (write-lock) */
@@ -218,7 +218,7 @@ i915_mmu_notifier_find(struct i915_mm_struct *mm) err =0 ; }mutex_unlock (&mm->i915->mm_lock); -up_write (&mm->mm->mmap_sem); +mm_write_unlock (mm->mm);if (mn && !IS_ERR (mn))kfree (mn);
@@ -466,7 +466,7 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work) ret = -EFAULT ;if (mmget_not_zero (mm)) { -down_read (&mm->mmap_sem); +mm_read_lock (mm);while (pinned < npages) { ret = get_user_pages_remote (work->task, mm,
@@ -479,7 +479,7 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work) pinned += ret; } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);mmput (mm); } }
gpu/drm/nouveau/nouveau_svm.c +10 -10
@@ -169,7 +169,7 @@ nouveau_svmm_bind(struct drm_device *dev, void *data,*/ mm =get_task_mm (current); -down_read (&mm->mmap_sem); +mm_read_lock (mm);for (addr = args->va_start, end = args->va_start + size; addr < end;) {struct vm_area_struct *vma;
@@ -192,7 +192,7 @@ nouveau_svmm_bind(struct drm_device *dev, void *data,*/ args->result =0 ; -up_read (&mm->mmap_sem); +mm_read_unlock (mm);mmput (mm);return 0 ;
@@ -342,7 +342,7 @@ nouveau_svmm_init(struct drm_device *dev, void *data,if (ret) goto out_free; -down_write (¤t->mm->mmap_sem); +mm_write_lock (current->mm); svmm->notifier.ops = &nouveau_mn_ops; ret =__mmu_notifier_register (&svmm->notifier, current->mm);if (ret)
@@ -351,12 +351,12 @@ nouveau_svmm_init(struct drm_device *dev, void *data, cli->svm.svmm = svmm; cli->svm.cli = cli; -up_write (¤t->mm->mmap_sem); +mm_write_unlock (current->mm);mutex_unlock (&cli->mutex);return 0 ; out_mm_unlock: -up_write (¤t->mm->mmap_sem); +mm_write_unlock (current->mm); out_free:mutex_unlock (&cli->mutex);kfree (svmm);
@@ -540,9 +540,9 @@ static int nouveau_range_fault(struct nouveau_svmm *svmm, range.notifier_seq =mmu_interval_read_begin (range.notifier); range.default_flags =0 ; range.pfn_flags_mask =-1UL ; -down_read (&mm->mmap_sem); +mm_read_lock (mm); ret =hmm_range_fault (&range,0 ); -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (ret <=0 ) {if (ret ==0 || ret == -EBUSY )continue ;
@@ -671,18 +671,18 @@ nouveau_svm_fault(struct nvif_notify *notify)/* Intersect fault window with the CPU VMA, cancelling * the fault if the address is invalid. */ -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma_intersection (mm, start, limit);if (!vma) {SVMM_ERR (svmm,"wndw %016llx-%016llx" , start, limit); -up_read (&mm->mmap_sem); +mm_read_unlock (mm);mmput (mm);nouveau_svm_fault_cancel_fault (svm, buffer->fault[fi]);continue ; } start =max_t (u64, start, vma->vm_start); limit =min_t (u64, limit, vma->vm_end); -up_read (&mm->mmap_sem); +mm_read_unlock (mm);SVMM_DBG (svmm,"wndw %016llx-%016llx" , start, limit);if (buffer->fault[fi]->addr != start) {
gpu/drm/radeon/radeon_cs.c +2 -2
@@ -196,12 +196,12 @@ static int radeon_cs_parser_relocs(struct radeon_cs_parser *p) p->vm_bos =radeon_vm_get_bos (p->rdev, p->ib.vm, &p->validated);if (need_mmap_lock) -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); r =radeon_bo_list_validate (p->rdev, &p->ticket, &p->validated, p->ring);if (need_mmap_lock) -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);return r; }
gpu/drm/radeon/radeon_gem.c +3 -3
@@ -341,17 +341,17 @@ int radeon_gem_userptr_ioctl(struct drm_device *dev, void *data, }if (args->flags &RADEON_GEM_USERPTR_VALIDATE ) { -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); r =radeon_bo_reserve (bo, true);if (r) { -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); goto release_object; }radeon_ttm_placement_from_domain (bo,RADEON_GEM_DOMAIN_GTT ); r =ttm_bo_validate (&bo->tbo, &bo->placement, &ctx);radeon_bo_unreserve (bo); -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (r) goto release_object; }
gpu/drm/ttm/ttm_bo_vm.c +2 -2
@@ -67,7 +67,7 @@ static vm_fault_t ttm_bo_vm_fault_idle(struct ttm_buffer_object *bo, goto out_unlock;ttm_bo_get (bo); -up_read (&vmf->vma->vm_mm->mmap_sem); +mm_read_unlock (vmf->vma->vm_mm); (void )dma_fence_wait (bo->moving, true);dma_resv_unlock (bo->base.resv);ttm_bo_put (bo);
@@ -138,7 +138,7 @@ vm_fault_t ttm_bo_vm_reserve(struct ttm_buffer_object *bo,if (vmf->flags &FAULT_FLAG_ALLOW_RETRY ) {if (!(vmf->flags &FAULT_FLAG_RETRY_NOWAIT )) {ttm_bo_get (bo); -up_read (&vmf->vma->vm_mm->mmap_sem); +mm_read_unlock (vmf->vma->vm_mm); (void )ttm_bo_wait_unreserved (bo);ttm_bo_put (bo); }
infiniband/core/umem.c +3 -3
@@ -266,14 +266,14 @@ struct ib_umem *ib_umem_get(struct ib_udata *udata, unsigned long addr, sg = umem->sg_head.sgl;while (npages) { -down_read (&mm->mmap_sem); +mm_read_lock (mm); ret =get_user_pages (cur_base, min_t(unsigned long, npages,PAGE_SIZE /sizeof (struct page *)), gup_flags |FOLL_LONGTERM , page_list,NULL );if (ret <0 ) { -up_read (&mm->mmap_sem); +mm_read_unlock (mm); goto umem_release; }
@@ -284,7 +284,7 @@ struct ib_umem *ib_umem_get(struct ib_udata *udata, unsigned long addr,dma_get_max_seg_size (context->device->dma_device), &umem->sg_nents); -up_read (&mm->mmap_sem); +mm_read_unlock (mm); }sg_mark_end (sg);
infiniband/core/umem_odp.c +5 -5
@@ -246,16 +246,16 @@ struct ib_umem_odp *ib_umem_odp_get(struct ib_udata *udata, unsigned long addr,struct vm_area_struct *vma;struct hstate *h; -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm,ib_umem_start (umem_odp));if (!vma || !is_vm_hugetlb_page (vma)) { -up_read (&mm->mmap_sem); +mm_read_unlock (mm); ret = -EINVAL ; goto err_free; } h =hstate_vma (vma); umem_odp->page_shift =huge_page_shift (h); -up_read (&mm->mmap_sem); +mm_read_unlock (mm); } umem_odp->tgid =get_task_pid (current->group_leader,PIDTYPE_PID );
@@ -443,7 +443,7 @@ int ib_umem_odp_map_dma_pages(struct ib_umem_odp *umem_odp, u64 user_virt, (bcnt +BIT (page_shift) -1 ) >> page_shift, PAGE_SIZE / sizeof(struct page *)); -down_read (&owning_mm->mmap_sem); +mm_read_lock (owning_mm);/* * Note: this might result in redundent page getting. We can * avoid this by checking dma_list to be 0 before calling
@@ -454,7 +454,7 @@ int ib_umem_odp_map_dma_pages(struct ib_umem_odp *umem_odp, u64 user_virt, npages =get_user_pages_remote (owning_process, owning_mm, user_virt, gup_num_pages, flags, local_page_list,NULL ,NULL ); -up_read (&owning_mm->mmap_sem); +mm_read_unlock (owning_mm);if (npages <0 ) {if (npages != -EAGAIN )
infiniband/core/uverbs_main.c +2 -2
@@ -939,7 +939,7 @@ void uverbs_user_mmap_disassociate(struct ib_uverbs_file *ufile)* at a time to get the lock ordering right. Typically there * will only be one mm, so no big deal. */ -down_read (&mm->mmap_sem); +mm_read_lock (mm);if (!mmget_still_valid (mm)) goto skip_mm;mutex_lock (&ufile->umap_lock);
@@ -961,7 +961,7 @@ void uverbs_user_mmap_disassociate(struct ib_uverbs_file *ufile) }mutex_unlock (&ufile->umap_lock); skip_mm: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);mmput (mm); } }
infiniband/hw/mlx4/mr.c +2 -2
@@ -380,7 +380,7 @@ static struct ib_umem *mlx4_get_umem_mr(struct ib_udata *udata, u64 start,unsigned long untagged_start =untagged_addr (start);struct vm_area_struct *vma; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm);/* * FIXME: Ideally this would iterate over all the vmas that * cover the memory, but for now it requires a single vma to
@@ -395,7 +395,7 @@ static struct ib_umem *mlx4_get_umem_mr(struct ib_udata *udata, u64 start, access_flags |=IB_ACCESS_LOCAL_WRITE ; } -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); }return ib_umem_get (udata, start, length, access_flags);
infiniband/hw/qib/qib_user_pages.c +3 -3
@@ -106,18 +106,18 @@ int qib_get_user_pages(unsigned long start_page, size_t num_pages, goto bail; } -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm);for (got =0 ; got < num_pages; got += ret) { ret =get_user_pages (start_page + got *PAGE_SIZE , num_pages - got,FOLL_LONGTERM |FOLL_WRITE |FOLL_FORCE , p + got,NULL );if (ret <0 ) { -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); goto bail_release; } } -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);return 0 ; bail_release:
infiniband/hw/usnic/usnic_uiom.c +2 -2
@@ -123,7 +123,7 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, npages =PAGE_ALIGN (size + (addr & ~PAGE_MASK )) >>PAGE_SHIFT ; uiomr->owning_mm = mm = current->mm; -down_read (&mm->mmap_sem); +mm_read_lock (mm); locked =atomic64_add_return (npages, ¤t->mm->pinned_vm); lock_limit =rlimit (RLIMIT_MEMLOCK ) >>PAGE_SHIFT ;
@@ -187,7 +187,7 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, }else mmgrab (uiomr->owning_mm); -up_read (&mm->mmap_sem); +mm_read_unlock (mm);free_page ((unsigned long ) page_list);return ret; }
infiniband/sw/siw/siw_mem.c +2 -2
@@ -397,7 +397,7 @@ struct siw_umem *siw_umem_get(u64 start, u64 len, bool writable)if (!writable) foll_flags |=FOLL_FORCE ; -down_read (&mm_s->mmap_sem); +mm_read_lock (mm_s); mlock_limit =rlimit (RLIMIT_MEMLOCK ) >>PAGE_SHIFT ;
@@ -441,7 +441,7 @@ struct siw_umem *siw_umem_get(u64 start, u64 len, bool writable) num_pages -= got; } out_sem_up: -up_read (&mm_s->mmap_sem); +mm_read_unlock (mm_s);if (rv >0 )return umem;
iommu/amd_iommu_v2.c +2 -2
@@ -487,7 +487,7 @@ static void do_fault(struct work_struct *work) flags |=FAULT_FLAG_WRITE ; flags |=FAULT_FLAG_REMOTE ; -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_extend_vma (mm, address);if (!vma || address < vma->vm_start)/* failed to get a vma in the right range */
@@ -499,7 +499,7 @@ static void do_fault(struct work_struct *work) ret =handle_mm_fault (vma, address, flags); out: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (ret &VM_FAULT_ERROR )/* failed to service fault */
iommu/intel-svm.c +2 -2
@@ -591,7 +591,7 @@ static irqreturn_t prq_event_thread(int irq, void *d)if (!is_canonical_address (address)) goto bad_req; -down_read (&svm->mm->mmap_sem); +mm_read_lock (svm->mm); vma =find_extend_vma (svm->mm, address);if (!vma || address < vma->vm_start) goto invalid;
@@ -606,7 +606,7 @@ static irqreturn_t prq_event_thread(int irq, void *d) result =QI_RESP_SUCCESS ; invalid: -up_read (&svm->mm->mmap_sem); +mm_read_unlock (svm->mm);mmput (svm->mm); bad_req:/* Accounting for major/minor faults? */
media/v4l2-core/videobuf-core.c +2 -2
@@ -534,7 +534,7 @@ int videobuf_qbuf(struct videobuf_queue *q, struct v4l2_buffer *b)MAGIC_CHECK (q->int_ops->magic,MAGIC_QTYPE_OPS );if (b->memory ==V4L2_MEMORY_MMAP ) -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm);videobuf_queue_lock (q); retval = -EBUSY ;
@@ -621,7 +621,7 @@ int videobuf_qbuf(struct videobuf_queue *q, struct v4l2_buffer *b)videobuf_queue_unlock (q);if (b->memory ==V4L2_MEMORY_MMAP ) -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);return retval; }
media/v4l2-core/videobuf-dma-contig.c +2 -2
@@ -169,7 +169,7 @@ static int videobuf_dma_contig_user_get(struct videobuf_dma_contig_memory *mem, mem->size =PAGE_ALIGN (vb->size + offset); ret = -EINVAL ; -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, untagged_baddr);if (!vma)
@@ -201,7 +201,7 @@ static int videobuf_dma_contig_user_get(struct videobuf_dma_contig_memory *mem, } out_up: -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);return ret; }
media/v4l2-core/videobuf-dma-sg.c +2 -2
@@ -200,9 +200,9 @@ static int videobuf_dma_init_user(struct videobuf_dmabuf *dma, int direction, {int ret; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); ret =videobuf_dma_init_user_locked (dma, direction, data, size); -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);return ret; }
misc/cxl/cxllib.c +2 -2
@@ -207,7 +207,7 @@ static int get_vma_info(struct mm_struct *mm, u64 addr,struct vm_area_struct *vma =NULL ;int rc =0 ; -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =find_vma (mm, addr);if (!vma) {
@@ -218,7 +218,7 @@ static int get_vma_info(struct mm_struct *mm, u64 addr, *vma_start = vma->vm_start; *vma_end = vma->vm_end; out: -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return rc; }
misc/cxl/fault.c +2 -2
@@ -321,7 +321,7 @@ static void cxl_prefault_vma(struct cxl_context *ctx)return ; } -down_read (&mm->mmap_sem); +mm_read_lock (mm);for (vma = mm->mmap; vma; vma = vma->vm_next) {for (ea = vma->vm_start; ea < vma->vm_end; ea =next_segment (ea, slb.vsid)) {
@@ -336,7 +336,7 @@ static void cxl_prefault_vma(struct cxl_context *ctx) last_esid = slb.esid; } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);mmput (mm); }
misc/sgi-gru/grufault.c +8 -8
@@ -69,14 +69,14 @@ static struct gru_thread_state *gru_find_lock_gts(unsigned long vaddr)struct vm_area_struct *vma;struct gru_thread_state *gts =NULL ; -down_read (&mm->mmap_sem); +mm_read_lock (mm); vma =gru_find_vma (vaddr);if (vma) gts =gru_find_thread_state (vma,TSID (vaddr, vma));if (gts)mutex_lock (>s->ts_ctxlock);else -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return gts; }
@@ -86,7 +86,7 @@ static struct gru_thread_state *gru_alloc_locked_gts(unsigned long vaddr)struct vm_area_struct *vma;struct gru_thread_state *gts =ERR_PTR (-EINVAL ); -down_write (&mm->mmap_sem); +mm_write_lock (mm); vma =gru_find_vma (vaddr);if (!vma) goto err;
@@ -95,11 +95,11 @@ static struct gru_thread_state *gru_alloc_locked_gts(unsigned long vaddr)if (IS_ERR (gts)) goto err;mutex_lock (>s->ts_ctxlock); -downgrade_write (&mm->mmap_sem); +mm_downgrade_write_lock (mm);return gts; err: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return gts; }
@@ -109,7 +109,7 @@ static struct gru_thread_state *gru_alloc_locked_gts(unsigned long vaddr)static void gru_unlock_gts (struct gru_thread_state *gts) {mutex_unlock (>s->ts_ctxlock); -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm); }/*
@@ -575,9 +575,9 @@ static irqreturn_t gru_intr(int chiplet, int blade)*/ gts->ustats.fmm_tlbmiss++;if (!gts->ts_force_cch_reload && -down_read_trylock (>s->ts_mm->mmap_sem)) { +mm_read_trylock (gts->ts_mm)) {gru_try_dropin (gru, gts, tfh,NULL ); -up_read (>s->ts_mm->mmap_sem); +mm_read_unlock (gts->ts_mm); }else {tfh_user_polling_mode (tfh);STAT (intr_mm_lock_failed);
misc/sgi-gru/grufile.c +2 -2
@@ -135,7 +135,7 @@ static int gru_create_new_context(unsigned long arg)if (!(req.options &GRU_OPT_MISS_MASK )) req.options |=GRU_OPT_MISS_FMM_INTR ; -down_write (¤t->mm->mmap_sem); +mm_write_lock (current->mm); vma =gru_find_vma (req.gseg);if (vma) { vdata = vma->vm_private_data;
@@ -146,7 +146,7 @@ static int gru_create_new_context(unsigned long arg) vdata->vd_tlb_preload_count = req.tlb_preload_count; ret =0 ; } -up_write (¤t->mm->mmap_sem); +mm_write_unlock (current->mm);return ret; }
oprofile/buffer_sync.c +5 -5
@@ -91,11 +91,11 @@ munmap_notify(struct notifier_block *self, unsigned long val, void *data)struct mm_struct *mm = current->mm;struct vm_area_struct *mpnt; -down_read (&mm->mmap_sem); +mm_read_lock (mm); mpnt =find_vma (mm, addr);if (mpnt && mpnt->vm_file && (mpnt->vm_flags &VM_EXEC )) { -up_read (&mm->mmap_sem); +mm_read_unlock (mm);/* To avoid latency problems, we only process the current CPU, * hoping that most samples for the task are on this CPU */
@@ -103,7 +103,7 @@ munmap_notify(struct notifier_block *self, unsigned long val, void *data)return 0 ; } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return 0 ; }
@@ -256,7 +256,7 @@ lookup_dcookie(struct mm_struct *mm, unsigned long addr, off_t *offset)unsigned long cookie =NO_COOKIE ;struct vm_area_struct *vma; -down_read (&mm->mmap_sem); +mm_read_lock (mm);for (vma =find_vma (mm, addr); vma; vma = vma->vm_next) {if (addr < vma->vm_start || addr >= vma->vm_end)
@@ -276,7 +276,7 @@ lookup_dcookie(struct mm_struct *mm, unsigned long addr, off_t *offset)if (!vma) cookie =INVALID_COOKIE ; -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return cookie; }
staging/kpc2000/kpc_dma/fileops.c +2 -2
@@ -76,9 +76,9 @@ static int kpc_dma_transfer(struct dev_private_data *priv, }// Lock the user buffer pages in memory, and hold on to the page pointers (for the sglist) -down_read (¤t->mm->mmap_sem);/* get memory map semaphore */ +mm_read_lock (current->mm);/* get memory map semaphore */ rv =get_user_pages (iov_base, acd->page_count,FOLL_TOUCH |FOLL_WRITE |FOLL_GET , acd->user_pages,NULL ); -up_read (¤t->mm->mmap_sem);/* release the semaphore */ +mm_read_unlock (current->mm);/* release the semaphore */ if (rv != acd->page_count) {dev_err (&priv->ldev->pldev->dev,"Couldn't get_user_pages (%ld)\n" , rv); goto err_get_user_pages;
tee/optee/call.c +2 -2
@@ -561,10 +561,10 @@ static int check_mem_type(unsigned long start, size_t num_pages)if (virt_addr_valid (start))return 0 ; -down_read (&mm->mmap_sem); +mm_read_lock (mm); rc =__check_mem_type (find_vma (mm, start), start + num_pages *PAGE_SIZE ); -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return rc; }
vfio/vfio_iommu_type1.c +6 -6
@@ -277,11 +277,11 @@ static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async)if (!mm)return -ESRCH ;/* process exited */ - ret =down_write_killable (&mm->mmap_sem); + ret =mm_write_lock_killable (mm);if (!ret) { ret =__account_locked_vm (mm,abs (npage), npage >0 , dma->task, dma->lock_cap); -up_write (&mm->mmap_sem); +mm_write_unlock (mm); }if (async)
@@ -329,7 +329,7 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,if (prot &IOMMU_WRITE ) flags |=FOLL_WRITE ; -down_read (&mm->mmap_sem); +mm_read_lock (mm);if (mm == current->mm) { ret =get_user_pages (vaddr,1 , flags |FOLL_LONGTERM , page, vmas);
@@ -348,14 +348,14 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,put_page (page[0 ]); } } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);if (ret ==1 ) { *pfn =page_to_pfn (page[0 ]);return 0 ; } -down_read (&mm->mmap_sem); +mm_read_lock (mm); vaddr =untagged_addr (vaddr);
@@ -367,7 +367,7 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr, ret =0 ; } -up_read (&mm->mmap_sem); +mm_read_unlock (mm);return ret; }
xen/gntdev.c +2 -2
@@ -625,7 +625,7 @@ static long gntdev_ioctl_get_offset_for_vaddr(struct gntdev_priv *priv,return -EFAULT ;pr_debug ("priv %p, offset for vaddr %lx\n" , priv, (unsigned long )op.vaddr); -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); vma =find_vma (current->mm, op.vaddr);if (!vma || vma->vm_ops != &gntdev_vmops) goto out_unlock;
@@ -639,7 +639,7 @@ static long gntdev_ioctl_get_offset_for_vaddr(struct gntdev_priv *priv, rv =0 ; out_unlock: -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (rv ==0 &©_to_user (u, &op,sizeof (op)) !=0 )return -EFAULT ;
xen/privcmd.c +7 -7
@@ -278,7 +278,7 @@ static long privcmd_ioctl_mmap(struct file *file, void __user *udata)if (rc ||list_empty (&pagelist)) goto out; -down_write (&mm->mmap_sem); +mm_write_lock (mm); { struct page *page = list_first_entry(&pagelist,
@@ -303,7 +303,7 @@ static long privcmd_ioctl_mmap(struct file *file, void __user *udata) out_up: -up_write (&mm->mmap_sem); +mm_write_unlock (mm); out:free_page_list (&pagelist);
@@ -499,7 +499,7 @@ static long privcmd_ioctl_mmap_batch( } } -down_write (&mm->mmap_sem); +mm_write_lock (mm); vma =find_vma (mm, m.addr); if (!vma ||
@@ -555,7 +555,7 @@ static long privcmd_ioctl_mmap_batch(BUG_ON (traverse_pages_block (m.num,sizeof (xen_pfn_t), &pagelist, mmap_batch_fn, &state)); -up_write (&mm->mmap_sem); +mm_write_unlock (mm);if (state.global_error) {/* Write back errors in second pass. */
@@ -576,7 +576,7 @@ static long privcmd_ioctl_mmap_batch(return ret; out_unlock: -up_write (&mm->mmap_sem); +mm_write_unlock (mm); goto out; }
@@ -741,7 +741,7 @@ static long privcmd_ioctl_mmap_resource(struct file *file, void __user *udata)if (data->domid !=DOMID_INVALID && data->domid != kdata.dom)return -EPERM ; -down_write (&mm->mmap_sem); +mm_write_lock (mm); vma =find_vma (mm, kdata.addr);if (!vma || vma->vm_ops != &privcmd_vm_ops) {
@@ -820,7 +820,7 @@ static long privcmd_ioctl_mmap_resource(struct file *file, void __user *udata) } out: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);kfree (pfns);return rc;
aio.c +2 -2
@@ -519,7 +519,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events) ctx->mmap_size = nr_pages *PAGE_SIZE ;pr_debug ("attempting mmap of %lu bytes\n" , ctx->mmap_size); -if (down_write_killable (&mm->mmap_sem)) { +if (mm_write_lock_killable (mm)) { ctx->mmap_size =0 ;aio_free_ring (ctx);return -EINTR ;
@@ -528,7 +528,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events) ctx->mmap_base =do_mmap_pgoff (ctx->aio_ring_file,0 , ctx->mmap_size,PROT_READ |PROT_WRITE ,MAP_SHARED ,0 , &unused,NULL ); -up_write (&mm->mmap_sem); +mm_write_unlock (mm);if (IS_ERR ((void *)ctx->mmap_base)) { ctx->mmap_size =0 ;aio_free_ring (ctx);
coredump.c +2 -2
@@ -443,12 +443,12 @@ static int coredump_wait(int exit_code, struct core_state *core_state) core_state->dumper.task = tsk; core_state->dumper.next =NULL ; -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ;if (!mm->core_state) core_waiters =zap_threads (tsk, mm, core_state, exit_code); -up_write (&mm->mmap_sem); +mm_write_unlock (mm);if (core_waiters >0 ) {struct core_thread *ptr;
exec.c +8 -8
@@ -250,7 +250,7 @@ static int __bprm_mm_init(struct linux_binprm *bprm)return -ENOMEM ;vma_set_anonymous (vma); -if (down_write_killable (&mm->mmap_sem)) { +if (mm_write_lock_killable (mm)) { err = -EINTR ; goto err_free; }
@@ -273,11 +273,11 @@ static int __bprm_mm_init(struct linux_binprm *bprm) mm->stack_vm = mm->total_vm =1 ;arch_bprm_mm_init (mm, vma); -up_write (&mm->mmap_sem); +mm_write_unlock (mm); bprm->p = vma->vm_end -sizeof (void *);return 0 ; err: -up_write (&mm->mmap_sem); +mm_write_unlock (mm); err_free: bprm->vma =NULL ;vm_area_free (vma);
@@ -738,7 +738,7 @@ int setup_arg_pages(struct linux_binprm *bprm, bprm->loader -= stack_shift; bprm->exec -= stack_shift; -if (down_write_killable (&mm->mmap_sem)) +if (mm_write_lock_killable (mm))return -EINTR ; vm_flags =VM_STACK_FLAGS ;
@@ -795,7 +795,7 @@ int setup_arg_pages(struct linux_binprm *bprm, ret = -EFAULT ; out_unlock: -up_write (&mm->mmap_sem); +mm_write_unlock (mm);return ret; }EXPORT_SYMBOL (setup_arg_pages);
@@ -1024,9 +1024,9 @@ static int exec_mmap(struct mm_struct *mm)* through with the exec. We must hold mmap_sem around * checking core_state and changing tsk->mm. */ -down_read (&old_mm->mmap_sem); +mm_read_lock (old_mm);if (unlikely (old_mm->core_state)) { -up_read (&old_mm->mmap_sem); +mm_read_unlock (old_mm);return -EINTR ; } }
@@ -1040,7 +1040,7 @@ static int exec_mmap(struct mm_struct *mm)vmacache_flush (tsk);task_unlock (tsk);if (old_mm) { -up_read (&old_mm->mmap_sem); +mm_read_unlock (old_mm);BUG_ON (active_mm != old_mm);setmax_mm_hiwater_rss (&tsk->signal->maxrss, old_mm);mm_update_next_owner (old_mm);
io_uring.c +2 -2
@@ -4822,7 +4822,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, void __user *arg, } ret =0 ; -down_read (¤t->mm->mmap_sem); +mm_read_lock (current->mm); pret =get_user_pages (ubuf, nr_pages,FOLL_WRITE |FOLL_LONGTERM , pages, vmas);
@@ -4840,7 +4840,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, void __user *arg, }else { ret = pret <0 ? pret : -EFAULT ; } -up_read (¤t->mm->mmap_sem); +mm_read_unlock (current->mm);if (ret) {/* * if we did partial map, or found file backed vmas,
proc/base.c +6 -6
@@ -1979,11 +1979,11 @@ static int map_files_d_revalidate(struct dentry *dentry, unsigned int flags) goto out;if (!dname_to_vma_addr (dentry, &vm_start, &vm_end)) { - status =down_read_killable (&mm->mmap_sem); + status =mm_read_lock_killable (mm);if (!status) { exact_vma_exists = !!find_exact_vma (mm, vm_start, vm_end); -up_read (&mm->mmap_sem); +mm_read_unlock (mm); } }
@@ -2030,7 +2030,7 @@ static int map_files_get_link(struct dentry *dentry, struct path *path)if (rc) goto out_mmput; - rc =down_read_killable (&mm->mmap_sem); + rc =mm_read_lock_killable (mm);if (rc) goto out_mmput;
@@ -2041,7 +2041,7 @@ static int map_files_get_link(struct dentry *dentry, struct path *path)path_get (path); rc =0 ; } -up_read (&mm->mmap_sem); +mm_read_unlock (mm); out_mmput:mmput (mm);
@@ -2131,7 +2131,7 @@ static struct dentry *proc_map_files_lookup(struct inode *dir, goto out_put_task; result =ERR_PTR (-EINTR ); -if (down_read_killable (&mm->mmap_sem)) +if (mm_read_lock_killable (mm)) goto out_put_mm; result =ERR_PTR (-ENOENT );
@@ -2144,7 +2144,7 @@ static struct dentry *proc_map_files_lookup(struct inode *dir, (void *)(unsigned long)vma->vm_file->f_mode); out_no_vma: -up_read (&mm->mmap_sem); +mm_read_unlock (mm); out_put_mm:mmput (mm); out_put_task:
proc/task_mmu.c +14 -14
@@ -128,7 +128,7 @@ static void vma_stop(struct proc_maps_private *priv)struct mm_struct *mm = priv->mm;release_task_mempolicy (priv); -up_read (&mm->mmap_sem); +mm_read_unlock (mm);mmput (mm); }
@@ -166,7 +166,7 @@ static void *m_start(struct seq_file *m, loff_t *ppos)if (!mm || !mmget_not_zero (mm))return NULL ; -if (down_read_killable (&mm->mmap_sem)) { +if (mm_read_lock_killable (mm)) {mmput (mm);return ERR_PTR (-EINTR ); }
@@ -873,7 +873,7 @@ static int show_smaps_rollup(struct seq_file *m, void *v)memset (&mss,0 ,sizeof (mss)); - ret =down_read_killable (&mm->mmap_sem); + ret =mm_read_lock_killable (mm);if (ret) goto out_put_mm;
@@ -892,7 +892,7 @@ static int show_smaps_rollup(struct seq_file *m, void *v)__show_smap (m, &mss, true);release_task_mempolicy (priv); -up_read (&mm->mmap_sem); +mm_read_unlock (mm); out_put_mm:mmput (mm);
@@ -1166,7 +1166,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, };if (type ==CLEAR_REFS_MM_HIWATER_RSS ) { -if (down_write_killable (&mm->mmap_sem)) { +if (mm_write_lock_killable (mm)) { count = -EINTR ; goto out_mm; }
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.25.0.341.g760bfbb309-goog
Part of the diff is not shown here. View the raw message.
RFC PATCH 4/24 MM locking API: add range arguments
In reply to Michel LespinasseThis change extends the MM locking API to pass ranges.
The ranges will be used to implement range locking, but for now we only check that the passed ranges match between lock and unlock calls.
Add a new CONFIG_MM_LOCK_RWSEM_CHECKED config option to verify that ranges are correctly paired accross lock/unlock function calls.
To ensure an easy transition, the existing coarse MM locking calls are using a default range, which is represented by a per-task structure. This allows a task's paired coarse lock/unlock calls to be translated into correctly paired struct mm_lock_range locks and unlocks.
Add some small additional changes to kernel/fork.c (dup_mmap had a single task locking two MM's at once, so it has to explicitly manage the corresponding struct mm_lock_range) and kernel/bpf/stackmap.c (dumping user stacks from interrupt context requires explicit tracking of struct mm_lock_range).
--- arch/um/include/asm/mmu_context.h | 4 +- include/linux/mm_lock.h | 148 ++++++++++++++++++++++++++++-- include/linux/mm_types_task.h | 6 ++ include/linux/sched.h | 2 + init/init_task.c | 1 + kernel/bpf/stackmap.c | 26 ++++-- kernel/fork.c | 7 +- mm/Kconfig | 18 ++++ mm/Makefile | 1 + mm/mm_lock_rwsem_checked.c | 131 ++++++++++++++++++++++++++ 10 files changed, 322 insertions(+), 22 deletions(-) create mode 100644 mm/mm_lock_rwsem_checked.c
um/include/asm/mmu_context.h +3 -1
@@ -47,12 +47,14 @@ extern void force_flush_all(void);static inline void activate_mm (struct mm_struct *old,struct mm_struct *new) { +struct mm_lock_range mm_range =MM_COARSE_LOCK_RANGE_INITIALIZER ; +/* * This is called by fs/exec.c and sys_unshare() * when the new ->mm is used for the first time. */ __switch_mm (&new->context.id); -down_write_nested (&new->mmap_sem,1 ); +mm_write_range_lock_nested (new, &mm_range,1 );uml_setup_stubs (new);mm_write_unlock (new); }
linux/mm_lock.h +139 -9
195 lines not on this page · raw message
linux/mm_types_task.h +6 -0
@@ -96,4 +96,10 @@ struct tlbflush_unmap_batch { #endif }; +struct mm_lock_range { +#ifdef CONFIG_MM_LOCK_RWSEM_CHECKED +struct mm_struct *mm; +#endif +}; +#endif /* _LINUX_MM_TYPES_TASK_H */
linux/sched.h +2 -0
@@ -1281,6 +1281,8 @@ struct task_struct {unsigned long prev_lowest_stack;#endif +struct mm_lock_range mm_coarse_lock_range; +/* * New fields for task_struct should be added above here, so that * they are included in the randomized portion of task_struct.
init_task.c +1 -0
@@ -181,6 +181,7 @@ struct task_struct init_task#ifdef CONFIG_SECURITY .security = NULL, #endif + .mm_coarse_lock_range =MM_COARSE_LOCK_RANGE_INITIALIZER , };EXPORT_SYMBOL (init_task);
bpf/stackmap.c +16 -10
@@ -33,7 +33,8 @@ struct bpf_stack_map {/* irq_work to run up_read() for build_id lookup in nmi context */ struct stack_map_irq_work {struct irq_work irq_work; -struct rw_semaphore *sem; +struct mm_struct *mm; +struct mm_lock_range mm_range; };static void do_up_read (struct irq_work *entry)
@@ -41,8 +42,7 @@ static void do_up_read(struct irq_work *entry)struct stack_map_irq_work *work; work = container_of(entry, struct stack_map_irq_work, irq_work); -up_read_non_owner (work->sem); - work->sem =NULL ; +mm_read_range_unlock_non_owner (work->mm, &work->mm_range); } static DEFINE_PER_CPU(struct stack_map_irq_work, up_read_work);
@@ -286,12 +286,17 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,struct vm_area_struct *vma;bool irq_work_busy = false;struct stack_map_irq_work *work =NULL ; +struct mm_lock_range mm_range =MM_COARSE_LOCK_RANGE_INITIALIZER ; +struct mm_lock_range *mm_range_ptr = &mm_range;if (irqs_disabled ()) { work =this_cpu_ptr (&up_read_work); -if (atomic_read (&work->irq_work.flags) &IRQ_WORK_BUSY ) +if (atomic_read (&work->irq_work.flags) &IRQ_WORK_BUSY ) {/* cannot queue more up_read, fallback */ irq_work_busy = true; + }else { + mm_range_ptr = &work->mm_range; + } }/*
@@ -305,7 +310,7 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,* with build_id. */ if (!user || !current || !current->mm || irq_work_busy || -mm_read_trylock (current->mm) ==0 ) { + !mm_read_range_trylock (current->mm, mm_range_ptr)) {/* cannot access current->mm, fall back to ips */ for (i =0 ; i < trace_nr; i++) { id_offs[i].status =BPF_STACK_BUILD_ID_IP ;
@@ -330,16 +335,16 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }if (!work) { -mm_read_unlock (current->mm); +mm_read_range_unlock (current->mm, mm_range_ptr); }else { - work->sem = ¤t->mm->mmap_sem; + work->mm = current->mm;irq_work_queue (&work->irq_work);/* * The irq_work will release the mmap_sem with -* up_read_non_owner(). The rwsem_release() is called -* here to release the lock from lockdep's perspective. +* mm_read_range_unlock_non_owner(). mm_read_release() is +* called here to release the lock from lockdep's perspective. */ -rwsem_release (¤t->mm->mmap_sem.dep_map, _RET_IP_); +mm_read_release (current->mm, _RET_IP_); } }
@@ -626,6 +631,7 @@ static int __init stack_map_init(void)for_each_possible_cpu (cpu) { work =per_cpu_ptr (&up_read_work, cpu);init_irq_work (&work->irq_work, do_up_read); +mm_init_coarse_lock_range (&work->mm_range); }return 0 ; }
fork.c +5 -2
@@ -486,6 +486,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,int retval;unsigned long charge;LIST_HEAD (uf); +struct mm_lock_range mm_range =MM_COARSE_LOCK_RANGE_INITIALIZER ;uprobe_start_dup_mmap ();if (mm_write_lock_killable (oldmm)) {
@@ -497,7 +498,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,/* * Not linked in yet - no deadlock potential: */ -down_write_nested (&mm->mmap_sem,SINGLE_DEPTH_NESTING ); +mm_write_range_lock_nested (mm, &mm_range,SINGLE_DEPTH_NESTING );/* No ordering required: file already has been exposed. */ RCU_INIT_POINTER (mm->exe_file,get_mm_exe_file (oldmm));
@@ -612,7 +613,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,/* a new mm has just been created */ retval =arch_dup_mmap (oldmm, mm); out: -mm_write_unlock (mm); +mm_write_range_unlock (mm, &mm_range);flush_tlb_mm (oldmm);mm_write_unlock (oldmm);dup_userfaultfd_complete (&uf);
@@ -947,6 +948,8 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)#ifdef CONFIG_MEMCG tsk->active_memcg =NULL ;#endif + +mm_init_coarse_lock_range (&tsk->mm_coarse_lock_range);return tsk; free_stack:
Kconfig +18 -0
@@ -739,4 +739,22 @@ config ARCH_HAS_HUGEPD config MAPPING_DIRTY_HELPERS bool +choice + prompt "MM lock implementation (mmap_sem)" + default MM_LOCK_RWSEM_CHECKED + +config MM_LOCK_RWSEM_INLINE + bool "rwsem, inline" + help + This option preserves the traditional MM lock implementation as + inline read-write semaphone operations. + +config MM_LOCK_RWSEM_CHECKED + bool "rwsem, checked" + help + This option implements the MM lock using a read-write semaphore, + ignoring the passed address range but checking its validity. + +endchoice + endmenu
Makefile +1 -0
@@ -108,3 +108,4 @@ obj-$(CONFIG_ZONE_DEVICE) += memremap.o obj-$(CONFIG_HMM_MIRROR) += hmm.o obj-$(CONFIG_MEMFD_CREATE) += memfd.o obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o +obj-$(CONFIG_MM_LOCK_RWSEM_CHECKED) += mm_lock_rwsem_checked.o
mm_lock_rwsem_checked.c new file +131 -0
131 lines not on this page · raw message
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 5/24 MM locking API: allow for sleeping during unlock
In reply to Michel LespinasseFollowing changes will implement fine-grained range locks for the MM locking API, using a data structure to represent the existing locks, and a mutex to protect that data structure.
As a result, we need to prepare for the possibility that unlocking a memory range may need to sleep when acquiring that mutex.
--- kernel/bpf/stackmap.c | 6 +++++- kernel/exit.c | 2 +- mm/mm_lock_rwsem_checked.c | 3 +++ 3 files changed, 9 insertions(+), 2 deletions(-)
bpf/stackmap.c +5 -1
@@ -308,8 +308,12 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,* * Same fallback is used for kernel stack (!user) on a stackmap * with build_id. +* +* FIXME - currently disabling the build_id lookup feature +* as mm_read_range_unlock() may block, which is not always +* possible to do here. */ -if (!user || !current || !current->mm || irq_work_busy || +if (true || !user || !current || !current->mm || irq_work_busy || !mm_read_range_trylock (current->mm, mm_range_ptr)) {/* cannot access current->mm, fall back to ips */ for (i =0 ; i < trace_nr; i++) {
exit.c +1 -1
@@ -478,9 +478,9 @@ static void exit_mm(void)/* more a memory barrier than a real lock */ task_lock (current); current->mm =NULL ; -mm_read_unlock (mm);enter_lazy_tlb (mm, current);task_unlock (current); +mm_read_unlock (mm);mm_update_next_owner (mm);mmput (mm);if (test_thread_flag (TIF_MEMDIE ))
mm_lock_rwsem_checked.c +3 -0
@@ -61,6 +61,7 @@ EXPORT_SYMBOL(mm_write_range_trylock);void mm_write_range_unlock (struct mm_struct *mm,struct mm_lock_range *range) { +might_sleep ();if (range->mm != mm)mm_lock_dump ("mm_write_range_unlock" ); range->mm =NULL ;
@@ -113,6 +114,7 @@ EXPORT_SYMBOL(mm_read_range_trylock);void mm_read_range_unlock (struct mm_struct *mm,struct mm_lock_range *range) { +might_sleep ();if (range->mm != mm)mm_lock_dump ("mm_read_range_unlock" ); range->mm =NULL ;
@@ -123,6 +125,7 @@ EXPORT_SYMBOL(mm_read_range_unlock);void mm_read_range_unlock_non_owner (struct mm_struct *mm,struct mm_lock_range *range) { +might_sleep ();if (range->mm != mm)mm_lock_dump ("mm_read_range_unlock_non_owner" ); range->mm =NULL ;
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 6/24 MM locking API: implement fine grained range locks
In reply to Michel LespinasseThis change implements fine grained reader-writer range locks.
Existing locked ranges are represented as an augmented rbtree protected by a mutex. The locked ranges hold information about two overlapping interval trees, representing the reader and writer locks respectively. This data structure allows quickly searching for existing readers, writers, or both, intersecting a given address range.
When locking a range, a count of all existing conflicting ranges (either already locked, or queued) is added to mm_lock_range struct. If the count is non-zero, the locking task is put to sleep until all conflicting lock ranges are released.
When unlocking a range, the conflict count for all existing (queued) conflicting ranges is decremented. If the count reaches zero, the locker task is woken up - it now has a lock on its desired address range.
The general approach for this range locking implementation was first proposed by Jan Kara back in 2013, and later worked on by at least Laurent Dufour and Davidlohr Bueso. I have extended on the approach by using separate indexes for the reader and writer range locks.
--- arch/x86/kernel/tboot.c | 2 +- drivers/firmware/efi/efi.c | 2 +- include/linux/mm_lock.h | 96 ++++- include/linux/mm_types.h | 20 + include/linux/mm_types_task.h | 15 + mm/Kconfig | 9 +- mm/Makefile | 1 + mm/init-mm.c | 3 +- mm/mm_lock_range.c | 691 ++++++++++++++++++++++++++++++++++ 9 files changed, 827 insertions(+), 12 deletions(-) create mode 100644 mm/mm_lock_range.c
x86/kernel/tboot.c +1 -1
@@ -90,7 +90,7 @@ static struct mm_struct tboot_mm = { .pgd = swapper_pg_dir, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), - .mmap_sem = __RWSEM_INITIALIZER(init_mm.mmap_sem), + .mmap_sem = MM_LOCK_INITIALIZER(init_mm.mmap_sem), .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), };
firmware/efi/efi.c +1 -1
@@ -60,7 +60,7 @@ struct mm_struct efi_mm = { .mm_rb = RB_ROOT, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), - .mmap_sem = __RWSEM_INITIALIZER(efi_mm.mmap_sem), + .mmap_sem = MM_LOCK_INITIALIZER(efi_mm.mmap_sem), .page_table_lock = __SPIN_LOCK_UNLOCKED(efi_mm.page_table_lock), .mmlist = LIST_HEAD_INIT(efi_mm.mmlist), .cpu_bitmap = { [BITS_TO_LONGS(NR_CPUS)] = 0},
linux/mm_lock.h +88 -8
138 lines not on this page · raw message
linux/mm_types.h +20 -0
@@ -283,6 +283,21 @@ struct vm_userfaultfd_ctx {struct vm_userfaultfd_ctx {};#endif /* CONFIG_USERFAULTFD */ +/* +* struct mm_lock stores locked address ranges for a given mm, +* implementing a fine-grained replacement for the mmap_sem rwsem. +*/ +#ifdef CONFIG_MM_LOCK_RANGE +struct mm_lock { +struct mutex mutex; +struct rb_root rb_root; +unsigned long seq; +#ifdef CONFIG_DEBUG_LOCK_ALLOC +struct lockdep_map dep_map; +#endif +}; +#endif +/* * This struct defines a memory VMM memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory
@@ -426,7 +441,12 @@ struct mm_struct {spinlock_t page_table_lock;/* Protects page tables and some * counters */ + +#ifndef CONFIG_MM_LOCK_RANGE struct rw_semaphore mmap_sem; +#else +struct mm_lock mmap_sem; +#endif struct list_head mmlist;/* List of maybe swapped mm's. These * are globally strung together off
linux/mm_types_task.h +15 -0
@@ -12,6 +12,7 @@#include <linux/threads.h> #include <linux/atomic.h> #include <linux/cpumask.h> +#include <linux/rbtree.h> #include <asm/page.h>
@@ -100,6 +101,20 @@ struct mm_lock_range {#ifdef CONFIG_MM_LOCK_RWSEM_CHECKED struct mm_struct *mm;#endif +#ifdef CONFIG_MM_LOCK_RANGE +/* First cache line - used in insert / remove / iter */ +struct rb_node rb; +long flags_count; +unsigned long start;/* First address of the range. */ +unsigned long end;/* First address after the range. */ +struct { +unsigned long read_end;/* Largest end in reader nodes. */ +unsigned long write_end;/* Largest end in writer nodes. */ + } __subtree;/* Subtree augmented information. */ +/* Second cache line - used in wait and wake. */ +unsigned long seq;/* Killable wait sequence number. */ +struct task_struct *task;/* Task trying to lock this range. */ +#endif };#endif /* _LINUX_MM_TYPES_TASK_H */
Kconfig +8 -1
@@ -741,7 +741,7 @@ config MAPPING_DIRTY_HELPERS choice prompt "MM lock implementation (mmap_sem)" - default MM_LOCK_RWSEM_CHECKED + default MM_LOCK_RANGE config MM_LOCK_RWSEM_INLINE bool "rwsem, inline"
@@ -755,6 +755,13 @@ config MM_LOCK_RWSEM_CHECKED This option implements the MM lock using a read-write semaphore, ignoring the passed address range but checking its validity. +config MM_LOCK_RANGE + bool "range lock" + help + This option implements the MM lock as a read-write range lock, + thus avoiding false conflicts between operations that operate + on non-overlapping address ranges. + endchoice endmenu
Makefile +1 -0
@@ -109,3 +109,4 @@ obj-$(CONFIG_HMM_MIRROR) += hmm.o obj-$(CONFIG_MEMFD_CREATE) += memfd.o obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o obj-$(CONFIG_MM_LOCK_RWSEM_CHECKED) += mm_lock_rwsem_checked.o +obj-$(CONFIG_MM_LOCK_RANGE) += mm_lock_range.o
init-mm.c +2 -1
@@ -1,5 +1,6 @@// SPDX-License-Identifier: GPL-2.0 #include <linux/mm_types.h> +#include <linux/mm_lock.h> #include <linux/rbtree.h> #include <linux/rwsem.h> #include <linux/spinlock.h>
@@ -31,7 +32,7 @@ struct mm_struct init_mm = { .pgd = swapper_pg_dir, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), - .mmap_sem = __RWSEM_INITIALIZER(init_mm.mmap_sem), + .mmap_sem = MM_LOCK_INITIALIZER(init_mm.mmap_sem), .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist),
mm_lock_range.c new file +691 -0
691 lines not on this page · raw message
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 7/24 mm/memory: add range field to struct vm_fault
In reply to Michel LespinasseAdd a range field to struct vm_fault. This carries the range that was locked for the given fault.
Faults that release the mmap_sem should pass the specified range.
--- include/linux/mm.h | 1 + mm/hugetlb.c | 1 + mm/khugepaged.c | 1 + mm/memory.c | 1 + 4 files changed, 4 insertions(+)
linux/mm.h +1 -0
@@ -451,6 +451,7 @@ struct vm_fault {* page table to avoid allocation from * atomic context. */ +struct mm_lock_range *range;/* MM read lock range. */ };/* page entry size for vm->huge_fault() */
hugetlb.c +1 -0
@@ -3831,6 +3831,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, .vma = vma, .address = haddr, .flags = flags, + .range = mm_coarse_lock_range(),/* * Hard to debug if it ends up being * used by a callee that assumes
khugepaged.c +1 -0
@@ -900,6 +900,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm, .flags = FAULT_FLAG_ALLOW_RETRY, .pmd = pmd, .pgoff = linear_page_index(vma, address), + .range = mm_coarse_lock_range(), };/* we only decide to swapin, if there is enough young ptes */
memory.c +1 -0
@@ -4047,6 +4047,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, .flags = flags, .pgoff = linear_page_index(vma, address), .gfp_mask = __get_fault_gfp_mask(vma), + .range = mm_coarse_lock_range(), };unsigned int dirty = flags &FAULT_FLAG_WRITE ;struct mm_struct *mm = vma->vm_mm;
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 8/24 mm/memory: allow specifying MM lock range to handle_mm_fault()
In reply to Michel LespinasseThis change adds a new handle_mm_fault_range() function, which behaves like handle_mm_fault() but specifies an explicit MM lock range.
handle_mm_fault() remains as an inline wrapper which passes the default coarse locking range.
--- include/linux/hugetlb.h | 5 +++-- include/linux/mm.h | 11 +++++++++-- mm/hugetlb.c | 14 +++++++++----- mm/memory.c | 16 +++++++++------- 4 files changed, 30 insertions(+), 16 deletions(-)
linux/hugetlb.h +3 -2
@@ -88,7 +88,8 @@ int hugetlb_report_node_meminfo(int, char *);void hugetlb_show_meminfo (void );unsigned long hugetlb_total_pages (void );vm_fault_t hugetlb_fault (struct mm_struct *mm,struct vm_area_struct *vma, -unsigned long address,unsigned int flags); +unsigned long address,unsigned int flags, +struct mm_lock_range *range); int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, pte_t *dst_pte, struct vm_area_struct *dst_vma, unsigned long dst_addr,
@@ -307,7 +308,7 @@ static inline void __unmap_hugepage_range(struct mmu_gather *tlb,static inline vm_fault_t hugetlb_fault (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address, -unsigned int flags) +unsigned int flags,struct mm_lock_range *range) {BUG ();return 0 ;
linux/mm.h +9 -2
@@ -1460,8 +1460,15 @@ int generic_error_remove_page(struct address_space *mapping, struct page *page);int invalidate_inode_page (struct page *page);#ifdef CONFIG_MMU -extern vm_fault_t handle_mm_fault (struct vm_area_struct *vma, -unsigned long address,unsigned int flags); +extern vm_fault_t handle_mm_fault_range (struct vm_area_struct *vma, +unsigned long address,unsigned int flags, +struct mm_lock_range *range); +static inline vm_fault_t handle_mm_fault (struct vm_area_struct *vma, +unsigned long address,unsigned int flags) +{ +return handle_mm_fault_range (vma, address, flags, +mm_coarse_lock_range ()); +}extern int fixup_user_fault (struct task_struct *tsk,struct mm_struct *mm,unsigned long address,unsigned int fault_flags,bool *unlocked);
hugetlb.c +9 -5
@@ -3788,7 +3788,8 @@ int huge_add_to_page_cache(struct page *page, struct address_space *mapping,static vm_fault_t hugetlb_no_page (struct mm_struct *mm,struct vm_area_struct *vma,struct address_space *mapping,pgoff_t idx, -unsigned long address,pte_t *ptep,unsigned int flags) +unsigned long address,pte_t *ptep,unsigned int flags, +struct mm_lock_range *range) {struct hstate *h =hstate_vma (vma);vm_fault_t ret =VM_FAULT_SIGBUS ;
@@ -3831,7 +3832,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, .vma = vma, .address = haddr, .flags = flags, - .range = mm_coarse_lock_range(), + .range = range,/* * Hard to debug if it ends up being * used by a callee that assumes
@@ -3997,7 +3998,8 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx)#endif vm_fault_t hugetlb_fault (struct mm_struct *mm,struct vm_area_struct *vma, -unsigned long address,unsigned int flags) +unsigned long address,unsigned int flags, +struct mm_lock_range *range) {pte_t *ptep, entry;spinlock_t *ptl;
@@ -4039,7 +4041,8 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, entry =huge_ptep_get (ptep);if (huge_pte_none (entry)) { - ret =hugetlb_no_page (mm, vma, mapping, idx, address, ptep, flags); + ret =hugetlb_no_page (mm, vma, mapping, idx, address, ptep, + flags, range); goto out_mutex; }
@@ -4348,7 +4351,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, FAULT_FLAG_ALLOW_RETRY); fault_flags |=FAULT_FLAG_TRIED ; } - ret =hugetlb_fault (mm, vma, vaddr, fault_flags); + ret =hugetlb_fault (mm, vma, vaddr, fault_flags, +mm_coarse_lock_range ());if (ret &VM_FAULT_ERROR ) { err =vm_fault_to_errno (ret, flags); remainder =0 ;
memory.c +9 -7
@@ -4039,7 +4039,8 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)* return value. See filemap_fault() and __lock_page_or_retry(). */ static vm_fault_t __handle_mm_fault (struct vm_area_struct *vma, -unsigned long address,unsigned int flags) +unsigned long address,unsigned int flags, +struct mm_lock_range *range) {struct vm_fault vmf = { .vma = vma,
@@ -4047,7 +4048,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, .flags = flags, .pgoff = linear_page_index(vma, address), .gfp_mask = __get_fault_gfp_mask(vma), - .range = mm_coarse_lock_range(), + .range = range, };unsigned int dirty = flags &FAULT_FLAG_WRITE ;struct mm_struct *mm = vma->vm_mm;
@@ -4134,8 +4135,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,* The mmap_sem may have been released depending on flags and our * return value. See filemap_fault() and __lock_page_or_retry(). */ -vm_fault_t handle_mm_fault (struct vm_area_struct *vma,unsigned long address, -unsigned int flags) +vm_fault_t handle_mm_fault_range (struct vm_area_struct *vma, +unsigned long address,unsigned int flags, +struct mm_lock_range *range) {vm_fault_t ret;
@@ -4160,9 +4162,9 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,mem_cgroup_enter_user_fault ();if (unlikely (is_vm_hugetlb_page (vma))) - ret =hugetlb_fault (vma->vm_mm, vma, address, flags); + ret =hugetlb_fault (vma->vm_mm, vma, address, flags, range);else - ret =__handle_mm_fault (vma, address, flags); + ret =__handle_mm_fault (vma, address, flags, range);if (flags &FAULT_FLAG_USER ) {mem_cgroup_exit_user_fault ();
@@ -4178,7 +4180,7 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,return ret; } -EXPORT_SYMBOL_GPL (handle_mm_fault); +EXPORT_SYMBOL_GPL (handle_mm_fault_range);#ifndef __PAGETABLE_P4D_FOLDED/*
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 9/24 do_swap_page: use the vmf->range field when dropping mmap_sem
In reply to Michel LespinasseChange do_swap_page() and lock_page_or_retry() so that the proper range will be released when swapping in.
--- include/linux/pagemap.h | 7 ++++--- mm/filemap.c | 6 +++--- mm/memory.c | 2 +- 3 files changed, 8 insertions(+), 7 deletions(-)
linux/pagemap.h +4 -3
@@ -458,7 +458,7 @@ static inline pgoff_t linear_page_index(struct vm_area_struct *vma,extern void __lock_page (struct page *page);extern int __lock_page_killable (struct page *page);extern int __lock_page_or_retry (struct page *page,struct mm_struct *mm, -unsigned int flags); +unsigned int flags,struct mm_lock_range *range);extern void unlock_page (struct page *page);/*
@@ -501,10 +501,11 @@ static inline int lock_page_killable(struct page *page)* __lock_page_or_retry(). */ static inline int lock_page_or_retry (struct page *page,struct mm_struct *mm, -unsigned int flags) +unsigned int flags,struct mm_lock_range *range) {might_sleep (); -return trylock_page (page) ||__lock_page_or_retry (page, mm, flags); +return trylock_page (page) ||__lock_page_or_retry (page, mm, flags, + range); }/*
filemap.c +3 -3
@@ -1406,7 +1406,7 @@ EXPORT_SYMBOL_GPL(__lock_page_killable);* with the page locked and the mmap_sem unperturbed. */ int __lock_page_or_retry (struct page *page,struct mm_struct *mm, -unsigned int flags) +unsigned int flags,struct mm_lock_range *range) {if (flags &FAULT_FLAG_ALLOW_RETRY ) {/*
@@ -1416,7 +1416,7 @@ int __lock_page_or_retry(struct page *page, struct mm_struct *mm,if (flags &FAULT_FLAG_RETRY_NOWAIT )return 0 ; -mm_read_unlock (mm); +mm_read_range_unlock (mm, range);if (flags &FAULT_FLAG_KILLABLE )wait_on_page_locked_killable (page);else
@@ -1428,7 +1428,7 @@ int __lock_page_or_retry(struct page *page, struct mm_struct *mm, ret =__lock_page_killable (page);if (ret) { -mm_read_unlock (mm); +mm_read_range_unlock (mm, range);return 0 ; } }else
memory.c +1 -1
@@ -2964,7 +2964,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_release; } - locked =lock_page_or_retry (page, vma->vm_mm, vmf->flags); + locked =lock_page_or_retry (page, vma->vm_mm, vmf->flags, vmf->range);delayacct_clear_flag (DELAYACCT_PF_SWAPIN );if (!locked) {
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 10/24 handle_userfault: use the vmf->range field when dropping mmap_sem
In reply to Michel LespinasseChange handle_userfault to drop the proper memory range as indicated in the vmf.
--- fs/userfaultfd.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
userfaultfd.c +2 -2
@@ -489,7 +489,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) must_wait =userfaultfd_huge_must_wait (ctx, vmf->vma, vmf->address, vmf->flags, reason); -mm_read_unlock (mm); +mm_read_range_unlock (mm, vmf->range); if (likely(must_wait && !READ_ONCE(ctx->released) && (return_to_userland ? !signal_pending(current) :
@@ -543,7 +543,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)* and there's no need to retake the mmap_sem * in such case. */ -mm_read_lock (mm); +mm_read_range_lock (mm, vmf->range); ret =VM_FAULT_NOPAGE ; } }
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 11/24 x86 fault handler: merge bad_area() functions
In reply to Michel LespinasseThis merges the bad_area(), bad_area_access_error() and the underlying __bad_area() functions into one single unified function.
Passing a NULL vma triggers the prior bad_area() behavior, while passing a non-NULL vma triggers the prior bad_area_access_error() behavior.
The control flow is very similar in all cases, and we now release the mmap_sem read lock in one single place rather than 3.
Text size is reduced by 356 bytes here.
--- arch/x86/mm/fault.c | 54 ++++++++++++++++++++------------------------- 1 file changed, 24 insertions(+), 30 deletions(-)
x86/mm/fault.c +24 -30
@@ -919,26 +919,6 @@ bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code,__bad_area_nosemaphore (regs, error_code, address,0 ,SEGV_MAPERR ); } -static void -__bad_area (struct pt_regs *regs,unsigned long error_code, -unsigned long address,u32 pkey,int si_code) -{ -struct mm_struct *mm = current->mm; -/* -* Something tried to access memory that isn't in our memory map.. -* Fix it, but check if it's kernel or user first.. -*/ -mm_read_unlock (mm); - -__bad_area_nosemaphore (regs, error_code, address, pkey, si_code); -} - -static noinline void -bad_area (struct pt_regs *regs,unsigned long error_code,unsigned long address) -{ -__bad_area (regs, error_code, address,0 ,SEGV_MAPERR ); -} -static inline bool bad_area_access_from_pkeys (unsigned long error_code,struct vm_area_struct *vma) {
@@ -957,9 +937,15 @@ static inline bool bad_area_access_from_pkeys(unsigned long error_code, } static noinline void -bad_area_access_error (struct pt_regs *regs,unsigned long error_code, -unsigned long address,struct vm_area_struct *vma) +bad_area (struct pt_regs *regs,unsigned long error_code, +unsigned long address,struct vm_area_struct *vma) { +u32 pkey =0 ; +int si_code =SEGV_MAPERR ; + +if (!vma) + goto unlock; +/* * This OSPKE check is not strictly necessary at runtime. * But, doing it this way allows compiler optimizations
@@ -986,12 +972,20 @@ bad_area_access_error(struct pt_regs *regs, unsigned long error_code,* 6. T1 : reaches here, sees vma_pkey(vma)=5, when we really * faulted on a pte with its pkey=4. */ -u32 pkey =vma_pkey (vma); - -__bad_area (regs, error_code, address, pkey,SEGV_PKUERR ); + pkey =vma_pkey (vma); + si_code =SEGV_PKUERR ; }else { -__bad_area (regs, error_code, address,0 ,SEGV_ACCERR ); + si_code =SEGV_ACCERR ; } + +unlock: +/* +* Something tried to access memory that isn't in our memory map.. +* Fix it, but check if it's kernel or user first.. +*/ +mm_read_unlock (current->mm); + +__bad_area_nosemaphore (regs, error_code, address, pkey, si_code); } static void
@@ -1401,17 +1395,17 @@ void do_user_addr_fault(struct pt_regs *regs, vma =find_vma (mm, address);if (unlikely (!vma)) { -bad_area (regs, hw_error_code, address); +bad_area (regs, hw_error_code, address,NULL );return ; }if (likely (vma->vm_start <= address)) goto good_area;if (unlikely (!(vma->vm_flags &VM_GROWSDOWN ))) { -bad_area (regs, hw_error_code, address); +bad_area (regs, hw_error_code, address,NULL );return ; }if (unlikely (expand_stack (vma, address))) { -bad_area (regs, hw_error_code, address); +bad_area (regs, hw_error_code, address,NULL );return ; }
@@ -1421,7 +1415,7 @@ void do_user_addr_fault(struct pt_regs *regs,*/ good_area:if (unlikely (access_error (hw_error_code, vma))) { -bad_area_access_error (regs, hw_error_code, address, vma); +bad_area (regs, hw_error_code, address, vma);return ; }
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 12/24 x86 fault handler: use an explicit MM lock range
In reply to Michel LespinasseUse an explicit memory range throughthe fault handler and any called functions.
--- arch/x86/mm/fault.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-)
x86/mm/fault.c +14 -10
@@ -938,7 +938,8 @@ static inline bool bad_area_access_from_pkeys(unsigned long error_code, static noinline voidbad_area (struct pt_regs *regs,unsigned long error_code, -unsigned long address,struct vm_area_struct *vma) +unsigned long address,struct vm_area_struct *vma, +struct mm_lock_range *range) {u32 pkey =0 ;int si_code =SEGV_MAPERR ;
@@ -983,7 +984,7 @@ bad_area(struct pt_regs *regs, unsigned long error_code,* Something tried to access memory that isn't in our memory map.. * Fix it, but check if it's kernel or user first.. */ -mm_read_unlock (current->mm); +mm_read_range_unlock (current->mm, range);__bad_area_nosemaphore (regs, error_code, address, pkey, si_code); }
@@ -1277,6 +1278,7 @@ void do_user_addr_fault(struct pt_regs *regs, unsigned long hw_error_code, unsigned long address) { +struct mm_lock_range *range;struct vm_area_struct *vma;struct task_struct *tsk;struct mm_struct *mm;
@@ -1361,6 +1363,8 @@ void do_user_addr_fault(struct pt_regs *regs, }#endif + range =mm_coarse_lock_range (); +/* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception
@@ -1373,7 +1377,7 @@ void do_user_addr_fault(struct pt_regs *regs,* 1. Failed to acquire mmap_sem, and * 2. The access did not originate in userspace. */ -if (unlikely (!mm_read_trylock (mm))) { +if (unlikely (!mm_read_range_trylock (mm, range))) {if (!user_mode (regs) && !search_exception_tables (regs->ip)) {/* * Fault from code in kernel from
@@ -1383,7 +1387,7 @@ void do_user_addr_fault(struct pt_regs *regs,return ; } retry: -mm_read_lock (mm); +mm_read_range_lock (mm, range); }else {/* * The above down_read_trylock() might have succeeded in
@@ -1395,17 +1399,17 @@ void do_user_addr_fault(struct pt_regs *regs, vma =find_vma (mm, address);if (unlikely (!vma)) { -bad_area (regs, hw_error_code, address,NULL ); +bad_area (regs, hw_error_code, address,NULL , range);return ; }if (likely (vma->vm_start <= address)) goto good_area;if (unlikely (!(vma->vm_flags &VM_GROWSDOWN ))) { -bad_area (regs, hw_error_code, address,NULL ); +bad_area (regs, hw_error_code, address,NULL , range);return ; }if (unlikely (expand_stack (vma, address))) { -bad_area (regs, hw_error_code, address,NULL ); +bad_area (regs, hw_error_code, address,NULL , range);return ; }
@@ -1415,7 +1419,7 @@ void do_user_addr_fault(struct pt_regs *regs,*/ good_area:if (unlikely (access_error (hw_error_code, vma))) { -bad_area (regs, hw_error_code, address, vma); +bad_area (regs, hw_error_code, address, vma, range);return ; }
@@ -1432,7 +1436,7 @@ void do_user_addr_fault(struct pt_regs *regs,* userland). The return to userland is identified whenever * FAULT_FLAG_USER|FAULT_FLAG_KILLABLE are both set in flags. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault_range (vma, address, flags, range); major |= fault &VM_FAULT_MAJOR ;/*
@@ -1458,7 +1462,7 @@ void do_user_addr_fault(struct pt_regs *regs,return ; } -mm_read_unlock (mm); +mm_read_range_unlock (mm, range);if (unlikely (fault &VM_FAULT_ERROR )) {mm_fault_error (regs, hw_error_code, address, fault);return ;
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 13/24 mm/memory: add prepare_mm_fault() function
In reply to Michel LespinasseAdd a prepare_mm_fault() function, which may allocate an anon_vma if required for the incoming fault.
This is because the anon_vma must be allocated in the vma of record, while in the range locked case, the fault will operate on a pseudo-vma.
--- include/linux/mm.h | 14 ++++++++++++++ mm/memory.c | 26 ++++++++++++++++++++++++++ 2 files changed, 40 insertions(+)
linux/mm.h +14 -0
@@ -1460,6 +1460,15 @@ int generic_error_remove_page(struct address_space *mapping, struct page *page);int invalidate_inode_page (struct page *page);#ifdef CONFIG_MMU +extern vm_fault_t __prepare_mm_fault (struct vm_area_struct *vma, +unsigned int flags); +static inline vm_fault_t prepare_mm_fault (struct vm_area_struct *vma, +unsigned int flags) +{ +if (likely (vma->anon_vma)) +return 0 ; +return __prepare_mm_fault (vma, flags); +}extern vm_fault_t handle_mm_fault_range (struct vm_area_struct *vma,unsigned long address,unsigned int flags,struct mm_lock_range *range);
@@ -1477,6 +1486,11 @@ void unmap_mapping_pages(struct address_space *mapping,void unmap_mapping_range (struct address_space *mapping,loff_t const holebegin,loff_t const holelen,int even_cows);#else +static inline vm_fault_t prepare_mm_fault (struct vm_area_struct *vma, +unsigned int flags) +{ +return 0 ; +}static inline vm_fault_t handle_mm_fault (struct vm_area_struct *vma,unsigned long address,unsigned int flags) {
memory.c +26 -0
@@ -4129,6 +4129,32 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,return handle_pte_fault (&vmf); } +vm_fault_t __prepare_mm_fault (struct vm_area_struct *vma,unsigned int flags) +{ +vm_fault_t ret =0 ; + +if (vma_is_anonymous (vma) || + ((flags &FAULT_FLAG_WRITE ) && !(vma->vm_flags &VM_SHARED )) || + (is_vm_hugetlb_page (vma) && !(vma->vm_flags &VM_MAYSHARE ))) { +if (flags &FAULT_FLAG_USER ) +mem_cgroup_enter_user_fault (); +if (unlikely (__anon_vma_prepare (vma))) + ret =VM_FAULT_OOM ; +if (flags &FAULT_FLAG_USER ) { +mem_cgroup_exit_user_fault (); +/* +* The task may have entered a memcg OOM situation but +* if the allocation error was handled gracefully (no +* VM_FAULT_OOM), there is no need to kill anything. +* Just clean up the OOM state peacefully. +*/ +if (task_in_memcg_oom (current) && !(ret &VM_FAULT_OOM )) +mem_cgroup_oom_synchronize (false); + } + } +return ret; +} +/* * By the time we get here, we already hold the mm semaphore *
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 14/24 mm/swap_state: disable swap vma readahead
In reply to Michel LespinasseThis change disables swap vma readahead. This is because swap_ra_info() updates vma->swap_readahead_info, which is not feasible when operating on pseudo-vmas.
This is a crude temporary solution. It may be possible to use a per-mm swap_readahead_info instead, or if not, to explicitly fetch the vma of record when updating the swap readahead statistics.
--- mm/swap_state.c | 6 ++++++ 1 file changed, 6 insertions(+)
swap_state.c +6 -0
@@ -298,6 +298,12 @@ void free_pages_and_swap_cache(struct page **pages, int nr)static inline bool swap_use_vma_readahead (void ) { +/* +* vma readahead overwrites vma->swap_readahead_info, +* which requires some form of vma locking... +*/ +return false; +return READ_ONCE (enable_vma_readahead) && !atomic_read (&nr_rotate_swap); }
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 15/24 x86 fault handler: use a pseudo-vma when operating on anonymous vmas.
In reply to Michel LespinasseUpdate the fault handler to use a pseudo-vma when the original vma is anonymous. This is in preparation to handling such faults with a fine grained range lock in a later change.
--- arch/x86/mm/fault.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
x86/mm/fault.c +19 -1
@@ -1279,7 +1279,7 @@ void do_user_addr_fault(struct pt_regs *regs, unsigned long address) {struct mm_lock_range *range; -struct vm_area_struct *vma; +struct vm_area_struct pvma, *vma;struct task_struct *tsk;struct mm_struct *mm;vm_fault_t fault, major =0 ;
@@ -1423,6 +1423,23 @@ void do_user_addr_fault(struct pt_regs *regs,return ; } +if (vma_is_anonymous (vma)) { +/* +* Allocate anon_vma if needed. +* This needs to operate on the vma of record. +*/ + fault =prepare_mm_fault (vma, flags); +if (fault) + goto got_fault; + +/* +* Copy vma attributes into a pseudo-vma. +* This will be required when using fine grained locks. +*/ + pvma = *vma; + vma = &pvma; + } +/* * If for any reason at all we couldn't handle the fault, * make sure we exit gracefully rather than endlessly redo
@@ -1437,6 +1454,7 @@ void do_user_addr_fault(struct pt_regs *regs,* FAULT_FLAG_USER|FAULT_FLAG_KILLABLE are both set in flags. */ fault =handle_mm_fault_range (vma, address, flags, range); +got_fault: major |= fault &VM_FAULT_MAJOR ;/*
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 16/24 MM locking API: add vma locking API
In reply to Michel LespinasseThis change adds the mm_vma_lock() and mm_vma_unlock() functions, which are to be used to protect per-mm global structures (such as the vma rbtree) when writers only hold a range lock.
The functions are no-ops when CONFIG_MM_LOCK_RANGE is not enabled, as mmap_sem already protects such structures in that case.
--- include/linux/mm_lock.h | 24 ++++++++++++++++++++---- include/linux/mm_types.h | 2 ++ 2 files changed, 22 insertions(+), 4 deletions(-)
linux/mm_lock.h +20 -4
@@ -14,6 +14,9 @@ static inline void mm_init_lock(struct mm_struct *mm)init_rwsem (&mm->mmap_sem); } +static inline void mm_vma_lock (struct mm_struct *mm) {} +static inline void mm_vma_unlock (struct mm_struct *mm) {} +static inline void mm_init_coarse_lock_range (struct mm_lock_range *range) {}static inline void mm_init_lock_range (struct mm_lock_range *range,unsigned long start,unsigned long end) {}
@@ -107,6 +110,9 @@ static inline void mm_init_lock(struct mm_struct *mm)init_rwsem (&mm->mmap_sem); } +static inline void mm_vma_lock (struct mm_struct *mm) {} +static inline void mm_vma_unlock (struct mm_struct *mm) {} +static inline void mm_init_coarse_lock_range (struct mm_lock_range *range) { range->mm =NULL ;
@@ -131,10 +137,11 @@ static inline bool mm_range_is_coarse(struct mm_lock_range *range)#define __DEP_MAP_MM_LOCK_INITIALIZER (lockname)#endif -#define MM_LOCK_INITIALIZER (name) { \ - .mutex = __MUTEX_INITIALIZER(name.mutex), \ - .rb_root = RB_ROOT, \ - __DEP_MAP_MM_LOCK_INITIALIZER(name) \ +#define MM_LOCK_INITIALIZER (name) { \ + .mutex = __MUTEX_INITIALIZER(name.mutex), \ + .rb_root = RB_ROOT, \ + .vma_mutex = __MUTEX_INITIALIZER(name.vma_mutex), \ + __DEP_MAP_MM_LOCK_INITIALIZER(name) \ }#define MM_COARSE_LOCK_RANGE_INITIALIZER { \
@@ -148,9 +155,18 @@ static inline void mm_init_lock(struct mm_struct *mm)mutex_init (&mm->mmap_sem.mutex); mm->mmap_sem.rb_root =RB_ROOT ; +mutex_init (&mm->mmap_sem.vma_mutex);lockdep_init_map (&mm->mmap_sem.dep_map,"&mm->mmap_sem" , &__key,0 ); } +static inline void mm_vma_lock (struct mm_struct *mm) { +mutex_lock (&mm->mmap_sem.vma_mutex); +} + +static inline void mm_vma_unlock (struct mm_struct *mm) { +mutex_unlock (&mm->mmap_sem.vma_mutex); +} +static inline void mm_init_lock_range (struct mm_lock_range *range,unsigned long start,unsigned long end) { range->start = start;
linux/mm_types.h +2 -0
@@ -292,9 +292,11 @@ struct mm_lock {struct mutex mutex;struct rb_root rb_root;unsigned long seq; +struct mutex vma_mutex;#ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map; #endif + };#endif
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 17/24 x86 fault handler: implement range locking
In reply to Michel LespinasseChange the x86 fault handler to implement range locking.
Initially we try to lock a pmd sized range around the faulting address, which is appropriate for anon vmas. After finding the correct vma for the faulting address, we verify that it is anonymous and fall back to a coarse grained lock if necessary. If a fine grained lock is workable, we copy the vma of record into a pseudo-vma and release the mm_vma_lock before handling the fault.
--- arch/x86/mm/fault.c | 40 ++++++++++++++++++++++++++++++++-------- 1 file changed, 32 insertions(+), 8 deletions(-)
x86/mm/fault.c +32 -8
@@ -941,6 +941,7 @@ bad_area(struct pt_regs *regs, unsigned long error_code, unsigned long address, struct vm_area_struct *vma, struct mm_lock_range *range) { + struct mm_struct *mm;u32 pkey =0 ;int si_code =SEGV_MAPERR ;
@@ -984,7 +985,10 @@ bad_area(struct pt_regs *regs, unsigned long error_code,* Something tried to access memory that isn't in our memory map.. * Fix it, but check if it's kernel or user first.. */ -mm_read_range_unlock (current->mm, range); + mm = current->mm; +if (!mm_range_is_coarse (range)) +mm_vma_unlock (mm); +mm_read_range_unlock (mm, range);__bad_area_nosemaphore (regs, error_code, address, pkey, si_code); }
@@ -1278,7 +1282,7 @@ void do_user_addr_fault(struct pt_regs *regs, unsigned long hw_error_code, unsigned long address) { -struct mm_lock_range *range; +struct mm_lock_range pmd_range, *range;struct vm_area_struct pvma, *vma;struct task_struct *tsk;struct mm_struct *mm;
@@ -1363,7 +1367,10 @@ void do_user_addr_fault(struct pt_regs *regs, }#endif - range =mm_coarse_lock_range (); +mm_init_lock_range (&pmd_range, + address &PMD_MASK , + (address &PMD_MASK ) +PMD_SIZE ); + range = &pmd_range;/* * Kernel-mode access to the user address space should only occur
@@ -1397,6 +1404,8 @@ void do_user_addr_fault(struct pt_regs *regs,might_sleep (); } +if (!mm_range_is_coarse (range)) +mm_vma_lock (mm); vma =find_vma (mm, address);if (unlikely (!vma)) {bad_area (regs, hw_error_code, address,NULL , range);
@@ -1408,6 +1417,10 @@ void do_user_addr_fault(struct pt_regs *regs,bad_area (regs, hw_error_code, address,NULL , range);return ; } +/* +* Note that if range is fine grained, we can still safely call +* expand_stack as we are protected by the mm_vma_lock(). +*/ if (unlikely (expand_stack (vma, address))) {bad_area (regs, hw_error_code, address,NULL , range);return ;
@@ -1423,23 +1436,34 @@ void do_user_addr_fault(struct pt_regs *regs,return ; } -if (vma_is_anonymous (vma)) { +if (!mm_range_is_coarse (range)) {/* * Allocate anon_vma if needed. * This needs to operate on the vma of record. */ fault =prepare_mm_fault (vma, flags); -if (fault) - goto got_fault;/* * Copy vma attributes into a pseudo-vma. -* This will be required when using fine grained locks. +* The vma of record is only valid until mm_vma_unlock(). */ pvma = *vma; vma = &pvma; - } +mm_vma_unlock (mm); +if (fault) + goto got_fault; + +/* +* Fall back to locking the entire MM +* when operating on file vma. +*/ +if (!vma_is_anonymous (vma)) { +mm_read_range_unlock (mm, range); + range =mm_coarse_lock_range (); + goto retry; + } + }/* * If for any reason at all we couldn't handle the fault, * make sure we exit gracefully rather than endlessly redo
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 18/24 shared file mappings: use the vmf->range field when dropping mmap_sem
In reply to Michel LespinasseModify lock_page_maybe_drop_mmap() and maybe_unlock_mmap_for_io() to use the vmf->range field when dropping mmap_sem.
This covers dropping mmap_sem during:
- filemap_fault()
- shmem_fault()
- do_fault() write to shared file mapping
[ through do_shared_fault and fault_dirty_shared_page() ]
- do_wp_page() write to shared file mapping
[ through wp_page_shared() and fault_dirty_shared_page() ]
--- mm/filemap.c | 3 ++- mm/internal.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
filemap.c +2 -1
@@ -2364,7 +2364,8 @@ static int lock_page_maybe_drop_mmap(struct vm_fault *vmf, struct page *page,* mmap_sem here and return 0 if we don't have a fpin. */ if (*fpin ==NULL ) -mm_read_unlock (vmf->vma->vm_mm); +mm_read_range_unlock (vmf->vma->vm_mm, + vmf->range);return 0 ; } }else
internal.h +1 -1
@@ -382,7 +382,7 @@ static inline struct file *maybe_unlock_mmap_for_io(struct vm_fault *vmf,if ((flags & (FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_RETRY_NOWAIT )) ==FAULT_FLAG_ALLOW_RETRY ) { fpin =get_file (vmf->vma->vm_file); -mm_read_unlock (vmf->vma->vm_mm); +mm_read_range_unlock (vmf->vma->vm_mm, vmf->range); }return fpin; }
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 19/24 mm: add field to annotate vm_operations that support range locking
In reply to Michel LespinasseAdd a fine_grained field to struct vm_operations_struct,
and set it in the filesystems we have converted to support range locking.
--- fs/ext4/file.c | 1 + include/linux/mm.h | 16 ++++++++++++++++ mm/filemap.c | 1 + mm/shmem.c | 1 + 4 files changed, 19 insertions(+)
ext4/file.c +1 -0
@@ -626,6 +626,7 @@ static const struct vm_operations_struct ext4_file_vm_ops = { .fault = ext4_filemap_fault, .map_pages = filemap_map_pages, .page_mkwrite = ext4_page_mkwrite, + .fine_grained = true, };static int ext4_file_mmap (struct file *file,struct vm_area_struct *vma)
linux/mm.h +16 -0
@@ -526,6 +526,22 @@ struct vm_operations_struct {*/ struct page *(*find_special_page)(struct vm_area_struct *vma,unsigned long addr); + +/* +* fine_grained indicates that the vm_operations support +* fine grained mm locking. +* - The methods may be called with a fine grained range lock +* covering a PMD sized region around the fault address; +* - The range lock does not protect against concurrent access +* to per-mmmm structures, so an appropriate lock must be used +* for such cases +* (such as mm_vma_lock() for accessing the vma rbtree); +* - if dropping mmap_sem, the vmf->range must be used +* to release the specific locked range only; +* - vmf->vma only holds a copy of the original vma. +* Any persistent vma updates must first look up the actual vma. +*/ +bool fine_grained; };static inline void vma_init (struct vm_area_struct *vma,struct mm_struct *mm)
filemap.c +1 -0
@@ -2699,6 +2699,7 @@ const struct vm_operations_struct generic_file_vm_ops = { .fault = filemap_fault, .map_pages = filemap_map_pages, .page_mkwrite = filemap_page_mkwrite, + .fine_grained = true, };/* This is used for a general mmap of a disk file */
shmem.c +1 -0
@@ -3865,6 +3865,7 @@ static const struct vm_operations_struct shmem_vm_ops = { .set_policy = shmem_set_policy, .get_policy = shmem_get_policy, #endif + .fine_grained = true, };int shmem_init_fs_context (struct fs_context *fc)
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 20/24 x86 fault handler: extend range locking to supported file vmas
In reply to Michel LespinasseChange the fault handler to operate with a fine grained range lock when operating on any of the explicitly supported file types.
--- arch/x86/mm/fault.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
x86/mm/fault.c +2 -2
@@ -1456,9 +1456,9 @@ void do_user_addr_fault(struct pt_regs *regs,/* * Fall back to locking the entire MM -* when operating on file vma. +* when the vm_ops do not support fine grained range locking. */ -if (!vma_is_anonymous (vma)) { +if (!vma_is_anonymous (vma) && !vma->vm_ops->fine_grained) {mm_read_range_unlock (mm, range); range =mm_coarse_lock_range (); goto retry;
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 21/24 do_mmap: add locked argument
In reply to Michel LespinasseChange the do_mmap() prototype to add a "locked" boolean argument. For now all call sites set it to true.
Also remove the do_mmap_pgoff() API, which was just wrapping do_mmap() with a forced vm_flags == 0 argument.
--- arch/x86/mm/mpx.c | 3 ++- fs/aio.c | 6 +++--- include/linux/mm.h | 13 ++----------- ipc/shm.c | 3 ++- mm/mmap.c | 8 ++++---- mm/nommu.c | 1 + mm/util.c | 4 ++-- 7 files changed, 16 insertions(+), 22 deletions(-)
x86/mm/mpx.c +2 -1
@@ -54,7 +54,8 @@ static unsigned long mpx_mmap(unsigned long len)mm_write_lock (mm); addr =do_mmap (NULL ,0 , len,PROT_READ |PROT_WRITE , -MAP_ANONYMOUS |MAP_PRIVATE ,VM_MPX ,0 , &populate,NULL ); +MAP_ANONYMOUS |MAP_PRIVATE ,VM_MPX ,0 , + true, &populate,NULL );mm_write_unlock (mm);if (populate)mm_populate (addr, populate);
aio.c +3 -3
@@ -525,9 +525,9 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events)return -EINTR ; } - ctx->mmap_base =do_mmap_pgoff (ctx->aio_ring_file,0 , ctx->mmap_size, -PROT_READ |PROT_WRITE , -MAP_SHARED ,0 , &unused,NULL ); + ctx->mmap_base =do_mmap (ctx->aio_ring_file,0 , ctx->mmap_size, +PROT_READ |PROT_WRITE , +MAP_SHARED ,0 ,0 , true, &unused,NULL );mm_write_unlock (mm);if (IS_ERR ((void *)ctx->mmap_base)) { ctx->mmap_size =0 ;
linux/mm.h +2 -11
@@ -2361,22 +2361,13 @@ extern unsigned long mmap_region(struct file *file, unsigned long addr, struct list_head *uf);extern unsigned long do_mmap (struct file *file,unsigned long addr,unsigned long len,unsigned long prot,unsigned long flags, -vm_flags_t vm_flags,unsigned long pgoff,unsigned long *populate, -struct list_head *uf); +vm_flags_t vm_flags,unsigned long pgoff,bool locked, +unsigned long *populate,struct list_head *uf);extern int __do_munmap (struct mm_struct *,unsigned long ,size_t ,struct list_head *uf,bool downgrade);extern int do_munmap (struct mm_struct *,unsigned long ,size_t ,struct list_head *uf); -static inline unsigned long -do_mmap_pgoff (struct file *file,unsigned long addr, -unsigned long len,unsigned long prot,unsigned long flags, -unsigned long pgoff,unsigned long *populate, -struct list_head *uf) -{ -return do_mmap (file, addr, len, prot, flags,0 , pgoff, populate, uf); -} -#ifdef CONFIG_MMU extern int __mm_populate (unsigned long addr,unsigned long len,int ignore_errors);
shm.c +2 -1
@@ -1558,7 +1558,8 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg, goto invalid; } - addr =do_mmap_pgoff (file, addr, size, prot, flags,0 , &populate,NULL ); + addr =do_mmap (file, addr, size, prot, flags,0 ,0 , + true, &populate,NULL ); *raddr = addr; err =0 ;if (IS_ERR_VALUE (addr))
mmap.c +4 -4
@@ -1369,8 +1369,8 @@ static inline bool file_mmap_ok(struct file *file, struct inode *inode,unsigned long do_mmap (struct file *file,unsigned long addr,unsigned long len,unsigned long prot,unsigned long flags,vm_flags_t vm_flags, -unsigned long pgoff,unsigned long *populate, -struct list_head *uf) +unsigned long pgoff,bool locked, +unsigned long *populate,struct list_head *uf) {struct mm_struct *mm = current->mm;int pkey =0 ;
@@ -2954,8 +2954,8 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size, } file =get_file (vma->vm_file); - ret =do_mmap_pgoff (vma->vm_file, start, size, - prot, flags, pgoff, &populate,NULL ); + ret =do_mmap (vma->vm_file, start, size, + prot, flags,0 , pgoff, true, &populate,NULL );fput (file); out:mm_write_unlock (mm);
nommu.c +1 -0
@@ -1102,6 +1102,7 @@ unsigned long do_mmap(struct file *file,unsigned long prot, unsigned long flags, vm_flags_t vm_flags, + bool locked, unsigned long pgoff, unsigned long *populate, struct list_head *uf)
util.c +2 -2
@@ -503,8 +503,8 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,if (!ret) {if (mm_write_lock_killable (mm))return -EINTR ; - ret =do_mmap_pgoff (file, addr, len, prot, flag, pgoff, - &populate, &uf); + ret =do_mmap (file, addr, len, prot, flag,0 , pgoff, + true, &populate, &uf);mm_write_unlock (mm);userfaultfd_unmap_complete (mm, &uf);if (populate)
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 22/24 do_mmap: implement locked argument
In reply to Michel LespinasseWhen locked is true, preserve the current behavior - the do_mmap() caller is expected to already hold a coarse write lock on current->mmap_sem
When locked is false, change do_mmap() to acquiring the appropriate MM locks. do_mmap() still acquires a coarse lock in this change, but can now be locally changed to acquire a fine grained lock in the future.
--- mm/mmap.c | 106 ++++++++++++++++++++++++++++++++++++----------------- mm/nommu.c | 19 +++++++++- 2 files changed, 89 insertions(+), 36 deletions(-)
mmap.c +72 -34
@@ -1406,22 +1406,29 @@ unsigned long do_mmap(struct file *file, unsigned long addr,if ((pgoff + (len >>PAGE_SHIFT )) < pgoff)return -EOVERFLOW ; +if (!locked &&mm_write_lock_killable (mm)) +return -EINTR ; +/* Too many mappings? */ -if (mm->map_count > sysctl_max_map_count) -return -ENOMEM ; +if (mm->map_count > sysctl_max_map_count) { + addr = -ENOMEM ; + goto unlock; + }/* Obtain the address to map to. we verify (or select) it and ensure * that it represents a valid section of the address space. */ addr =get_unmapped_area (file, addr, len, pgoff, flags);if (IS_ERR_VALUE (addr)) -return addr; + goto unlock;if (flags &MAP_FIXED_NOREPLACE ) {struct vm_area_struct *vma =find_vma (mm, addr); -if (vma && vma->vm_start < addr + len) -return -EEXIST ; +if (vma && vma->vm_start < addr + len) { + addr = -EEXIST ; + goto unlock; + } }if (prot ==PROT_EXEC ) {
@@ -1437,19 +1444,24 @@ unsigned long do_mmap(struct file *file, unsigned long addr, vm_flags |=calc_vm_prot_bits (prot, pkey) |calc_vm_flag_bits (flags) | mm->def_flags |VM_MAYREAD |VM_MAYWRITE |VM_MAYEXEC ; -if (flags &MAP_LOCKED ) -if (!can_do_mlock ()) -return -EPERM ; +if ((flags &MAP_LOCKED ) && !can_do_mlock ()) { + addr = -EPERM ; + goto unlock; + } -if (mlock_future_check (mm, vm_flags, len)) -return -EAGAIN ; +if (mlock_future_check (mm, vm_flags, len)) { + addr = -EAGAIN ; + goto unlock; + }if (file) {struct inode *inode =file_inode (file);unsigned long flags_mask; -if (!file_mmap_ok (file, inode, pgoff, len)) -return -EOVERFLOW ; +if (!file_mmap_ok (file, inode, pgoff, len)) { + addr = -EOVERFLOW ; + goto unlock; + } flags_mask =LEGACY_MAP_MASK | file->f_op->mmap_supported_flags;
@@ -1465,27 +1477,37 @@ unsigned long do_mmap(struct file *file, unsigned long addr, flags &=LEGACY_MAP_MASK ;/* fall through */ case MAP_SHARED_VALIDATE : -if (flags & ~flags_mask) -return -EOPNOTSUPP ; +if (flags & ~flags_mask) { + addr = -EOPNOTSUPP ; + goto unlock; + }if (prot &PROT_WRITE ) { -if (!(file->f_mode &FMODE_WRITE )) -return -EACCES ; -if (IS_SWAPFILE (file->f_mapping->host)) -return -ETXTBSY ; +if (!(file->f_mode &FMODE_WRITE )) { + addr = -EACCES ; + goto unlock; + } +if (IS_SWAPFILE (file->f_mapping->host)) { + addr = -ETXTBSY ; + goto unlock; + } }/* * Make sure we don't allow writing to an append-only * file.. */ -if (IS_APPEND (inode) && (file->f_mode &FMODE_WRITE )) -return -EACCES ; +if (IS_APPEND (inode) && (file->f_mode &FMODE_WRITE )) { + addr = -EACCES ; + goto unlock; + }/* * Make sure there are no mandatory locks on the file. */ -if (locks_verify_locked (file)) -return -EAGAIN ; +if (locks_verify_locked (file)) { + addr = -EAGAIN ; + goto unlock; + } vm_flags |=VM_SHARED |VM_MAYSHARE ;if (!(file->f_mode &FMODE_WRITE ))
@@ -1493,28 +1515,39 @@ unsigned long do_mmap(struct file *file, unsigned long addr,/* fall through */ case MAP_PRIVATE : -if (!(file->f_mode &FMODE_READ )) -return -EACCES ; +if (!(file->f_mode &FMODE_READ )) { + addr = -EACCES ; + goto unlock; + }if (path_noexec (&file->f_path)) { -if (vm_flags &VM_EXEC ) -return -EPERM ; +if (vm_flags &VM_EXEC ) { + addr = -EPERM ; + goto unlock; + } vm_flags &= ~VM_MAYEXEC ; } -if (!file->f_op->mmap) -return -ENODEV ; -if (vm_flags & (VM_GROWSDOWN |VM_GROWSUP )) -return -EINVAL ; +if (!file->f_op->mmap) { + addr = -ENODEV ; + goto unlock; + } +if (vm_flags & (VM_GROWSDOWN |VM_GROWSUP )) { + addr = -EINVAL ; + goto unlock; + }break ;default : -return -EINVAL ; + addr = -EINVAL ; + goto unlock; } }else {switch (flags &MAP_TYPE ) {case MAP_SHARED : -if (vm_flags & (VM_GROWSDOWN |VM_GROWSUP )) -return -EINVAL ; +if (vm_flags & (VM_GROWSDOWN |VM_GROWSUP )) { + addr = -EINVAL ; + goto unlock; + }/* * Ignore pgoff. */
@@ -1528,7 +1561,8 @@ unsigned long do_mmap(struct file *file, unsigned long addr, pgoff = addr >>PAGE_SHIFT ;break ;default : -return -EINVAL ; + addr = -EINVAL ; + goto unlock; } }
@@ -1551,6 +1585,10 @@ unsigned long do_mmap(struct file *file, unsigned long addr, ((vm_flags &VM_LOCKED ) || (flags & (MAP_POPULATE | MAP_NONBLOCK)) == MAP_POPULATE)) *populate = len; + +unlock: +if (!locked) +mm_write_unlock (mm);return addr; }
nommu.c +17 -2
@@ -1107,6 +1107,7 @@ unsigned long do_mmap(struct file *file,unsigned long *populate, struct list_head *uf) { + struct mm_struct *mm = current->mm;struct vm_area_struct *vma;struct vm_region *region;struct rb_node *rb;
@@ -1115,12 +1116,18 @@ unsigned long do_mmap(struct file *file, *populate =0 ; +if (!locked &&mm_write_lock_killable (mm)) +return -EINTR ; +/* decide whether we should attempt the mapping, and if so what sort of * mapping */ ret =validate_mmap_request (file, addr, len, prot, flags, pgoff, &capabilities); -if (ret <0 ) +if (ret <0 ) { +if (!locked) +mm_write_unlock (mm);return ret; + }/* we ignore the address hint */ addr =0 ;
@@ -1135,7 +1142,7 @@ unsigned long do_mmap(struct file *file,if (!region) goto error_getting_region; - vma =vm_area_alloc (current->mm); + vma =vm_area_alloc (mm);if (!vma) goto error_getting_vma;
@@ -1289,6 +1296,8 @@ unsigned long do_mmap(struct file *file, }up_write (&nommu_region_sem); +if (!locked) +mm_write_unlock (mm);return result;
@@ -1301,6 +1310,8 @@ unsigned long do_mmap(struct file *file,if (vma->vm_file)fput (vma->vm_file);vm_area_free (vma); +if (!locked) +mm_write_unlock (mm);return ret; sharing_violation:
@@ -1314,12 +1325,16 @@ unsigned long do_mmap(struct file *file,pr_warn ("Allocation of vma for %lu byte allocation from process %d failed\n" , len, current->pid);show_free_areas (0 ,NULL ); +if (!locked) +mm_write_unlock (mm);return -ENOMEM ; error_getting_region:pr_warn ("Allocation of vm region for %lu byte allocation from process %d failed\n" , len, current->pid);show_free_areas (0 ,NULL ); +if (!locked) +mm_write_unlock (mm);return -ENOMEM ; }
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 23/24 do_mmap: use locked=false in vm_mmap_pgoff() and aio_setup_ring()
In reply to Michel LespinasseChange vm_mmap_pgoff() and aio_setup_ring() to call do_mmap() with locked=false.
Moving the mmap_sem acquisition to within do_mmap() enables it to acquire a fine grained lock in the future.
--- fs/aio.c | 12 ++---------- mm/util.c | 8 +++----- 2 files changed, 5 insertions(+), 15 deletions(-)
aio.c +2 -10
@@ -460,7 +460,6 @@ static const struct address_space_operations aio_ctx_aops = {static int aio_setup_ring (struct kioctx *ctx,unsigned int nr_events) {struct aio_ring *ring; -struct mm_struct *mm = current->mm;unsigned long size, unused;int nr_pages;int i;
@@ -519,20 +518,13 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events) ctx->mmap_size = nr_pages *PAGE_SIZE ;pr_debug ("attempting mmap of %lu bytes\n" , ctx->mmap_size); -if (mm_write_lock_killable (mm)) { - ctx->mmap_size =0 ; -aio_free_ring (ctx); -return -EINTR ; - } - ctx->mmap_base =do_mmap (ctx->aio_ring_file,0 , ctx->mmap_size,PROT_READ |PROT_WRITE , -MAP_SHARED ,0 ,0 , true, &unused,NULL ); -mm_write_unlock (mm); +MAP_SHARED ,0 ,0 , false, &unused,NULL );if (IS_ERR ((void *)ctx->mmap_base)) { ctx->mmap_size =0 ;aio_free_ring (ctx); -return -ENOMEM ; +return (ctx->mmap_base == -EINTR ) ? -EINTR : -ENOMEM ; }pr_debug ("mmap address: 0x%08lx\n" , ctx->mmap_base);
util.c +3 -5
@@ -501,12 +501,10 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr, ret =security_mmap_file (file, prot, flag);if (!ret) { -if (mm_write_lock_killable (mm)) -return -EINTR ; ret =do_mmap (file, addr, len, prot, flag,0 , pgoff, - true, &populate, &uf); -mm_write_unlock (mm); -userfaultfd_unmap_complete (mm, &uf); + false, &populate, &uf); +if (ret != -EINTR ) +userfaultfd_unmap_complete (mm, &uf);if (populate)mm_populate (ret, populate); }
Signature · 1 line
2.25.0.341.g760bfbb309-goog
RFC PATCH 24/24 do_mmap: implement easiest cases of fine grained locking
In reply to Michel LespinasseUse a range lock in the easiest possible mmap case:
- the mmap address is known;
- there are no existing vmas within the mmap range;
- there is no file being mapped.
When these conditions are met, we can trivially support a fine grained range lock by just holding the mm_vma_lock accross the entire mmap operation. This is safe because the mmap only registers the new mapping using O(log N) operations, and does not have to call back into arbitrary code (such as file mmap handlers) or iterate over existing vmas and mapped pages.
--- mm/mmap.c | 36 +++++++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 7 deletions(-)
mmap.c +29 -7
@@ -1372,6 +1372,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr, unsigned long pgoff, bool locked, unsigned long *populate, struct list_head *uf) { + struct mm_lock_range mmap_range, *range = NULL;struct mm_struct *mm = current->mm;int pkey =0 ;
@@ -1406,8 +1407,18 @@ unsigned long do_mmap(struct file *file, unsigned long addr,if ((pgoff + (len >>PAGE_SHIFT )) < pgoff)return -EOVERFLOW ; -if (!locked &&mm_write_lock_killable (mm)) -return -EINTR ; +if (!locked) { +if (addr && !file) { +mm_init_lock_range (&mmap_range, addr, addr + len); + range = &mmap_range; + }else + range =mm_coarse_lock_range (); + retry: +if (mm_write_range_lock_killable (mm, range)) +return -EINTR ; +if (!mm_range_is_coarse (range)) +mm_vma_lock (mm); + }/* Too many mappings? */ if (mm->map_count > sysctl_max_map_count) {
@@ -1422,12 +1433,20 @@ unsigned long do_mmap(struct file *file, unsigned long addr,if (IS_ERR_VALUE (addr)) goto unlock; -if (flags &MAP_FIXED_NOREPLACE ) { +if ((flags &MAP_FIXED_NOREPLACE ) || + (!locked && !mm_range_is_coarse (range))) {struct vm_area_struct *vma =find_vma (mm, addr);if (vma && vma->vm_start < addr + len) { - addr = -EEXIST ; - goto unlock; +if (flags &MAP_FIXED_NOREPLACE ) { + addr = -EEXIST ; + goto unlock; + }else { +mm_vma_unlock (mm); +mm_write_range_unlock (mm, range); + range =mm_coarse_lock_range (); + goto retry; + } } }
@@ -1587,8 +1606,11 @@ unsigned long do_mmap(struct file *file, unsigned long addr, *populate = len; unlock: -if (!locked) -mm_write_unlock (mm); +if (!locked) { +if (!mm_range_is_coarse (range)) +mm_vma_unlock (mm); +mm_write_range_unlock (mm, range); + }return addr; }
Signature · 1 line
2.25.0.341.g760bfbb309-goog
Quoted 9 lines
Hi,
This is the first version of my work towards fine grained MM locking. This is still early work - I am happy with my page fault changes, but want to expand on the mmap/munmap side of things before I send the next version. I have previously shared this with some of the copied folks (for those who received that, there are no additional changes in this public resend). Please expect a v2 within a few weeks, with further changes for fine grained range locking in the mmap and munmap paths.
hello, Michel. I noticed rwsem has been renamed to mmap_lock and some apis were created for taking the lock.
but is the original fine grained mm locking series still under development? maybe i missed something but i failed to find v2 for it.
Thanks
Barry
Quoted 5 lines · Barry Song
hello, Michel. I noticed rwsem has been renamed to
mmap_lockand some apis were created for taking the lock.
but is the original fine grained mm locking series still under development? maybe i missed something but i failed to find v2 for it.
Most recently posted as [email protected]
Quoted 9 lines · Matthew Wilcox
Quoted 5 lines · Barry Song
hello, Michel. I noticed rwsem has been renamed to
mmap_lockand some apis were created for taking the lock.
but is the original fine grained mm locking series still under development? maybe i missed something but i failed to find v2 for it.Most recently posted as [email protected]
thanks!