Replace mmap_sem by a range lock
Subsystems (59)
cover Replace mmap_sem by a range lock
Following the series pushed by Davidlohr Bueso based on the Jan Kara's work [1] which introduces range locks, this series implements the first step of the attempt to replace the mmap_sem by a range lock.
While this series simply replaces the mmap_sem by a full range lock, the final goal is to introduce finer grain locking to allow better multi-thread performance in regards to the process's memory layout changes.
This series is currently supports x86 and PowerPc architectures only. Some drivers are also impacted to allow build and basic test on few platforms but a lot of additional works is required to complete the job for all the supported architectures.
The goal of this series is to check that no major performance hit happens for mono threaded process, as we could hope major improvements in the multi-threaded case once the finer grain locking is implemented.
I didn't do massive performance checking yet, but building a full kernel on a 80 threaded Power node, doesn't show performance hits. The build time is 11m56.701s on a vanilla kernel and 12m4.679s when range lock is used.
The next steps will attempt to implement finer grain locking but specific locking would certainly be required to protect mm data like the VMA cache.
This series applies on top of 4.11-rc7.
The first patch introduce a new parameter to some memory service which need to release the lock, since range lock require the range to specify, caller has to know about it.
The second patch is removing some assert which were based on lock service which are not provided by the range lock API. While some of these checks will not be valid with range locks, some may be reviewed.
The third patch is replacing nest locking operation on mmap_sem to simple locking operation has the nest lock check is not yet provided by the range lock API.
The latest patch is doing the job of replacing the mmap_sem by a range lock.
[1] "locking: Introduce range reader/writer lock"
http://www.mail-archive.com/[email protected]/msg1371431.html
Laurent Dufour (4):
Add additional range parameter to GUP() and handle_page_fault() Deactivate mmap_sem assert
Remove nest locking operation with mmap_sem
Change mmap_sem to range lock
arch/powerpc/kernel/vdso.c | 8 ++- arch/powerpc/kvm/book3s_64_mmu_hv.c | 6 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 6 +- arch/powerpc/kvm/book3s_64_vio.c | 6 +- arch/powerpc/kvm/book3s_hv.c | 8 ++- arch/powerpc/kvm/e500_mmu_host.c | 7 ++- arch/powerpc/mm/copro_fault.c | 8 ++- arch/powerpc/mm/fault.c | 12 ++-- arch/powerpc/mm/mmu_context_iommu.c | 6 +- arch/powerpc/mm/subpage-prot.c | 16 ++++-- arch/powerpc/oprofile/cell/spu_task_sync.c | 8 ++- arch/powerpc/platforms/cell/spufs/file.c | 4 +- arch/x86/entry/vdso/vma.c | 14 +++-- arch/x86/events/core.c | 1 - arch/x86/kernel/tboot.c | 2 +- arch/x86/kernel/vm86_32.c | 6 +- arch/x86/mm/fault.c | 39 +++++++------ arch/x86/mm/mpx.c | 20 ++++--- drivers/android/binder.c | 8 ++- drivers/firmware/efi/arm-runtime.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 9 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 8 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_events.c | 6 +- drivers/gpu/drm/amd/amdkfd/kfd_process.c | 6 +- drivers/gpu/drm/etnaviv/etnaviv_gem.c | 9 ++- drivers/gpu/drm/i915/i915_gem.c | 6 +- drivers/gpu/drm/i915/i915_gem_userptr.c | 14 +++-- drivers/gpu/drm/radeon/radeon_cs.c | 9 ++- drivers/gpu/drm/radeon/radeon_gem.c | 8 ++- drivers/gpu/drm/radeon/radeon_mn.c | 8 ++- drivers/gpu/drm/radeon/radeon_ttm.c | 2 +- drivers/gpu/drm/ttm/ttm_bo_vm.c | 6 +- drivers/gpu/drm/via/via_dmablit.c | 6 +- drivers/infiniband/core/umem.c | 22 +++++--- drivers/infiniband/core/umem_odp.c | 8 ++- drivers/infiniband/hw/hfi1/user_pages.c | 18 ++++-- drivers/infiniband/hw/mlx4/main.c | 6 +- drivers/infiniband/hw/mlx5/main.c | 6 +- drivers/infiniband/hw/mthca/mthca_memfree.c | 3 +- drivers/infiniband/hw/qib/qib_user_pages.c | 18 ++++-- drivers/infiniband/hw/usnic/usnic_uiom.c | 22 +++++--- drivers/iommu/amd_iommu_v2.c | 8 ++- drivers/iommu/intel-svm.c | 8 ++- drivers/media/v4l2-core/videobuf-core.c | 9 ++- drivers/media/v4l2-core/videobuf-dma-contig.c | 6 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 8 ++- drivers/misc/cxl/fault.c | 6 +- drivers/misc/mic/scif/scif_rma.c | 19 ++++--- drivers/oprofile/buffer_sync.c | 14 +++-- drivers/staging/lustre/lustre/llite/llite_mmap.c | 4 +- drivers/staging/lustre/lustre/llite/vvp_io.c | 6 +- .../interface/vchiq_arm/vchiq_2835_arm.c | 7 ++- .../vc04_services/interface/vchiq_arm/vchiq_arm.c | 6 +- drivers/vfio/vfio_iommu_spapr_tce.c | 13 +++-- drivers/vfio/vfio_iommu_type1.c | 24 +++++--- drivers/virt/fsl_hypervisor.c | 6 +- drivers/xen/gntdev.c | 6 +- drivers/xen/privcmd.c | 14 +++-- fs/aio.c | 7 ++- fs/coredump.c | 6 +- fs/exec.c | 26 ++++++--- fs/proc/base.c | 38 ++++++++----- fs/proc/internal.h | 1 + fs/proc/task_mmu.c | 30 ++++++---- fs/proc/task_nommu.c | 27 +++++---- fs/userfaultfd.c | 33 +++++------ include/linux/huge_mm.h | 2 - include/linux/hugetlb.h | 4 +- include/linux/mm.h | 21 ++++--- include/linux/mm_types.h | 3 +- include/linux/pagemap.h | 8 ++- include/linux/userfaultfd_k.h | 6 +- ipc/shm.c | 13 +++-- kernel/acct.c | 6 +- kernel/events/core.c | 6 +- kernel/events/uprobes.c | 28 ++++++---- kernel/exit.c | 10 ++-- kernel/fork.c | 21 ++++--- kernel/futex.c | 8 ++- kernel/sched/fair.c | 7 ++- kernel/sys.c | 31 ++++++++--- kernel/trace/trace_output.c | 6 +- mm/filemap.c | 9 +-- mm/frame_vector.c | 9 ++- mm/gup.c | 65 ++++++++++++---------- mm/hugetlb.c | 3 +- mm/init-mm.c | 2 +- mm/internal.h | 3 +- mm/khugepaged.c | 59 ++++++++++++-------- mm/ksm.c | 48 ++++++++++------ mm/madvise.c | 38 +++++++------ mm/memcontrol.c | 14 +++-- mm/memory.c | 43 +++++++------- mm/mempolicy.c | 32 +++++++---- mm/migrate.c | 12 ++-- mm/mincore.c | 6 +- mm/mlock.c | 25 ++++++--- mm/mmap.c | 51 +++++++++++------ mm/mmu_notifier.c | 6 +- mm/mprotect.c | 21 ++++--- mm/mremap.c | 6 +- mm/msync.c | 10 ++-- mm/nommu.c | 31 +++++++---- mm/oom_kill.c | 9 ++- mm/pagewalk.c | 3 - mm/process_vm_access.c | 9 ++- mm/shmem.c | 3 +- mm/swapfile.c | 8 ++- mm/userfaultfd.c | 25 +++++---- mm/util.c | 15 +++-- security/tomoyo/domain.c | 2 +- virt/kvm/async_pf.c | 8 ++- virt/kvm/kvm_main.c | 31 +++++++---- 115 files changed, 960 insertions(+), 555 deletions(-)
Signature · 7 lines
2.7.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 1/4 Add additional range parameter to GUP() and handle_page_fault()
Some functions which are called with the mmap_sem held by the caller may want to release it. Since mmap_sem will become a range lock a range_rwlock parameter must be added to allow the lock to freed correctly.
This patch add the additional parameter in the caller and the callee and also in the vm_fault structure.
Despite this additional parameter which is not used, there is no functional change in the patch.
Signed-off-by Laurent Dufour <[email protected]>
--- arch/powerpc/mm/copro_fault.c | 2 +- arch/powerpc/mm/fault.c | 2 +- arch/x86/mm/fault.c | 2 +- arch/x86/mm/mpx.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 2 +- drivers/gpu/drm/etnaviv/etnaviv_gem.c | 3 +- drivers/gpu/drm/i915/i915_gem_userptr.c | 2 +- drivers/gpu/drm/radeon/radeon_ttm.c | 2 +- drivers/infiniband/core/umem.c | 2 +- drivers/infiniband/core/umem_odp.c | 2 +- drivers/infiniband/hw/mthca/mthca_memfree.c | 3 +- drivers/infiniband/hw/qib/qib_user_pages.c | 2 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 2 +- drivers/iommu/intel-svm.c | 2 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 2 +- drivers/misc/mic/scif/scif_rma.c | 2 +- fs/exec.c | 2 +- fs/userfaultfd.c | 3 +- include/linux/hugetlb.h | 4 +-- include/linux/mm.h | 21 ++++++++----- include/linux/pagemap.h | 8 +++-- include/linux/userfaultfd_k.h | 6 ++-- kernel/events/uprobes.c | 4 +-- kernel/futex.c | 2 +- mm/filemap.c | 5 ++-- mm/frame_vector.c | 2 +- mm/gup.c | 46 ++++++++++++++++------------- mm/hugetlb.c | 3 +- mm/internal.h | 3 +- mm/khugepaged.c | 17 +++++++---- mm/ksm.c | 3 +- mm/madvise.c | 14 +++++---- mm/memory.c | 12 ++++---- mm/mempolicy.c | 2 +- mm/mmap.c | 4 +-- mm/mprotect.c | 2 +- mm/process_vm_access.c | 3 +- mm/userfaultfd.c | 8 +++-- security/tomoyo/domain.c | 2 +- virt/kvm/kvm_main.c | 12 ++++---- 40 files changed, 130 insertions(+), 92 deletions(-)
arch/powerpc/mm/copro_fault.c +1 -1
@@ -77,7 +77,7 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea, } ret =0 ; - *flt =handle_mm_fault (vma, ea, is_write ?FAULT_FLAG_WRITE :0 ); + *flt =handle_mm_fault (vma, ea, is_write ?FAULT_FLAG_WRITE :0 ,NULL );if (unlikely (*flt &VM_FAULT_ERROR )) {if (*flt &VM_FAULT_OOM ) { ret = -ENOMEM ;
arch/powerpc/mm/fault.c +1 -1
@@ -446,7 +446,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );/* * Handle the retry right now, the mmap_sem has been released in that
arch/x86/mm/fault.c +1 -1
@@ -1394,7 +1394,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,* the fault. Since we never set FAULT_FLAG_RETRY_NOWAIT, if * we get VM_FAULT_RETRY back, the mmap_sem has been unlocked. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL ); major |= fault &VM_FAULT_MAJOR ;/*
arch/x86/mm/mpx.c +1 -1
@@ -547,7 +547,7 @@ static int mpx_resolve_fault(long __user *addr, int write)int nr_pages =1 ; gup_ret =get_user_pages ((unsigned long )addr, nr_pages, - write ?FOLL_WRITE :0 ,NULL ,NULL ); + write ?FOLL_WRITE :0 ,NULL ,NULL ,NULL );/* * get_user_pages() returns number of pages gotten. * 0 means we failed to fault in and get anything,
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1 -1
@@ -627,7 +627,7 @@ int amdgpu_ttm_tt_get_user_pages(struct ttm_tt *ttm, struct page **pages)list_add (&guptask.list, >t->guptasks);spin_unlock (>t->guptasklock); - r =get_user_pages (userptr, num_pages, flags, p,NULL ); + r =get_user_pages (userptr, num_pages, flags, p,NULL ,NULL );spin_lock (>t->guptasklock);list_del (&guptask.list);
drivers/gpu/drm/etnaviv/etnaviv_gem.c +2 -1
@@ -761,7 +761,8 @@ static struct page **etnaviv_gem_userptr_do_get_pages(down_read (&mm->mmap_sem);while (pinned < npages) { ret =get_user_pages_remote (task, mm, ptr, npages - pinned, - flags, pvec + pinned,NULL ,NULL ); + flags, pvec + pinned,NULL ,NULL , +NULL );if (ret <0 )break ;
drivers/gpu/drm/i915/i915_gem_userptr.c +1 -1
@@ -516,7 +516,7 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work) obj->userptr.ptr + pinned *PAGE_SIZE , npages - pinned, flags, - pvec + pinned, NULL, NULL); + pvec + pinned, NULL, NULL, NULL);if (ret <0 )break ;
drivers/gpu/drm/radeon/radeon_ttm.c +1 -1
@@ -569,7 +569,7 @@ static int radeon_ttm_tt_pin_userptr(struct ttm_tt *ttm)struct page **pages = ttm->pages + pinned; r =get_user_pages (userptr, num_pages, write ?FOLL_WRITE :0 , - pages,NULL ); + pages,NULL ,NULL );if (r <0 ) goto release_pages;
drivers/infiniband/core/umem.c +1 -1
@@ -194,7 +194,7 @@ struct ib_umem *ib_umem_get(struct ib_ucontext *context, unsigned long addr, ret =get_user_pages (cur_base, min_t(unsigned long, npages,PAGE_SIZE /sizeof (struct page *)), - gup_flags, page_list, vma_list); + gup_flags, page_list, vma_list,NULL );if (ret <0 ) goto out;
drivers/infiniband/core/umem_odp.c +1 -1
@@ -649,7 +649,7 @@ int ib_umem_odp_map_dma_pages(struct ib_umem *umem, u64 user_virt, u64 bcnt,*/ npages =get_user_pages_remote (owning_process, owning_mm, user_virt, gup_num_pages, - flags, local_page_list,NULL ,NULL ); + flags, local_page_list,NULL ,NULL ,NULL );up_read (&owning_mm->mmap_sem);if (npages <0 )
drivers/infiniband/hw/mthca/mthca_memfree.c +2 -1
@@ -472,7 +472,8 @@ int mthca_map_user_db(struct mthca_dev *dev, struct mthca_uar *uar, goto out; } - ret =get_user_pages (uaddr &PAGE_MASK ,1 ,FOLL_WRITE , pages,NULL ); + ret =get_user_pages (uaddr &PAGE_MASK ,1 ,FOLL_WRITE , pages,NULL , +NULL );if (ret <0 ) goto out;
drivers/infiniband/hw/qib/qib_user_pages.c +1 -1
@@ -70,7 +70,7 @@ static int __qib_get_user_pages(unsigned long start_page, size_t num_pages, ret =get_user_pages (start_page + got *PAGE_SIZE , num_pages - got,FOLL_WRITE |FOLL_FORCE , - p + got,NULL ); + p + got,NULL ,NULL );if (ret <0 ) goto bail_release; }
drivers/infiniband/hw/usnic/usnic_uiom.c +1 -1
@@ -146,7 +146,7 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, ret =get_user_pages (cur_base, min_t(unsigned long, npages,PAGE_SIZE /sizeof (struct page *)), - gup_flags, page_list,NULL ); + gup_flags, page_list,NULL ,NULL );if (ret <0 ) goto out;
drivers/iommu/intel-svm.c +1 -1
@@ -591,7 +591,7 @@ static irqreturn_t prq_event_thread(int irq, void *d) goto invalid; ret =handle_mm_fault (vma, address, - req->wr_req ?FAULT_FLAG_WRITE :0 ); + req->wr_req ?FAULT_FLAG_WRITE :0 ,NULL );if (ret &VM_FAULT_ERROR ) goto invalid;
drivers/media/v4l2-core/videobuf-dma-sg.c +1 -1
@@ -186,7 +186,7 @@ static int videobuf_dma_init_user_locked(struct videobuf_dmabuf *dma, data, size, dma->nr_pages); err =get_user_pages (data &PAGE_MASK , dma->nr_pages, - flags, dma->pages,NULL ); + flags, dma->pages,NULL ,NULL );if (err != dma->nr_pages) { dma->nr_pages = (err >=0 ) ? err :0 ;
drivers/misc/mic/scif/scif_rma.c +1 -1
@@ -1401,7 +1401,7 @@ int __scif_pin_pages(void *addr, size_t len, int *out_prot, nr_pages, (prot &SCIF_PROT_WRITE ) ?FOLL_WRITE :0 , pinned_pages->pages, - NULL); + NULL, NULL);up_write (&mm->mmap_sem);if (nr_pages != pinned_pages->nr_pages) {if (try_upgrade) {
fs/exec.c +1 -1
@@ -214,7 +214,7 @@ static struct page *get_arg_page(struct linux_binprm *bprm, unsigned long pos,* doing the exec and bprm->mm is the new process's mm. */ ret =get_user_pages_remote (current, bprm->mm, pos,1 , gup_flags, - &page,NULL ,NULL ); + &page,NULL ,NULL ,NULL );if (ret <=0 )return NULL ;
fs/userfaultfd.c +2 -1
@@ -698,7 +698,8 @@ void mremap_userfaultfd_complete(struct vm_userfaultfd_ctx *vm_ctx, }bool userfaultfd_remove (struct vm_area_struct *vma, -unsigned long start,unsigned long end) +unsigned long start,unsigned long end, +struct range_rwlock *range) {struct mm_struct *mm = vma->vm_mm;struct userfaultfd_ctx *ctx;
include/linux/hugetlb.h +2 -2
@@ -66,7 +66,7 @@ int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct vm_arlong follow_hugetlb_page (struct mm_struct *,struct vm_area_struct *,struct page **,struct vm_area_struct **,unsigned long *,unsigned long *,long ,unsigned int , -int *); +int *,struct range_rwlock *);void unmap_hugepage_range (struct vm_area_struct *,unsigned long ,unsigned long ,struct page *); void __unmap_hugepage_range_final(struct mmu_gather *tlb,
@@ -137,7 +137,7 @@ static inline unsigned long hugetlb_total_pages(void)return 0 ; } -#define follow_hugetlb_page (m,v,p,vs,a,b,i,w,n) ({ BUG(); 0; }) +#define follow_hugetlb_page (m,v,p,vs,a,b,i,w,n,r) ({ BUG(); 0; })#define follow_huge_addr (mm, addr, write) ERR_PTR(-EINVAL)#define copy_hugetlb_page_range (src, dst, vma) ({ BUG(); 0; })static inline void hugetlb_report_meminfo (struct seq_file *m)
include/linux/mm.h +14 -7
@@ -23,6 +23,7 @@#include <linux/page_ext.h> #include <linux/err.h> #include <linux/page_ref.h> +#include <linux/range_rwlock.h> struct mempolicy ;struct anon_vma ;
@@ -344,6 +345,7 @@ struct vm_fault {* page table to avoid allocation from * atomic context. */ +struct range_rwlock *lockrange;/* RW range lock interval */ };/* page entry size for vm->huge_fault() */
@@ -1272,13 +1274,14 @@ int invalidate_inode_page(struct page *page);#ifdef CONFIG_MMU extern int handle_mm_fault (struct vm_area_struct *vma,unsigned long address, -unsigned int flags); +unsigned int flags,struct range_rwlock *range);extern int fixup_user_fault (struct task_struct *tsk,struct mm_struct *mm,unsigned long address,unsigned int fault_flags, -bool *unlocked); +bool *unlocked,struct range_rwlock *range);#else static inline int handle_mm_fault (struct vm_area_struct *vma, -unsigned long address,unsigned int flags) +unsigned long address,unsigned int flags, +struct range_rwlock *range) {/* should never happen if there's no MMU */ BUG ();
@@ -1286,7 +1289,8 @@ static inline int handle_mm_fault(struct vm_area_struct *vma, }static inline int fixup_user_fault (struct task_struct *tsk,struct mm_struct *mm,unsigned long address, -unsigned int fault_flags,bool *unlocked) +unsigned int fault_flags,bool *unlocked, +struct range_rwlock *range) {/* should never happen if there's no MMU */ BUG ();
@@ -1304,12 +1308,15 @@ extern int __access_remote_vm(struct task_struct *tsk, struct mm_struct *mm,long get_user_pages_remote (struct task_struct *tsk,struct mm_struct *mm,unsigned long start,unsigned long nr_pages,unsigned int gup_flags,struct page **pages, -struct vm_area_struct **vmas,int *locked); +struct vm_area_struct **vmas,int *locked, +struct range_rwlock *range);long get_user_pages (unsigned long start,unsigned long nr_pages,unsigned int gup_flags,struct page **pages, -struct vm_area_struct **vmas); +struct vm_area_struct **vmas, +struct range_rwlock *range);long get_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages,int *locked); +unsigned int gup_flags,struct page **pages,int *locked, +struct range_rwlock *range);long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages,struct page **pages,unsigned int gup_flags); int get_user_pages_fast(unsigned long start, int nr_pages, int write,
include/linux/pagemap.h +5 -3
@@ -434,7 +434,7 @@ static inline pgoff_t linear_page_index(struct vm_area_struct *vma,extern void __lock_page (struct page *page);extern int __lock_page_killable (struct page *page);extern int __lock_page_or_retry (struct page *page,struct mm_struct *mm, -unsigned int flags); +unsigned int flags,struct range_rwlock *range);extern void unlock_page (struct page *page);static inline int trylock_page (struct page *page)
@@ -474,10 +474,12 @@ static inline int lock_page_killable(struct page *page)* __lock_page_or_retry(). */ static inline int lock_page_or_retry (struct page *page,struct mm_struct *mm, -unsigned int flags) +unsigned int flags, +struct range_rwlock *range) {might_sleep (); -return trylock_page (page) ||__lock_page_or_retry (page, mm, flags); +return trylock_page (page) ||__lock_page_or_retry (page, mm, flags, + range); }/*
include/linux/userfaultfd_k.h +4 -2
@@ -63,7 +63,8 @@ extern void mremap_userfaultfd_complete(struct vm_userfaultfd_ctx *,extern bool userfaultfd_remove (struct vm_area_struct *vma,unsigned long start, -unsigned long end); +unsigned long end, +struct range_rwlock *range); extern int userfaultfd_unmap_prep(struct vm_area_struct *vma, unsigned long start, unsigned long end,
@@ -119,7 +120,8 @@ static inline void mremap_userfaultfd_complete(struct vm_userfaultfd_ctx *ctx,static inline bool userfaultfd_remove (struct vm_area_struct *vma,unsigned long start, -unsigned long end) +unsigned long end, +struct range_rwlock *range) {return true; }
kernel/events/uprobes.c +2 -2
@@ -309,7 +309,7 @@ int uprobe_write_opcode(struct mm_struct *mm, unsigned long vaddr, retry:/* Read the page with vaddr into memory */ ret =get_user_pages_remote (NULL , mm, vaddr,1 , -FOLL_FORCE |FOLL_SPLIT , &old_page, &vma,NULL ); +FOLL_FORCE |FOLL_SPLIT , &old_page, &vma,NULL ,NULL );if (ret <=0 )return ret;
@@ -1720,7 +1720,7 @@ static int is_trap_at_addr(struct mm_struct *mm, unsigned long vaddr)* essentially a kernel access to the memory. */ result =get_user_pages_remote (NULL , mm, vaddr,1 ,FOLL_FORCE , &page, -NULL ,NULL ); +NULL ,NULL ,NULL );if (result <0 )return result;
kernel/futex.c +1 -1
@@ -727,7 +727,7 @@ static int fault_in_user_writeable(u32 __user *uaddr)down_read (&mm->mmap_sem); ret =fixup_user_fault (current, mm, (unsigned long )uaddr, -FAULT_FLAG_WRITE ,NULL ); +FAULT_FLAG_WRITE ,NULL ,NULL );up_read (&mm->mmap_sem);return ret <0 ? ret :0 ;
mm/filemap.c +3 -2
@@ -1053,7 +1053,7 @@ EXPORT_SYMBOL_GPL(__lock_page_killable);* with the page locked and the mmap_sem unperturbed. */ int __lock_page_or_retry (struct page *page,struct mm_struct *mm, -unsigned int flags) +unsigned int flags,struct range_rwlock *range) {if (flags &FAULT_FLAG_ALLOW_RETRY ) {/*
@@ -2232,7 +2232,8 @@ int filemap_fault(struct vm_fault *vmf) goto no_cached_page; } -if (!lock_page_or_retry (page, vmf->vma->vm_mm, vmf->flags)) { +if (!lock_page_or_retry (page, vmf->vma->vm_mm, vmf->flags, + vmf->lockrange)) {put_page (page);return ret |VM_FAULT_RETRY ; }
mm/frame_vector.c +1 -1
@@ -56,7 +56,7 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames, vec->got_ref = true; vec->is_pfns = false; ret =get_user_pages_locked (start, nr_frames, - gup_flags, (struct page **)(vec->ptrs), &locked); + gup_flags, (struct page **)(vec->ptrs), &locked,NULL ); goto out; }
mm/gup.c +26 -20
160 lines not on this page · raw message
mm/hugetlb.c +2 -1
@@ -4089,7 +4089,8 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,long follow_hugetlb_page (struct mm_struct *mm,struct vm_area_struct *vma,struct page **pages,struct vm_area_struct **vmas,unsigned long *position,unsigned long *nr_pages, -long i,unsigned int flags,int *nonblocking) +long i,unsigned int flags,int *nonblocking, +struct range_rwlock *range) {unsigned long pfn_offset;unsigned long vaddr = *position;
mm/internal.h +2 -1
@@ -278,7 +278,8 @@ void __vma_link_list(struct mm_struct *mm, struct vm_area_struct *vma,#ifdef CONFIG_MMU extern long populate_vma_page_range (struct vm_area_struct *vma, -unsigned long start,unsigned long end,int *nonblocking); +unsigned long start,unsigned long end,int *nonblocking, +struct range_rwlock *range);extern void munlock_vma_pages_range (struct vm_area_struct *vma,unsigned long start,unsigned long end);static inline void munlock_vma_pages_all (struct vm_area_struct *vma)
mm/khugepaged.c +11 -6
@@ -875,7 +875,8 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,static bool __collapse_huge_page_swapin (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,pmd_t *pmd, -int referenced) +int referenced, +struct range_rwlock *range) {int swapped_in =0 , ret =0 ;struct vm_fault vmf = {
@@ -884,6 +885,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm, .flags = FAULT_FLAG_ALLOW_RETRY, .pmd = pmd, .pgoff = linear_page_index(vma, address), + .lockrange = range, };/* we only decide to swapin, if there is enough young ptes */
@@ -928,7 +930,8 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,static void collapse_huge_page (struct mm_struct *mm,unsigned long address,struct page **hpage, -int node,int referenced) +int node,int referenced, +struct range_rwlock *range) {pmd_t *pmd, _pmd;pte_t *pte;
@@ -986,7 +989,8 @@ static void collapse_huge_page(struct mm_struct *mm,* If it fails, we release mmap_sem and jump out_nolock. * Continuing to collapse causes inconsistency. */ -if (!__collapse_huge_page_swapin (mm, vma, address, pmd, referenced)) { +if (!__collapse_huge_page_swapin (mm, vma, address, pmd, referenced, + range)) {mem_cgroup_cancel_charge (new_page, memcg, true);up_read (&mm->mmap_sem); goto out_nolock;
@@ -1093,7 +1097,8 @@ static void collapse_huge_page(struct mm_struct *mm,static int khugepaged_scan_pmd (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address, -struct page **hpage) +struct page **hpage, +struct range_rwlock *range) {pmd_t *pmd;pte_t *pte, *_pte;
@@ -1207,7 +1212,7 @@ static int khugepaged_scan_pmd(struct mm_struct *mm,if (ret) { node =khugepaged_find_target_node ();/* collapse_huge_page will return with the mmap_sem released */ -collapse_huge_page (mm, address, hpage, node, referenced); +collapse_huge_page (mm, address, hpage, node, referenced, range); } out: trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced,
@@ -1728,7 +1733,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, }else { ret =khugepaged_scan_pmd (mm, vma, khugepaged_scan.address, - hpage); + hpage,NULL ); }/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ;
mm/ksm.c +2 -1
@@ -391,7 +391,8 @@ static int break_ksm(struct vm_area_struct *vma, unsigned long addr)break ;if (PageKsm (page)) ret =handle_mm_fault (vma, addr, -FAULT_FLAG_WRITE |FAULT_FLAG_REMOTE ); +FAULT_FLAG_WRITE |FAULT_FLAG_REMOTE , +NULL );else ret =VM_FAULT_WRITE ;put_page (page);
mm/madvise.c +8 -6
@@ -507,13 +507,14 @@ static long madvise_free(struct vm_area_struct *vma,*/ static long madvise_dontneed (struct vm_area_struct *vma,struct vm_area_struct **prev, -unsigned long start,unsigned long end) +unsigned long start,unsigned long end, +struct range_rwlock *range) { *prev = vma;if (!can_madv_dontneed_vma (vma))return -EINVAL ; -if (!userfaultfd_remove (vma, start, end)) { +if (!userfaultfd_remove (vma, start, end, range)) { *prev =NULL ;/* mmap_sem has been dropped, prev is stale */ down_read (¤t->mm->mmap_sem);
@@ -590,7 +591,7 @@ static long madvise_remove(struct vm_area_struct *vma,* mmap_sem. */ get_file (f); -if (userfaultfd_remove (vma, start, end)) { + if (userfaultfd_remove(vma, start, end, NULL) {/* mmap_sem was not released by userfaultfd_remove() */ up_read (¤t->mm->mmap_sem); }
@@ -643,7 +644,8 @@ static int madvise_hwpoison(int bhv, unsigned long start, unsigned long end)static long madvise_vma (struct vm_area_struct *vma,struct vm_area_struct **prev, -unsigned long start,unsigned long end,int behavior) +unsigned long start,unsigned long end,int behavior, +struct range_rwlock *range) {switch (behavior) {case MADV_REMOVE :
@@ -659,7 +661,7 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev,return madvise_free (vma, prev, start, end);/* passthrough */ case MADV_DONTNEED : -return madvise_dontneed (vma, prev, start, end); +return madvise_dontneed (vma, prev, start, end, range);default :return madvise_behavior (vma, prev, start, end, behavior); }
@@ -822,7 +824,7 @@ SYSCALL_DEFINE3(madvise, unsigned long, start, size_t, len_in, int, behavior) tmp = end;/* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ - error =madvise_vma (vma, &prev, start, tmp, behavior); + error =madvise_vma (vma, &prev, start, tmp, behavior,NULL );if (error) goto out; start = tmp;
mm/memory.c +7 -5
@@ -2732,7 +2732,8 @@ int do_swap_page(struct vm_fault *vmf) } swapcache = page; - locked =lock_page_or_retry (page, vma->vm_mm, vmf->flags); + locked =lock_page_or_retry (page, vma->vm_mm, vmf->flags, + vmf->lockrange);delayacct_clear_flag (DELAYACCT_PF_SWAPIN );if (!locked) {
@@ -3765,7 +3766,7 @@ static int handle_pte_fault(struct vm_fault *vmf)* return value. See filemap_fault() and __lock_page_or_retry(). */ static int __handle_mm_fault (struct vm_area_struct *vma,unsigned long address, -unsigned int flags) +unsigned int flags,struct range_rwlock *range) {struct vm_fault vmf = { .vma = vma,
@@ -3773,6 +3774,7 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address, .flags = flags, .pgoff = linear_page_index(vma, address), .gfp_mask = __get_fault_gfp_mask(vma), + .lockrange = range, };struct mm_struct *mm = vma->vm_mm;pgd_t *pgd;
@@ -3848,7 +3850,7 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,* return value. See filemap_fault() and __lock_page_or_retry(). */ int handle_mm_fault (struct vm_area_struct *vma,unsigned long address, -unsigned int flags) +unsigned int flags,struct range_rwlock *range) {int ret;
@@ -3875,7 +3877,7 @@ int handle_mm_fault(struct vm_area_struct *vma, unsigned long address,if (unlikely (is_vm_hugetlb_page (vma))) ret =hugetlb_fault (vma->vm_mm, vma, address, flags);else - ret =__handle_mm_fault (vma, address, flags); + ret =__handle_mm_fault (vma, address, flags, range);if (flags &FAULT_FLAG_USER ) {mem_cgroup_oom_disable ();
@@ -4169,7 +4171,7 @@ int __access_remote_vm(struct task_struct *tsk, struct mm_struct *mm,struct page *page =NULL ; ret =get_user_pages_remote (tsk, mm, addr,1 , - gup_flags, &page, &vma,NULL ); + gup_flags, &page, &vma,NULL ,NULL );if (ret <=0 ) {#ifndef CONFIG_HAVE_IOREMAP_PROT break ;
mm/mempolicy.c +1 -1
@@ -855,7 +855,7 @@ static int lookup_node(unsigned long addr)struct page *p;int err; - err =get_user_pages (addr &PAGE_MASK ,1 ,0 , &p,NULL ); + err =get_user_pages (addr &PAGE_MASK ,1 ,0 , &p,NULL ,NULL );if (err >=0 ) { err =page_to_nid (p);put_page (p);
mm/mmap.c +2 -2
@@ -2380,7 +2380,7 @@ find_extend_vma(struct mm_struct *mm, unsigned long addr)if (!prev ||expand_stack (prev, addr))return NULL ;if (prev->vm_flags &VM_LOCKED ) -populate_vma_page_range (prev, addr, prev->vm_end,NULL ); +populate_vma_page_range (prev, addr, prev->vm_end,NULL ,NULL ); return prev; }#else
@@ -2415,7 +2415,7 @@ find_extend_vma(struct mm_struct *mm, unsigned long addr)if (expand_stack (vma, addr))return NULL ;if (vma->vm_flags &VM_LOCKED ) -populate_vma_page_range (vma, addr, start,NULL ); +populate_vma_page_range (vma, addr, start,NULL ,NULL ); return vma; }#endif
mm/mprotect.c +1 -1
@@ -358,7 +358,7 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,*/ if ((oldflags & (VM_WRITE |VM_SHARED |VM_LOCKED )) ==VM_LOCKED && (newflags &VM_WRITE )) { -populate_vma_page_range (vma, start, end,NULL ); +populate_vma_page_range (vma, start, end,NULL ,NULL ); }vm_stat_account (mm, oldflags, -nrpages);
mm/process_vm_access.c +2 -1
@@ -111,7 +111,8 @@ static int process_vm_rw_single_vec(unsigned long addr,*/ down_read (&mm->mmap_sem); pages =get_user_pages_remote (task, mm, pa, pages, flags, - process_pages,NULL , &locked); + process_pages,NULL , &locked, +NULL );if (locked)up_read (&mm->mmap_sem);if (pages <=0 )
mm/userfaultfd.c +5 -3
@@ -156,7 +156,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,unsigned long dst_start, unsigned long src_start, unsigned long len, - bool zeropage) + bool zeropage, + struct range_rwlock *range) { int vm_alloc_shared = dst_vma->vm_flags & VM_SHARED;int vm_shared = dst_vma->vm_flags &VM_SHARED ;
@@ -368,7 +369,8 @@ extern ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,unsigned long dst_start, unsigned long src_start, unsigned long len, - bool zeropage); + bool zeropage, + struct range_rwlock *range);#endif /* CONFIG_HUGETLB_PAGE */ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm,
@@ -439,7 +441,7 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm,*/ if (is_vm_hugetlb_page (dst_vma))return __mcopy_atomic_hugetlb (dst_mm, dst_vma, dst_start, - src_start, len, zeropage); + src_start, len, zeropage,NULL );if (!vma_is_anonymous (dst_vma) && !vma_is_shmem (dst_vma)) goto out_unlock;
security/tomoyo/domain.c +1 -1
@@ -883,7 +883,7 @@ bool tomoyo_dump_page(struct linux_binprm *bprm, unsigned long pos,* the execve(). */ if (get_user_pages_remote (current, bprm->mm, pos,1 , -FOLL_FORCE , &page,NULL ,NULL ) <=0 ) +FOLL_FORCE , &page,NULL ,NULL ,NULL ) <=0 )return false;#else page = bprm->page[pos /PAGE_SIZE ];
virt/kvm/kvm_main.c +7 -5
@@ -1354,14 +1354,14 @@ static int get_user_page_nowait(unsigned long start, int write,if (write) flags |=FOLL_WRITE ; -return get_user_pages (start,1 , flags, page,NULL ); +return get_user_pages (start,1 , flags, page,NULL ,NULL ); }static inline int check_user_page_hwpoison (unsigned long addr) {int rc, flags =FOLL_HWPOISON |FOLL_WRITE ; - rc =get_user_pages (addr,1 , flags,NULL ,NULL ); + rc =get_user_pages (addr,1 , flags,NULL ,NULL ,NULL );return rc == -EHWPOISON ; }
@@ -1458,7 +1458,8 @@ static bool vma_is_valid(struct vm_area_struct *vma, bool write_fault)static int hva_to_pfn_remapped (struct vm_area_struct *vma,unsigned long addr,bool *async, -bool write_fault,kvm_pfn_t *p_pfn) +bool write_fault,kvm_pfn_t *p_pfn, +struct range_rwlock *range) {unsigned long pfn;int r;
@@ -1472,7 +1473,7 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma,bool unlocked = false; r =fixup_user_fault (current, current->mm, addr, (write_fault ?FAULT_FLAG_WRITE :0 ), - &unlocked); + &unlocked, range);if (unlocked)return -EAGAIN ;if (r)
@@ -1549,7 +1550,8 @@ static kvm_pfn_t hva_to_pfn(unsigned long addr, bool atomic, bool *async,if (vma ==NULL ) pfn =KVM_PFN_ERR_FAULT ;else if (vma->vm_flags & (VM_IO |VM_PFNMAP )) { - r =hva_to_pfn_remapped (vma, addr, async, write_fault, &pfn); + r =hva_to_pfn_remapped (vma, addr, async, write_fault, &pfn, +NULL );if (r == -EAGAIN ) goto retry;if (r <0 )
Signature · 7 lines
2.7.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 2/4 Deactivate mmap_sem assert
In reply to Laurent DufourWhen mmap_sem will be moved to a range lock, some assertion done in the code are no more valid, like the one ensuring mmap_sem is held.
This patch should be reverted later and some check might be reviewed once the range locking API provides dedicated services.
Signed-off-by Laurent Dufour <[email protected]>
--- arch/x86/events/core.c | 1 - fs/userfaultfd.c | 6 ------ include/linux/huge_mm.h | 2 -- mm/gup.c | 1 - mm/memory.c | 12 +++--------- mm/pagewalk.c | 3 --- 6 files changed, 3 insertions(+), 22 deletions(-)
arch/x86/events/core.c +0 -1
@@ -2120,7 +2120,6 @@ static void x86_pmu_event_mapped(struct perf_event *event)* For now, this can't happen because all callers hold mmap_sem * for write. If this changes, we'll need a different solution. */ -lockdep_assert_held_exclusive (¤t->mm->mmap_sem);if (atomic_inc_return (¤t->mm->context.perf_rdpmc_allowed) ==1 )on_each_cpu_mask (mm_cpumask (current->mm), refresh_pce,NULL ,1 );
fs/userfaultfd.c +0 -6
@@ -222,8 +222,6 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,pte_t *pte;bool ret = true; -VM_BUG_ON (!rwsem_is_locked (&mm->mmap_sem)); - pte =huge_pte_offset (mm, address);if (!pte) goto out;
@@ -271,8 +269,6 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,pte_t *pte;bool ret = true; -VM_BUG_ON (!rwsem_is_locked (&mm->mmap_sem)); - pgd =pgd_offset (mm, address);if (!pgd_present (*pgd)) goto out;
@@ -340,8 +336,6 @@ int handle_userfault(struct vm_fault *vmf, unsigned long reason)bool must_wait, return_to_userland;long blocking_state; -BUG_ON (!rwsem_is_locked (&mm->mmap_sem)); - ret =VM_FAULT_SIGBUS ; ctx = vmf->vma->vm_userfaultfd_ctx.ctx;if (!ctx)
include/linux/huge_mm.h +0 -2
@@ -161,7 +161,6 @@ extern spinlock_t *__pud_trans_huge_lock(pud_t *pud,static inline spinlock_t *pmd_trans_huge_lock (pmd_t *pmd,struct vm_area_struct *vma) { -VM_BUG_ON_VMA (!rwsem_is_locked (&vma->vm_mm->mmap_sem), vma);if (pmd_trans_huge (*pmd) ||pmd_devmap (*pmd))return __pmd_trans_huge_lock (pmd, vma);else
@@ -170,7 +169,6 @@ static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,static inline spinlock_t *pud_trans_huge_lock (pud_t *pud,struct vm_area_struct *vma) { -VM_BUG_ON_VMA (!rwsem_is_locked (&vma->vm_mm->mmap_sem), vma);if (pud_trans_huge (*pud) ||pud_devmap (*pud))return __pud_trans_huge_lock (pud, vma);else
mm/gup.c +0 -1
@@ -1040,7 +1040,6 @@ long populate_vma_page_range(struct vm_area_struct *vma,VM_BUG_ON (end & ~PAGE_MASK );VM_BUG_ON_VMA (start < vma->vm_start, vma);VM_BUG_ON_VMA (end > vma->vm_end, vma); -VM_BUG_ON_MM (!rwsem_is_locked (&mm->mmap_sem), mm); gup_flags =FOLL_TOUCH |FOLL_POPULATE |FOLL_MLOCK ;if (vma->vm_flags &VM_LOCKONFAULT )
mm/memory.c +3 -9
@@ -1298,8 +1298,7 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, next =pmd_addr_end (addr, end);if (pmd_trans_huge (*pmd) ||pmd_devmap (*pmd)) {if (next - addr !=HPAGE_PMD_SIZE ) { -VM_BUG_ON_VMA (vma_is_anonymous (vma) && - !rwsem_is_locked (&tlb->mm->mmap_sem), vma); +VM_BUG_ON_VMA (vma_is_anonymous (vma), vma);__split_huge_pmd (vma, pmd, addr, false,NULL ); }else if (zap_huge_pmd (tlb, vma, pmd, addr)) goto next;
@@ -1334,10 +1333,9 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb, do { next = pud_addr_end(addr, end); if (pud_trans_huge(*pud) || pud_devmap(*pud)) { - if (next - addr != HPAGE_PUD_SIZE) { - VM_BUG_ON_VMA(!rwsem_is_locked(&tlb->mm->mmap_sem), vma); + if (next - addr != HPAGE_PUD_SIZE) split_huge_pud(vma, pud, addr); - } else if (zap_huge_pud(tlb, vma, pud, addr)) + else if (zap_huge_pud(tlb, vma, pud, addr)) goto next; /* fall through */ }
@@ -4305,10 +4303,6 @@ void __might_fault(const char *file, int line)if (pagefault_disabled ())return ; __might_sleep(file, line, 0); -#if defined(CONFIG_DEBUG_ATOMIC_SLEEP ) -if (current->mm) -might_lock_read (¤t->mm->mmap_sem); -#endif }EXPORT_SYMBOL (__might_fault);#endif
mm/pagewalk.c +0 -3
@@ -293,8 +293,6 @@ int walk_page_range(unsigned long start, unsigned long end,if (!walk->mm)return -EINVAL ; -VM_BUG_ON_MM (!rwsem_is_locked (&walk->mm->mmap_sem), walk->mm); - vma =find_vma (walk->mm, start); do { if (!vma) { /* after the last vma */
@@ -336,7 +334,6 @@ int walk_page_vma(struct vm_area_struct *vma, struct mm_walk *walk)if (!walk->mm)return -EINVAL ; -VM_BUG_ON (!rwsem_is_locked (&walk->mm->mmap_sem));VM_BUG_ON (!vma); walk->vma = vma; err =walk_page_test (vma->vm_start, vma->vm_end, walk);
Signature · 7 lines
2.7.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [RFC 2/4] Deactivate mmap_sem assert
Quoted 3 lines · Laurent Dufour
When
mmap_semwill be moved to a range lock, some assertion done in the code are no more valid, like the one ensuringmmap_semis held.
Why are they no longer valid?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 2/4] Deactivate mmap_sem assert
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Laurent Dufour
When
mmap_semwill be moved to a range lock, some assertion done in the code are no more valid, like the one ensuringmmap_semis held.Why are they no longer valid?
I didn't explain that very well..
When using a range lock we can't check that the lock is simply held, but if the range we are interesting on is locked or not.
As I mentioned this patch will have to be reverted / reviewed once the range lock is providing dedicated APIs, but some check might be difficult to adapt to a range.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [RFC 2/4] Deactivate mmap_sem assert
Quoted 6 lines · Laurent Dufour
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Laurent Dufour
When
mmap_semwill be moved to a range lock, some assertion done in the code are no more valid, like the one ensuringmmap_semis held.
Quoted 6 lines
Quoted 1 line
Why are they no longer valid?
I didn't explain that very well..
When using a range lock we can't check that the lock is simply held, but if the range we are interesting on is locked or not.
I don't think it matters.. That is, in general you cannot assume anything about the ranges, therefore, for deadlock analysis you have to assume each range is the full range.
Once you're there, and assume that each range is the full range, this test is once again trivial.
The fact that not all ranges are the full range, is merely a performance consideration, but should not be a correctness issue.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 3/4 Remove nest locking operation with mmap_sem
In reply to Laurent DufourThe range locking framework doesn't yet provide nest locking operation, so remove them assuming that the check is now good.
Signed-off-by Laurent Dufour <[email protected]>
--- mm/mmap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/mmap.c +2 -2
@@ -3284,7 +3284,7 @@ static void vm_lock_anon_vma(struct mm_struct *mm, struct anon_vma *anon_vma)* The LSB of head.next can't change from under us * because we hold the mm_all_locks_mutex. */ -down_write_nest_lock (&anon_vma->root->rwsem, &mm->mmap_sem); +down_write (&anon_vma->root->rwsem);/* * We can safely modify head.next after taking the * anon_vma->root->rwsem. If some other vma in this mm shares
@@ -3314,7 +3314,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping)*/ if (test_and_set_bit (AS_MM_ALL_LOCKS , &mapping->flags))BUG (); -down_write_nest_lock (&mapping->i_mmap_rwsem, &mm->mmap_sem); +down_write (&mapping->i_mmap_rwsem); } }
Signature · 7 lines
2.7.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 4/4 Change mmap_sem to range lock
In reply to Laurent DufourChange the mmap_sem to a range lock to allow finer grain locking on the memory layout of a task.
This patch rename mmap_sem into mmap_rw_tree to avoid confusion and replace any locking (read or write) by complete range locking. So there is no functional change except in the way the underlying locking is achieved.
Currently, this patch only supports x86 and PowerPc architectures, furthermore it hould break the build of any others.
Signed-off-by Laurent Dufour <[email protected]>
--- arch/powerpc/kernel/vdso.c | 8 ++-- arch/powerpc/kvm/book3s_64_mmu_hv.c | 6 ++- arch/powerpc/kvm/book3s_64_mmu_radix.c | 6 ++- arch/powerpc/kvm/book3s_64_vio.c | 6 ++- arch/powerpc/kvm/book3s_hv.c | 8 ++-- arch/powerpc/kvm/e500_mmu_host.c | 7 +++- arch/powerpc/mm/copro_fault.c | 6 ++- arch/powerpc/mm/fault.c | 12 +++--- arch/powerpc/mm/mmu_context_iommu.c | 6 ++- arch/powerpc/mm/subpage-prot.c | 16 +++++--- arch/powerpc/oprofile/cell/spu_task_sync.c | 8 ++-- arch/powerpc/platforms/cell/spufs/file.c | 4 +- arch/x86/entry/vdso/vma.c | 14 ++++--- arch/x86/kernel/tboot.c | 2 +- arch/x86/kernel/vm86_32.c | 6 ++- arch/x86/mm/fault.c | 39 +++++++++++-------- arch/x86/mm/mpx.c | 18 ++++++--- drivers/android/binder.c | 8 ++-- drivers/firmware/efi/arm-runtime.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 9 +++-- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 8 ++-- drivers/gpu/drm/amd/amdkfd/kfd_events.c | 6 ++- drivers/gpu/drm/amd/amdkfd/kfd_process.c | 6 ++- drivers/gpu/drm/etnaviv/etnaviv_gem.c | 6 ++- drivers/gpu/drm/i915/i915_gem.c | 6 ++- drivers/gpu/drm/i915/i915_gem_userptr.c | 12 ++++-- drivers/gpu/drm/radeon/radeon_cs.c | 9 +++-- drivers/gpu/drm/radeon/radeon_gem.c | 8 ++-- drivers/gpu/drm/radeon/radeon_mn.c | 8 ++-- drivers/gpu/drm/ttm/ttm_bo_vm.c | 6 ++- drivers/gpu/drm/via/via_dmablit.c | 6 ++- drivers/infiniband/core/umem.c | 20 ++++++---- drivers/infiniband/core/umem_odp.c | 6 ++- drivers/infiniband/hw/hfi1/user_pages.c | 18 ++++++--- drivers/infiniband/hw/mlx4/main.c | 6 ++- drivers/infiniband/hw/mlx5/main.c | 6 ++- drivers/infiniband/hw/qib/qib_user_pages.c | 16 +++++--- drivers/infiniband/hw/usnic/usnic_uiom.c | 20 ++++++---- drivers/iommu/amd_iommu_v2.c | 8 ++-- drivers/iommu/intel-svm.c | 6 ++- drivers/media/v4l2-core/videobuf-core.c | 9 +++-- drivers/media/v4l2-core/videobuf-dma-contig.c | 6 ++- drivers/media/v4l2-core/videobuf-dma-sg.c | 6 ++- drivers/misc/cxl/fault.c | 6 ++- drivers/misc/mic/scif/scif_rma.c | 17 +++++--- drivers/oprofile/buffer_sync.c | 14 ++++--- drivers/staging/lustre/lustre/llite/llite_mmap.c | 4 +- drivers/staging/lustre/lustre/llite/vvp_io.c | 6 ++- .../interface/vchiq_arm/vchiq_2835_arm.c | 7 +++- .../vc04_services/interface/vchiq_arm/vchiq_arm.c | 6 ++- drivers/vfio/vfio_iommu_spapr_tce.c | 13 +++++-- drivers/vfio/vfio_iommu_type1.c | 24 +++++++----- drivers/virt/fsl_hypervisor.c | 6 ++- drivers/xen/gntdev.c | 6 ++- drivers/xen/privcmd.c | 14 ++++--- fs/aio.c | 7 +++- fs/coredump.c | 6 ++- fs/exec.c | 24 ++++++++---- fs/proc/base.c | 38 +++++++++++------- fs/proc/internal.h | 1 + fs/proc/task_mmu.c | 30 +++++++++------ fs/proc/task_nommu.c | 27 ++++++++----- fs/userfaultfd.c | 24 +++++++----- include/linux/mm_types.h | 3 +- ipc/shm.c | 13 +++++-- kernel/acct.c | 6 ++- kernel/events/core.c | 6 ++- kernel/events/uprobes.c | 24 ++++++++---- kernel/exit.c | 10 +++-- kernel/fork.c | 21 ++++++---- kernel/futex.c | 8 ++-- kernel/sched/fair.c | 7 ++-- kernel/sys.c | 31 ++++++++++----- kernel/trace/trace_output.c | 6 ++- mm/filemap.c | 4 +- mm/frame_vector.c | 9 +++-- mm/gup.c | 22 ++++++----- mm/init-mm.c | 2 +- mm/khugepaged.c | 44 +++++++++++++-------- mm/ksm.c | 45 ++++++++++++++-------- mm/madvise.c | 28 ++++++++------ mm/memcontrol.c | 14 ++++--- mm/memory.c | 21 +++++++--- mm/mempolicy.c | 30 +++++++++------ mm/migrate.c | 12 ++++-- mm/mincore.c | 6 ++- mm/mlock.c | 25 ++++++++---- mm/mmap.c | 43 ++++++++++++++------- mm/mmu_notifier.c | 6 ++- mm/mprotect.c | 19 ++++++--- mm/mremap.c | 6 ++- mm/msync.c | 10 +++-- mm/nommu.c | 31 ++++++++++----- mm/oom_kill.c | 9 +++-- mm/process_vm_access.c | 8 ++-- mm/shmem.c | 3 +- mm/swapfile.c | 8 ++-- mm/userfaultfd.c | 19 +++++---- mm/util.c | 15 ++++++-- virt/kvm/async_pf.c | 8 ++-- virt/kvm/kvm_main.c | 21 ++++++---- 102 files changed, 838 insertions(+), 452 deletions(-)
arch/powerpc/kernel/vdso.c +5 -3
@@ -154,6 +154,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)struct page **vdso_pagelist;unsigned long vdso_pages;unsigned long vdso_base; +struct range_rwlock range;int rc;if (!vdso_ready)
@@ -196,7 +197,8 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)* and end up putting it elsewhere. * Add enough to the size so that the result can be aligned. */ -if (down_write_killable (&mm->mmap_sem)) +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return -EINTR ; vdso_base = get_unmapped_area(NULL, vdso_base, (vdso_pages << PAGE_SHIFT) +
@@ -236,11 +238,11 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) goto fail_mmapsem; } -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return 0 ; fail_mmapsem: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return rc; }
arch/powerpc/kvm/book3s_64_mmu_hv.c +4 -2
@@ -485,6 +485,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,struct vm_area_struct *vma;unsigned long rcbits;long mmio_update; +struct range_rwlock range;if (kvm_is_radix (kvm))return kvmppc_book3s_radix_page_fault (run, vcpu, ea, dsisr);
@@ -568,7 +569,8 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, npages =get_user_pages_fast (hva,1 , writing, pages);if (npages <1 ) {/* Check if it's an I/O mapping */ -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); vma =find_vma (current->mm, hva);if (vma && vma->vm_start <= hva && hva + psize <= vma->vm_end && (vma->vm_flags &VM_PFNMAP )) {
@@ -578,7 +580,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, is_ci =pte_ci (__pte ((pgprot_val (vma->vm_page_prot)))); write_ok = vma->vm_flags &VM_WRITE ; } -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (!pfn) goto out_put; }else {
arch/powerpc/kvm/book3s_64_mmu_radix.c +4 -2
@@ -305,6 +305,7 @@ int kvmppc_book3s_radix_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,pte_t pte, *ptep;unsigned long pgflags;unsigned int shift, level; +struct range_rwlock range;/* Check for unusual errors */ if (dsisr &DSISR_UNSUPP_MMU ) {
@@ -394,7 +395,8 @@ int kvmppc_book3s_radix_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, npages =get_user_pages_fast (hva,1 , writing, pages);if (npages <1 ) {/* Check if it's an I/O mapping */ -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); vma =find_vma (current->mm, hva);if (vma && vma->vm_start <= hva && hva < vma->vm_end && (vma->vm_flags &VM_PFNMAP )) {
@@ -402,7 +404,7 @@ int kvmppc_book3s_radix_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, ((hva - vma->vm_start) >>PAGE_SHIFT ); pgflags =pgprot_val (vma->vm_page_prot); } -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (!pfn)return -EFAULT ; }else {
arch/powerpc/kvm/book3s_64_vio.c +4 -2
@@ -57,11 +57,13 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)static long kvmppc_account_memlimit (unsigned long stt_pages,bool inc) {long ret =0 ; +struct range_rwlock range;if (!current || !current->mm)return ret;/* process exited */ -down_write (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (¤t->mm->mmap_rw_tree, &range);if (inc) {unsigned long locked, lock_limit;
@@ -86,7 +88,7 @@ static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)rlimit (RLIMIT_MEMLOCK ), ret ? " - exceeded" : ""); -up_write (¤t->mm->mmap_sem); +range_write_unlock (¤t->mm->mmap_rw_tree, &range);return ret; }
arch/powerpc/kvm/book3s_hv.c +5 -3
@@ -3192,6 +3192,7 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu)unsigned long lpcr =0 , senc;unsigned long psize, porder;int srcu_idx; +struct range_rwlock range;mutex_lock (&kvm->lock);if (kvm->arch.hpte_setup_done)
@@ -3228,7 +3229,8 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu)/* Look up the VMA for the start of this memory slot */ hva = memslot->userspace_addr; -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); vma =find_vma (current->mm, hva);if (!vma || vma->vm_start > hva || (vma->vm_flags &VM_IO )) goto up_out;
@@ -3236,7 +3238,7 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu) psize =vma_kernel_pagesize (vma); porder =__ilog2 (psize); -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);/* We can handle 4k, 64k or 16M pages in the VRMA */ err = -EINVAL ;
@@ -3270,7 +3272,7 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu)return err; up_out: -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); goto out_srcu; }
arch/powerpc/kvm/e500_mmu_host.c +5 -2
@@ -357,7 +357,10 @@ static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500,if (tlbsel ==1 ) {struct vm_area_struct *vma; -down_read (¤t->mm->mmap_sem); +struct range_rwlock range; + +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); vma =find_vma (current->mm, hva); if (vma && hva >= vma->vm_start &&
@@ -443,7 +446,7 @@ static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500, tsize =max (BOOK3E_PAGESZ_4K , tsize & ~1 ); } -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); }if (likely (!pfnmap)) {
arch/powerpc/mm/copro_fault.c +4 -2
@@ -37,6 +37,7 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea, unsigned long dsisr, unsigned *flt) { struct vm_area_struct *vma; +struct range_rwlock range;unsigned long is_write;int ret;
@@ -46,7 +47,8 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,if (mm->pgd ==NULL )return -EFAULT ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); ret = -EFAULT ; vma =find_vma (mm, ea);if (!vma)
@@ -95,7 +97,7 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea, current->min_flt++; out_unlock: -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return ret; }EXPORT_SYMBOL_GPL (copro_handle_mm_fault);
arch/powerpc/mm/fault.c +7 -5
@@ -208,6 +208,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,int is_exec = trap ==0x400 ;int fault;int rc =0 , store_update_sp =0 ; +struct range_rwlock range;#if !(defined(CONFIG_4xx) || defined(CONFIG_BOOKE ))/*
@@ -308,12 +309,13 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,* source. If this is invalid we can skip the address space check, * thus avoiding the deadlock. */ -if (!down_read_trylock (&mm->mmap_sem)) { +range_rwlock_init_full (&range);/* XXX finer grain required here */ +if (!range_read_trylock (&mm->mmap_rw_tree, &range)) {if (!user_mode (regs) && !search_exception_tables (regs->nip)) goto bad_area_nosemaphore; retry: -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); }else {/* * The above down_read_trylock() might have succeeded in
@@ -446,7 +448,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, &range);/* * Handle the retry right now, the mmap_sem has been released in that
@@ -466,7 +468,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address, }/* We will enter mm_fault_error() below */ }else -up_read (¤t->mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);if (unlikely (fault & (VM_FAULT_RETRY |VM_FAULT_ERROR ))) {if (fault &VM_FAULT_SIGSEGV )
@@ -505,7 +507,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address, goto bail; bad_area: -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); bad_area_nosemaphore:/* User mode accesses cause a SIGSEGV */
arch/powerpc/mm/mmu_context_iommu.c +4 -2
@@ -36,11 +36,13 @@ static long mm_iommu_adjust_locked_vm(struct mm_struct *mm, unsigned long npages, bool incr) { long ret = 0, locked, lock_limit; +struct range_rwlock range;if (!npages)return 0 ; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range);if (incr) { locked = mm->locked_vm + npages;
@@ -61,7 +63,7 @@ static long mm_iommu_adjust_locked_vm(struct mm_struct *mm, npages <<PAGE_SHIFT , mm->locked_vm <<PAGE_SHIFT , rlimit(RLIMIT_MEMLOCK)); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }
arch/powerpc/mm/subpage-prot.c +10 -6
@@ -98,8 +98,10 @@ static void subpage_prot_clear(unsigned long addr, unsigned long len)unsigned long i;size_t nw;unsigned long next, limit; +struct range_rwlock range; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); limit = addr + len;if (limit > spt->maxaddr) limit = spt->maxaddr;
@@ -127,7 +129,7 @@ static void subpage_prot_clear(unsigned long addr, unsigned long len)/* now flush any existing HPTEs for the range */ hpte_flush_range (mm, addr, nw); } -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); }#ifdef CONFIG_TRANSPARENT_HUGEPAGE
@@ -194,6 +196,7 @@ long sys_subpage_prot(unsigned long addr, unsigned long len, u32 __user *map)size_t nw;unsigned long next, limit;int err; +struct range_rwlock range;/* Check parameters */ if ((addr & ~PAGE_MASK) || (len & ~PAGE_MASK) ||
@@ -212,7 +215,8 @@ long sys_subpage_prot(unsigned long addr, unsigned long len, u32 __user *map)if (!access_ok (VERIFY_READ , map, (len >>PAGE_SHIFT ) *sizeof (u32)))return -EFAULT ; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range);subpage_mark_vma_nohuge (mm, addr, len);for (limit = addr + len; addr < limit; addr = next) { next =pmd_addr_end (addr, limit);
@@ -247,11 +251,11 @@ long sys_subpage_prot(unsigned long addr, unsigned long len, u32 __user *map)if (addr + (nw <<PAGE_SHIFT ) > next) nw = (next - addr) >>PAGE_SHIFT ; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (__copy_from_user (spp, map, nw *sizeof (u32)))return -EFAULT ; map += nw; -down_write (&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &range);/* now flush any existing HPTEs for the range */ hpte_flush_range (mm, addr, nw);
@@ -260,6 +264,6 @@ long sys_subpage_prot(unsigned long addr, unsigned long len, u32 __user *map) spt->maxaddr = limit; err =0 ; out: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return err; }
arch/powerpc/oprofile/cell/spu_task_sync.c +5 -3
@@ -325,6 +325,7 @@ get_exec_dcookie_and_offset(struct spu *spu, unsigned int *offsetp,struct vm_area_struct *vma;struct file *exe_file;struct mm_struct *mm = spu->mm; +struct range_rwlock range;if (!mm) goto out;
@@ -336,7 +337,8 @@ get_exec_dcookie_and_offset(struct spu *spu, unsigned int *offsetp,fput (exe_file); } -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (vma = mm->mmap; vma; vma = vma->vm_next) {if (vma->vm_start > spu_ref || vma->vm_end <= spu_ref)continue ;
@@ -353,13 +355,13 @@ get_exec_dcookie_and_offset(struct spu *spu, unsigned int *offsetp, *spu_bin_dcookie =fast_get_dcookie (&vma->vm_file->f_path);pr_debug ("got dcookie for %pD\n" , vma->vm_file); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); out:return app_cookie; fail_no_image_cookie: -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); printk(KERN_ERR "SPU_PROF: " "%s, line %d: Cannot find dcookie for SPU binary\n",
arch/powerpc/platforms/cell/spufs/file.c +2 -2
@@ -347,11 +347,11 @@ static int spufs_ps_fault(struct vm_fault *vmf, goto refault;if (ctx->state ==SPU_STATE_SAVED ) { -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, vmf->lockrange);spu_context_nospu_trace (spufs_ps_fault__sleep, ctx); ret =spufs_wait (ctx->run_wq, ctx->state ==SPU_STATE_RUNNABLE );spu_context_trace (spufs_ps_fault__wake, ctx, ctx->spu); -down_read (¤t->mm->mmap_sem); +range_read_lock (¤t->mm->mmap_rw_tree, vmf->lockrange); }else { area = ctx->spu->problem_phys + ps_offs;vm_insert_pfn (vmf->vma, vmf->address, (area + offset) >>PAGE_SHIFT );
arch/x86/entry/vdso/vma.c +9 -5
@@ -148,10 +148,12 @@ static int map_vdso(const struct vdso_image *image, unsigned long addr) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; +struct range_rwlock range;unsigned long text_start;int ret =0 ; -if (down_write_killable (&mm->mmap_sem)) +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return -EINTR ; addr = get_unmapped_area(NULL, addr,
@@ -194,7 +196,7 @@ static int map_vdso(const struct vdso_image *image, unsigned long addr) } up_fail: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }
@@ -255,8 +257,10 @@ int map_vdso_once(const struct vdso_image *image, unsigned long addr) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; +struct range_rwlock range; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range);/* * Check if we have already mapped vdso blob - fail to prevent * abusing from userspace install_speciall_mapping, which may
@@ -267,11 +271,11 @@ int map_vdso_once(const struct vdso_image *image, unsigned long addr)for (vma = mm->mmap; vma; vma = vma->vm_next) {if (vma_is_special_mapping (vma, &vdso_mapping) ||vma_is_special_mapping (vma, &vvar_mapping)) { -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return -EEXIST ; } } -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return map_vdso (image, addr); }
arch/x86/kernel/tboot.c +1 -1
@@ -104,7 +104,7 @@ static struct mm_struct tboot_mm = { .pgd = swapper_pg_dir, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), - .mmap_sem = __RWSEM_INITIALIZER(init_mm.mmap_sem), + .mmap_rw_tree = __RANGE_RWLOCK_TREE_INITIALIZER(init_mm.mmap_rw_tree), .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), };
arch/x86/kernel/vm86_32.c +4 -2
@@ -162,6 +162,7 @@ void save_v86_state(struct kernel_vm86_regs *regs, int retval)static void mark_screen_rdonly (struct mm_struct *mm) {struct vm_area_struct *vma; +struct range_rwlock range;spinlock_t *ptl;pgd_t *pgd;pud_t *pud;
@@ -169,7 +170,8 @@ static void mark_screen_rdonly(struct mm_struct *mm)pte_t *pte;int i; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); pgd =pgd_offset (mm,0xA0000 );if (pgd_none_or_clear_bad (pgd)) goto out;
@@ -192,7 +194,7 @@ static void mark_screen_rdonly(struct mm_struct *mm) }pte_unmap_unlock (pte, ptl); out: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);flush_tlb (); }
arch/x86/mm/fault.c +22 -17
@@ -922,7 +922,8 @@ bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code,static void __bad_area (struct pt_regs *regs,unsigned long error_code, -unsigned long address,struct vm_area_struct *vma,int si_code) +unsigned long address,struct vm_area_struct *vma,int si_code, +struct range_rwlock *range) {struct mm_struct *mm = current->mm;
@@ -930,15 +931,16 @@ __bad_area(struct pt_regs *regs, unsigned long error_code,* Something tried to access memory that isn't in our memory map.. * Fix it, but check if it's kernel or user first.. */ -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, range);__bad_area_nosemaphore (regs, error_code, address, vma, si_code); } static noinline void -bad_area (struct pt_regs *regs,unsigned long error_code,unsigned long address) +bad_area (struct pt_regs *regs,unsigned long error_code,unsigned long address, +struct range_rwlock *range) { -__bad_area (regs, error_code, address,NULL ,SEGV_MAPERR ); +__bad_area (regs, error_code, address,NULL ,SEGV_MAPERR , range); } static inline bool bad_area_access_from_pkeys(unsigned long error_code,
@@ -960,7 +962,8 @@ static inline bool bad_area_access_from_pkeys(unsigned long error_code, static noinline voidbad_area_access_error (struct pt_regs *regs,unsigned long error_code, -unsigned long address,struct vm_area_struct *vma) +unsigned long address,struct vm_area_struct *vma, +struct range_rwlock *range) {/* * This OSPKE check is not strictly necessary at runtime.
@@ -968,9 +971,9 @@ bad_area_access_error(struct pt_regs *regs, unsigned long error_code,* if pkeys are compiled out. */ if (bad_area_access_from_pkeys (error_code, vma)) -__bad_area (regs, error_code, address, vma,SEGV_PKUERR ); +__bad_area (regs, error_code, address, vma,SEGV_PKUERR , range);else -__bad_area (regs, error_code, address, vma,SEGV_ACCERR ); +__bad_area (regs, error_code, address, vma,SEGV_ACCERR , range); } static void
@@ -1218,6 +1221,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,struct vm_area_struct *vma;struct task_struct *tsk;struct mm_struct *mm; +struct range_rwlock range;int fault, major =0 ;unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ;
@@ -1230,7 +1234,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,*/ if (kmemcheck_active (regs))kmemcheck_hide (regs); -prefetchw (&mm->mmap_sem); +prefetchw (&mm->mmap_rw_tree);if (unlikely (kmmio_fault (regs, address)))return ;
@@ -1333,14 +1337,15 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,* validate the source. If this is invalid we can skip the address * space check, thus avoiding the deadlock: */ -if (unlikely (!down_read_trylock (&mm->mmap_sem))) { +range_rwlock_init_full (&range); +if (unlikely (!range_read_trylock (&mm->mmap_rw_tree, &range))) {if ((error_code &PF_USER ) ==0 && !search_exception_tables (regs->ip)) {bad_area_nosemaphore (regs, error_code, address,NULL );return ; } retry: -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); }else {/* * The above down_read_trylock() might have succeeded in
@@ -1352,13 +1357,13 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, vma =find_vma (mm, address);if (unlikely (!vma)) { -bad_area (regs, error_code, address); +bad_area (regs, error_code, address, &range);return ; }if (likely (vma->vm_start <= address)) goto good_area;if (unlikely (!(vma->vm_flags &VM_GROWSDOWN ))) { -bad_area (regs, error_code, address); +bad_area (regs, error_code, address, &range);return ; }if (error_code &PF_USER ) {
@@ -1369,12 +1374,12 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,* 32 pointers and then decrements %sp by 65535.) */ if (unlikely (address +65536 +32 *sizeof (unsigned long ) < regs->sp)) { -bad_area (regs, error_code, address); +bad_area (regs, error_code, address, &range);return ; } }if (unlikely (expand_stack (vma, address))) { -bad_area (regs, error_code, address); +bad_area (regs, error_code, address, &range);return ; }
@@ -1384,7 +1389,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,*/ good_area:if (unlikely (access_error (error_code, vma))) { -bad_area_access_error (regs, error_code, address, vma); +bad_area_access_error (regs, error_code, address, vma, &range);return ; }
@@ -1394,7 +1399,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,* the fault. Since we never set FAULT_FLAG_RETRY_NOWAIT, if * we get VM_FAULT_RETRY back, the mmap_sem has been unlocked. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, &range); major |= fault &VM_FAULT_MAJOR ;/*
@@ -1420,7 +1425,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,return ; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);if (unlikely (fault &VM_FAULT_ERROR )) {mm_fault_error (regs, error_code, address, vma, fault);return ;
arch/x86/mm/mpx.c +12 -6
@@ -44,16 +44,18 @@ static inline unsigned long mpx_bt_size_bytes(struct mm_struct *mm)static unsigned long mpx_mmap (unsigned long len) {struct mm_struct *mm = current->mm; +struct range_rwlock range;unsigned long addr, populate;/* Only bounds table can be allocated here */ if (len !=mpx_bt_size_bytes (mm))return -EINVAL ; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); addr =do_mmap (NULL ,0 , len,PROT_READ |PROT_WRITE ,MAP_ANONYMOUS |MAP_PRIVATE ,VM_MPX ,0 , &populate,NULL ); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (populate)mm_populate (addr, populate);
@@ -340,6 +342,7 @@ int mpx_enable_management(void) {void __user *bd_base =MPX_INVALID_BOUNDS_DIR ;struct mm_struct *mm = current->mm; +struct range_rwlock range;int ret =0 ;/*
@@ -354,25 +357,28 @@ int mpx_enable_management(void)* unmap path; we can just use mm->context.bd_addr instead. */ bd_base =mpx_get_bounds_dir (); -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); mm->context.bd_addr = bd_base;if (mm->context.bd_addr ==MPX_INVALID_BOUNDS_DIR ) ret = -ENXIO ; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }int mpx_disable_management (void ) {struct mm_struct *mm = current->mm; +struct range_rwlock range;if (!cpu_feature_enabled (X86_FEATURE_MPX ))return -ENXIO ; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); mm->context.bd_addr =MPX_INVALID_BOUNDS_DIR ; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return 0 ; }
drivers/android/binder.c +5 -3
@@ -581,6 +581,7 @@ static int binder_update_page_range(struct binder_proc *proc, int allocate,unsigned long user_page_addr;struct page **page;struct mm_struct *mm; +struct range_rwlock range; binder_debug(BINDER_DEBUG_BUFFER_ALLOC, "%d: %s pages %p-%p\n", proc->pid,
@@ -597,7 +598,8 @@ static int binder_update_page_range(struct binder_proc *proc, int allocate, mm =get_task_mm (proc->tsk);if (mm) { -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); vma = proc->vma;if (vma && mm != proc->vma_vm_mm) { pr_err("%d: vma mm and task mm mismatch\n",
@@ -647,7 +649,7 @@ static int binder_update_page_range(struct binder_proc *proc, int allocate,/* vm_insert_page does not seem to increment the refcount */ }if (mm) { -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm); }return 0 ;
@@ -669,7 +671,7 @@ static int binder_update_page_range(struct binder_proc *proc, int allocate, } err_no_vma:if (mm) { -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm); }return -ENOMEM ;
drivers/firmware/efi/arm-runtime.c +1 -1
@@ -34,7 +34,7 @@ static struct mm_struct efi_mm = { .mm_rb = RB_ROOT, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), - .mmap_sem = __RWSEM_INITIALIZER(efi_mm.mmap_sem), + .mmap_rw_tree = __RANGE_RWLOCK_TREE_INITIALIZER(efi_mm.mmap_rw_tree), .page_table_lock = __SPIN_LOCK_UNLOCKED(efi_mm.page_table_lock), .mmlist = LIST_HEAD_INIT(efi_mm.mmlist), };
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +6 -3
@@ -509,6 +509,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,struct amdgpu_fpriv *fpriv = p->filp->driver_priv;struct amdgpu_bo_list_entry *e;struct list_head duplicates; +struct range_rwlock range;bool need_mmap_lock = false;unsigned i, tries =10 ;int r;
@@ -528,8 +529,10 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,if (p->uf_entry.robj)list_add (&p->uf_entry.tv.head, &p->validated); -if (need_mmap_lock) -down_read (¤t->mm->mmap_sem); +if (need_mmap_lock) { +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); + }while (1 ) {struct list_head need_pages;
@@ -686,7 +689,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, error_free_pages:if (need_mmap_lock) -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (p->bo_list) { for (i = p->bo_list->first_userptr;
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +5 -3
@@ -269,6 +269,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,struct drm_amdgpu_gem_userptr *args = data;struct drm_gem_object *gobj;struct amdgpu_bo *bo; +struct range_rwlock range;uint32_t handle;int r;
@@ -309,7 +310,8 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data, }if (args->flags &AMDGPU_GEM_USERPTR_VALIDATE ) { -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); r =amdgpu_ttm_tt_get_user_pages (bo->tbo.ttm, bo->tbo.ttm->pages);
@@ -326,7 +328,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,if (r) goto free_pages; -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); } r =drm_gem_handle_create (filp, gobj, &handle);
@@ -342,7 +344,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,release_pages (bo->tbo.ttm->pages, bo->tbo.ttm->num_pages, false); unlock_mmap_sem: -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); release_object:drm_gem_object_unreference_unlocked (gobj);
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +5 -3
@@ -229,10 +229,12 @@ static struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev) {struct mm_struct *mm = current->mm;struct amdgpu_mn *rmn; +struct range_rwlock range;int r; +range_rwlock_init_full (&range);mutex_lock (&adev->mn_lock); -if (down_write_killable (&mm->mmap_sem)) { +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range)) {mutex_unlock (&adev->mn_lock);return ERR_PTR (-EINTR ); }
@@ -260,13 +262,13 @@ static struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev)hash_add (adev->mn_hash, &rmn->node, (unsigned long )mm); release_locks: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mutex_unlock (&adev->mn_lock);return rmn; free_rmn: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mutex_unlock (&adev->mn_lock);kfree (rmn);
drivers/gpu/drm/amd/amdkfd/kfd_events.c +4 -2
@@ -897,6 +897,7 @@ void kfd_signal_iommu_event(struct kfd_dev *dev, unsigned int pasid, {struct kfd_hsa_memory_exception_data memory_exception_data;struct vm_area_struct *vma; +struct range_rwlock range;/* * Because we are called from arbitrary context (workqueue) as opposed
@@ -910,7 +911,8 @@ void kfd_signal_iommu_event(struct kfd_dev *dev, unsigned int pasid,memset (&memory_exception_data,0 ,sizeof (memory_exception_data)); -down_read (&p->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&p->mm->mmap_rw_tree, &range); vma =find_vma (p->mm, address); memory_exception_data.gpu_id = dev->id;
@@ -937,7 +939,7 @@ void kfd_signal_iommu_event(struct kfd_dev *dev, unsigned int pasid, } } -up_read (&p->mm->mmap_sem); +range_read_unlock (&p->mm->mmap_rw_tree, &range);mutex_lock (&p->event_mutex);
drivers/gpu/drm/amd/amdkfd/kfd_process.c +4 -2
@@ -78,6 +78,7 @@ void kfd_process_destroy_wq(void)struct kfd_process *kfd_create_process (const struct task_struct *thread) {struct kfd_process *process; +struct range_rwlock range;BUG_ON (!kfd_process_wq);
@@ -89,7 +90,8 @@ struct kfd_process *kfd_create_process(const struct task_struct *thread)return ERR_PTR (-EINVAL );/* Take mmap_sem because we call __mmu_notifier_register inside */ -down_write (&thread->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&thread->mm->mmap_rw_tree, &range);/* * take kfd processes mutex before starting of process creation
@@ -108,7 +110,7 @@ struct kfd_process *kfd_create_process(const struct task_struct *thread)mutex_unlock (&kfd_processes_mutex); -up_write (&thread->mm->mmap_sem); +range_write_unlock (&thread->mm->mmap_rw_tree, &range);return process; }
drivers/gpu/drm/etnaviv/etnaviv_gem.c +4 -2
@@ -745,6 +745,7 @@ static struct page **etnaviv_gem_userptr_do_get_pages( {int ret =0 , pinned, npages = etnaviv_obj->base.size >>PAGE_SHIFT ;struct page **pvec; +struct range_rwlock range;uintptr_t ptr;unsigned int flags =0 ;
@@ -758,7 +759,8 @@ static struct page **etnaviv_gem_userptr_do_get_pages( pinned =0 ; ptr = etnaviv_obj->userptr.ptr; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);while (pinned < npages) { ret = get_user_pages_remote(task, mm, ptr, npages - pinned, flags, pvec + pinned, NULL, NULL,
@@ -769,7 +771,7 @@ static struct page **etnaviv_gem_userptr_do_get_pages( ptr += ret *PAGE_SIZE ; pinned += ret; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);if (ret <0 ) {release_pages (pvec, pinned,0 );
drivers/gpu/drm/i915/i915_gem.c +4 -2
@@ -1673,8 +1673,10 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,if (args->flags &I915_MMAP_WC ) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; +struct range_rwlock range; -if (down_write_killable (&mm->mmap_sem)) { +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range)) {i915_gem_object_put (obj);return -EINTR ; }
@@ -1684,7 +1686,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,pgprot_writecombine (vm_get_page_prot (vma->vm_flags)); else addr = -ENOMEM ; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);/* This may race, but that's ok, it only gets set */ WRITE_ONCE (obj->frontbuffer_ggtt_origin,ORIGIN_CPU );
drivers/gpu/drm/i915/i915_gem_userptr.c +8 -4
@@ -203,12 +203,14 @@ static struct i915_mmu_notifier * i915_mmu_notifier_find(struct i915_mm_struct *mm) {struct i915_mmu_notifier *mn = mm->mn; +struct range_rwlock range; mn = mm->mn;if (mn)return mn; -down_write (&mm->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mm->mmap_rw_tree, &range);mutex_lock (&mm->i915->mm_lock);if ((mn = mm->mn) ==NULL ) { mn =i915_mmu_notifier_create (mm->mm);
@@ -216,7 +218,7 @@ i915_mmu_notifier_find(struct i915_mm_struct *mm) mm->mn = mn; }mutex_unlock (&mm->i915->mm_lock); -up_write (&mm->mm->mmap_sem); +range_write_unlock (&mm->mm->mmap_rw_tree, &range);return mn; }
@@ -501,6 +503,7 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work) pvec =drm_malloc_gfp (npages,sizeof (struct page *),GFP_TEMPORARY );if (pvec !=NULL ) { +struct range_rwlock range;struct mm_struct *mm = obj->userptr.mm->mm;unsigned int flags =0 ;
@@ -509,7 +512,8 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work) ret = -EFAULT ;if (mmget_not_zero (mm)) { -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);while (pinned < npages) { ret = get_user_pages_remote (work->task, mm,
@@ -522,7 +526,7 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work) pinned += ret; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);mmput (mm); } }
drivers/gpu/drm/radeon/radeon_cs.c +6 -3
@@ -76,6 +76,7 @@ static int radeon_cs_parser_relocs(struct radeon_cs_parser *p) {struct radeon_cs_chunk *chunk;struct radeon_cs_buckets buckets; +struct range_rwlock range;unsigned i;bool need_mmap_lock = false;int r;
@@ -176,13 +177,15 @@ static int radeon_cs_parser_relocs(struct radeon_cs_parser *p)if (p->cs_flags &RADEON_CS_USE_VM ) p->vm_bos =radeon_vm_get_bos (p->rdev, p->ib.vm, &p->validated); -if (need_mmap_lock) -down_read (¤t->mm->mmap_sem); +if (need_mmap_lock) { +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); + } r =radeon_bo_list_validate (p->rdev, &p->ticket, &p->validated, p->ring);if (need_mmap_lock) -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);return r; }
drivers/gpu/drm/radeon/radeon_gem.c +5 -3
@@ -285,6 +285,7 @@ int radeon_gem_userptr_ioctl(struct drm_device *dev, void *data,struct drm_radeon_gem_userptr *args = data;struct drm_gem_object *gobj;struct radeon_bo *bo; +struct range_rwlock range;uint32_t handle;int r;
@@ -331,17 +332,18 @@ int radeon_gem_userptr_ioctl(struct drm_device *dev, void *data, }if (args->flags &RADEON_GEM_USERPTR_VALIDATE ) { -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); r =radeon_bo_reserve (bo, true);if (r) { -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); goto release_object; }radeon_ttm_placement_from_domain (bo,RADEON_GEM_DOMAIN_GTT ); r =ttm_bo_validate (&bo->tbo, &bo->placement, true, false);radeon_bo_unreserve (bo); -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (r) goto release_object; }
drivers/gpu/drm/radeon/radeon_mn.c +5 -3
@@ -184,9 +184,11 @@ static struct radeon_mn *radeon_mn_get(struct radeon_device *rdev) {struct mm_struct *mm = current->mm;struct radeon_mn *rmn; +struct range_rwlock range;int r; -if (down_write_killable (&mm->mmap_sem)) +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return ERR_PTR (-EINTR );mutex_lock (&rdev->mn_lock);
@@ -215,13 +217,13 @@ static struct radeon_mn *radeon_mn_get(struct radeon_device *rdev) release_locks:mutex_unlock (&rdev->mn_lock); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return rmn; free_rmn:mutex_unlock (&rdev->mn_lock); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);kfree (rmn);return ERR_PTR (r);
drivers/gpu/drm/ttm/ttm_bo_vm.c +4 -2
@@ -66,7 +66,8 @@ static int ttm_bo_vm_fault_idle(struct ttm_buffer_object *bo, goto out_unlock;ttm_bo_reference (bo); -up_read (&vmf->vma->vm_mm->mmap_sem); +range_read_unlock (&vmf->vma->vm_mm->mmap_rw_tree, + vmf->lockrange); (void )dma_fence_wait (bo->moving, true);ttm_bo_unreserve (bo);ttm_bo_unref (&bo);
@@ -124,7 +125,8 @@ static int ttm_bo_vm_fault(struct vm_fault *vmf)if (vmf->flags &FAULT_FLAG_ALLOW_RETRY ) {if (!(vmf->flags &FAULT_FLAG_RETRY_NOWAIT )) {ttm_bo_reference (bo); -up_read (&vmf->vma->vm_mm->mmap_sem); +range_read_unlock (&vmf->vma->vm_mm->mmap_rw_tree, + vmf->lockrange); (void )ttm_bo_wait_unreserved (bo);ttm_bo_unref (&bo); }
drivers/gpu/drm/via/via_dmablit.c +4 -2
@@ -230,6 +230,7 @@ via_fire_dmablit(struct drm_device *dev, drm_via_sg_info_t *vsg, int engine)static int via_lock_all_dma_pages (drm_via_sg_info_t *vsg,drm_via_dmablit_t *xfer) { +struct range_rwlock range;int ret;unsigned long first_pfn =VIA_PFN (xfer->mem_addr); vsg->num_pages = VIA_PFN(xfer->mem_addr + (xfer->num_lines * xfer->mem_stride - 1)) -
@@ -238,13 +239,14 @@ via_lock_all_dma_pages(drm_via_sg_info_t *vsg, drm_via_dmablit_t *xfer) vsg->pages =vzalloc (sizeof (struct page *) * vsg->num_pages);if (NULL == vsg->pages)return -ENOMEM ; -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); ret =get_user_pages ((unsigned long )xfer->mem_addr, vsg->num_pages, (vsg->direction ==DMA_FROM_DEVICE ) ?FOLL_WRITE :0 , vsg->pages,NULL ); -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (ret != vsg->num_pages) {if (ret <0 )return ret;
drivers/infiniband/core/umem.c +13 -7
@@ -86,6 +86,7 @@ struct ib_umem *ib_umem_get(struct ib_ucontext *context, unsigned long addr,struct ib_umem *umem;struct page **page_list;struct vm_area_struct **vma_list; +struct range_rwlock range;unsigned long locked;unsigned long lock_limit;unsigned long cur_base;
@@ -163,7 +164,8 @@ struct ib_umem *ib_umem_get(struct ib_ucontext *context, unsigned long addr, npages =ib_umem_num_pages (umem); -down_write (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (¤t->mm->mmap_rw_tree, &range); locked = npages + current->mm->pinned_vm; lock_limit =rlimit (RLIMIT_MEMLOCK ) >>PAGE_SHIFT ;
@@ -236,7 +238,7 @@ struct ib_umem *ib_umem_get(struct ib_ucontext *context, unsigned long addr, }else current->mm->pinned_vm = locked; -up_write (¤t->mm->mmap_sem); +range_write_unlock (¤t->mm->mmap_rw_tree, &range);if (vma_list)free_page ((unsigned long ) vma_list);free_page ((unsigned long ) page_list);
@@ -248,10 +250,12 @@ EXPORT_SYMBOL(ib_umem_get);static void ib_umem_account (struct work_struct *work) { struct ib_umem *umem = container_of(work, struct ib_umem, work); +struct range_rwlock range; -down_write (&umem->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&umem->mm->mmap_rw_tree, &range); umem->mm->pinned_vm -= umem->diff; -up_write (&umem->mm->mmap_sem); +range_write_unlock (&umem->mm->mmap_rw_tree, &range);mmput (umem->mm);kfree (umem); }
@@ -265,6 +269,7 @@ void ib_umem_release(struct ib_umem *umem)struct ib_ucontext *context = umem->context;struct mm_struct *mm;struct task_struct *task; +struct range_rwlock range;unsigned long diff;if (umem->odp_data) {
@@ -293,8 +298,9 @@ void ib_umem_release(struct ib_umem *umem)* up here and not be able to take the mmap_sem. In that case * we defer the vm_locked accounting to the system workqueue. */ +range_rwlock_init_full (&range);if (context->closing) { -if (!down_write_trylock (&mm->mmap_sem)) { +if (!range_write_trylock (&mm->mmap_rw_tree, &range)) {INIT_WORK (&umem->work, ib_umem_account); umem->mm = mm; umem->diff = diff;
@@ -303,10 +309,10 @@ void ib_umem_release(struct ib_umem *umem)return ; } }else -down_write (&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &range); mm->pinned_vm -= diff; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm); out:kfree (umem);
drivers/infiniband/core/umem_odp.c +4 -2
@@ -595,6 +595,7 @@ int ib_umem_odp_map_dma_pages(struct ib_umem *umem, u64 user_virt, u64 bcnt,struct task_struct *owning_process =NULL ;struct mm_struct *owning_mm =NULL ;struct page **local_page_list =NULL ; +struct range_rwlock range;u64 off;int j, k, ret =0 , start_idx, npages =0 ;u64 base_virt_addr;
@@ -639,7 +640,8 @@ int ib_umem_odp_map_dma_pages(struct ib_umem *umem, u64 user_virt, u64 bcnt,min_t (size_t,ALIGN (bcnt,PAGE_SIZE ) /PAGE_SIZE ,PAGE_SIZE /sizeof (struct page *)); -down_read (&owning_mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&owning_mm->mmap_rw_tree, &range);/* * Note: this might result in redundent page getting. We can * avoid this by checking dma_list to be 0 before calling
@@ -650,7 +652,7 @@ int ib_umem_odp_map_dma_pages(struct ib_umem *umem, u64 user_virt, u64 bcnt, npages =get_user_pages_remote (owning_process, owning_mm, user_virt, gup_num_pages, flags, local_page_list,NULL ,NULL ,NULL ); -up_read (&owning_mm->mmap_sem); +range_read_unlock (&owning_mm->mmap_rw_tree, &range);if (npages <0 )break ;
drivers/infiniband/hw/hfi1/user_pages.c +12 -6
@@ -71,6 +71,7 @@ MODULE_PARM_DESC(cache_size, "Send and receive side cache size limit (in MB)");bool hfi1_can_pin_pages (struct hfi1_devdata *dd,struct mm_struct *mm,u32 nlocked,u32 npages) { +struct range_rwlock range;unsigned long ulimit =rlimit (RLIMIT_MEMLOCK ), pinned, cache_limit, size = (cache_size * (1UL <<20 ));/* convert to bytes */ unsigned usr_ctxts = dd->num_rcv_contexts - dd->first_user_ctxt;
@@ -90,9 +91,10 @@ bool hfi1_can_pin_pages(struct hfi1_devdata *dd, struct mm_struct *mm,/* Convert to number of pages */ size =DIV_ROUND_UP (size,PAGE_SIZE ); -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); pinned = mm->pinned_vm; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);/* First, check the absolute limit against all pinned pages. */ if (pinned + npages >= ulimit && !can_lock)
@@ -104,15 +106,17 @@ bool hfi1_can_pin_pages(struct hfi1_devdata *dd, struct mm_struct *mm,int hfi1_acquire_user_pages (struct mm_struct *mm,unsigned long vaddr,size_t npages,bool writable,struct page **pages) { +struct range_rwlock range;int ret; ret =get_user_pages_fast (vaddr, npages, writable, pages);if (ret <0 )return ret; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); mm->pinned_vm += ret; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }
@@ -120,6 +124,7 @@ int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t npvoid hfi1_release_user_pages (struct mm_struct *mm,struct page **p,size_t npages,bool dirty) { +struct range_rwlock range;size_t i;for (i =0 ; i < npages; i++) {
@@ -129,8 +134,9 @@ void hfi1_release_user_pages(struct mm_struct *mm, struct page **p, }if (mm) {/* during close after signal, mm can be NULL */ -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); mm->pinned_vm -= npages; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); } }
drivers/infiniband/hw/mlx4/main.c +4 -2
@@ -1142,6 +1142,7 @@ static void mlx4_ib_disassociate_ucontext(struct ib_ucontext *ibcontext)struct mlx4_ib_ucontext *context =to_mucontext (ibcontext);struct task_struct *owning_process =NULL ;struct mm_struct *owning_mm =NULL ; +struct range_rwlock range; owning_process =get_pid_task (ibcontext->tgid,PIDTYPE_PID );if (!owning_process)
@@ -1173,7 +1174,8 @@ static void mlx4_ib_disassociate_ucontext(struct ib_ucontext *ibcontext)/* need to protect from a race on closing the vma as part of * mlx4_ib_vma_close(). */ -down_read (&owning_mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&owning_mm->mmap_rw_tree, &range);for (i =0 ; i <HW_BAR_COUNT ; i++) { vma = context->hw_bar_info[i].vma;if (!vma)
@@ -1191,7 +1193,7 @@ static void mlx4_ib_disassociate_ucontext(struct ib_ucontext *ibcontext) context->hw_bar_info[i].vma->vm_ops =NULL ; } -up_read (&owning_mm->mmap_sem); +range_read_unlock (&owning_mm->mmap_rw_tree, &range);mmput (owning_mm);put_task_struct (owning_process); }
drivers/infiniband/hw/mlx5/main.c +4 -2
@@ -1449,6 +1449,7 @@ static void mlx5_ib_disassociate_ucontext(struct ib_ucontext *ibcontext)struct mlx5_ib_ucontext *context =to_mucontext (ibcontext);struct task_struct *owning_process =NULL ;struct mm_struct *owning_mm =NULL ; +struct range_rwlock range; owning_process =get_pid_task (ibcontext->tgid,PIDTYPE_PID );if (!owning_process)
@@ -1478,7 +1479,8 @@ static void mlx5_ib_disassociate_ucontext(struct ib_ucontext *ibcontext)/* need to protect from a race on closing the vma as part of * mlx5_ib_vma_close. */ -down_read (&owning_mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&owning_mm->mmap_rw_tree, &range);list_for_each_entry_safe (vma_private, n, &context->vma_private_list, list) { vma = vma_private->vma;
@@ -1492,7 +1494,7 @@ static void mlx5_ib_disassociate_ucontext(struct ib_ucontext *ibcontext)list_del (&vma_private->list);kfree (vma_private); } -up_read (&owning_mm->mmap_sem); +range_read_unlock (&owning_mm->mmap_rw_tree, &range);mmput (owning_mm);put_task_struct (owning_process); }
drivers/infiniband/hw/qib/qib_user_pages.c +11 -5
@@ -133,26 +133,32 @@ dma_addr_t qib_map_page(struct pci_dev *hwdev, struct page *page,int qib_get_user_pages (unsigned long start_page,size_t num_pages,struct page **p) { +struct range_rwlock range;int ret; -down_write (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (¤t->mm->mmap_rw_tree, &range); ret =__qib_get_user_pages (start_page, num_pages, p); -up_write (¤t->mm->mmap_sem); +range_write_unlock (¤t->mm->mmap_rw_tree, &range);return ret; }void qib_release_user_pages (struct page **p,size_t num_pages) { -if (current->mm)/* during close after signal, mm can be NULL */ -down_write (¤t->mm->mmap_sem); +struct range_rwlock range; + +if (current->mm) {/* during close after signal, mm can be NULL */ +range_rwlock_init_full (&range); +range_write_lock (¤t->mm->mmap_rw_tree, &range); + }__qib_release_user_pages (p, num_pages,1 );if (current->mm) { current->mm->pinned_vm -= num_pages; -up_write (¤t->mm->mmap_sem); +range_write_unlock (¤t->mm->mmap_rw_tree, &range); } }
drivers/infiniband/hw/usnic/usnic_uiom.c +13 -7
@@ -55,12 +55,14 @@ static struct workqueue_struct *usnic_uiom_wq;static void usnic_uiom_reg_account (struct work_struct *work) { +struct range_rwlock range;struct usnic_uiom_reg *umem =container_of (work, struct usnic_uiom_reg, work); -down_write (&umem->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&umem->mm->mmap_rw_tree, &range); umem->mm->locked_vm -= umem->diff; -up_write (&umem->mm->mmap_sem); +range_write_unlock (&umem->mm->mmap_rw_tree, &range);mmput (umem->mm);kfree (umem); }
@@ -103,6 +105,7 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable,struct page **page_list;struct scatterlist *sg;struct usnic_uiom_chunk *chunk; +struct range_rwlock range;unsigned long locked;unsigned long lock_limit;unsigned long cur_base;
@@ -125,7 +128,8 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, npages =PAGE_ALIGN (size + (addr & ~PAGE_MASK )) >>PAGE_SHIFT ; -down_write (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (¤t->mm->mmap_rw_tree, &range); locked = npages + current->mm->locked_vm; lock_limit =rlimit (RLIMIT_MEMLOCK ) >>PAGE_SHIFT ;
@@ -188,7 +192,7 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable,else current->mm->locked_vm = locked; -up_write (¤t->mm->mmap_sem); +range_write_unlock (¤t->mm->mmap_rw_tree, &range);free_page ((unsigned long ) page_list);return ret; }
@@ -423,6 +427,7 @@ struct usnic_uiom_reg *usnic_uiom_reg_get(struct usnic_uiom_pd *pd,void usnic_uiom_reg_release (struct usnic_uiom_reg *uiomr,int closing) {struct mm_struct *mm; +struct range_rwlock range;unsigned long diff;__usnic_uiom_reg_release (uiomr->pd, uiomr,1 );
@@ -443,8 +448,9 @@ void usnic_uiom_reg_release(struct usnic_uiom_reg *uiomr, int closing)* up here and not be able to take the mmap_sem. In that case * we defer the vm_locked accounting to the system workqueue. */ +range_rwlock_init_full (&range);if (closing) { -if (!down_write_trylock (&mm->mmap_sem)) { +if (!range_write_trylock (&mm->mmap_rw_tree, &range)) {INIT_WORK (&uiomr->work, usnic_uiom_reg_account); uiomr->mm = mm; uiomr->diff = diff;
@@ -453,10 +459,10 @@ void usnic_uiom_reg_release(struct usnic_uiom_reg *uiomr, int closing)return ; } }else -down_write (&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &range); current->mm->locked_vm -= diff; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm);kfree (uiomr); }
drivers/iommu/amd_iommu_v2.c +5 -3
@@ -518,6 +518,7 @@ static void do_fault(struct work_struct *work)int ret =VM_FAULT_ERROR ;unsigned int flags =0 ;struct mm_struct *mm; +struct range_rwlock range;u64 address; mm = fault->state->mm;
@@ -529,7 +530,8 @@ static void do_fault(struct work_struct *work) flags |=FAULT_FLAG_WRITE ; flags |=FAULT_FLAG_REMOTE ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_extend_vma (mm, address);if (!vma || address < vma->vm_start)/* failed to get a vma in the right range */
@@ -539,9 +541,9 @@ static void do_fault(struct work_struct *work)if (access_error (vma, fault)) goto out; - ret =handle_mm_fault (vma, address, flags); + ret =handle_mm_fault (vma, address, flags,NULL ); out: -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);if (ret &VM_FAULT_ERROR )/* failed to service fault */
drivers/iommu/intel-svm.c +4 -2
@@ -542,6 +542,7 @@ static irqreturn_t prq_event_thread(int irq, void *d)struct vm_area_struct *vma;struct page_req_dsc *req;struct qi_desc resp; +struct range_rwlock range;int ret, result;u64 address;
@@ -582,7 +583,8 @@ static irqreturn_t prq_event_thread(int irq, void *d)/* If the mm is already defunct, don't handle faults. */ if (!mmget_not_zero (svm->mm)) goto bad_req; -down_read (&svm->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&svm->mm->mmap_rw_tree, &range); vma =find_extend_vma (svm->mm, address);if (!vma || address < vma->vm_start) goto invalid;
@@ -597,7 +599,7 @@ static irqreturn_t prq_event_thread(int irq, void *d) result =QI_RESP_SUCCESS ; invalid: -up_read (&svm->mm->mmap_sem); +range_read_unlock (&svm->mm->mmap_rw_tree, &range);mmput (svm->mm); bad_req:/* Accounting for major/minor faults? */
drivers/media/v4l2-core/videobuf-core.c +6 -3
@@ -530,14 +530,17 @@ EXPORT_SYMBOL_GPL(videobuf_querybuf);int videobuf_qbuf (struct videobuf_queue *q,struct v4l2_buffer *b) {struct videobuf_buffer *buf; +struct range_rwlock range;enum v4l2_field field;unsigned long flags =0 ;int retval;MAGIC_CHECK (q->int_ops->magic,MAGIC_QTYPE_OPS ); -if (b->memory ==V4L2_MEMORY_MMAP ) -down_read (¤t->mm->mmap_sem); +if (b->memory ==V4L2_MEMORY_MMAP ) { +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); + }videobuf_queue_lock (q); retval = -EBUSY ;
@@ -624,7 +627,7 @@ int videobuf_qbuf(struct videobuf_queue *q, struct v4l2_buffer *b)videobuf_queue_unlock (q);if (b->memory ==V4L2_MEMORY_MMAP ) -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);return retval; }
drivers/media/v4l2-core/videobuf-dma-contig.c +4 -2
@@ -162,6 +162,7 @@ static int videobuf_dma_contig_user_get(struct videobuf_dma_contig_memory *mem, {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; +struct range_rwlock range;unsigned long prev_pfn, this_pfn;unsigned long pages_done, user_address;unsigned int offset;
@@ -171,7 +172,8 @@ static int videobuf_dma_contig_user_get(struct videobuf_dma_contig_memory *mem, mem->size =PAGE_ALIGN (vb->size + offset); ret = -EINVAL ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, vb->baddr);if (!vma)
@@ -203,7 +205,7 @@ static int videobuf_dma_contig_user_get(struct videobuf_dma_contig_memory *mem, } out_up: -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);return ret; }
drivers/media/v4l2-core/videobuf-dma-sg.c +4 -2
@@ -199,11 +199,13 @@ static int videobuf_dma_init_user_locked(struct videobuf_dmabuf *dma,static int videobuf_dma_init_user (struct videobuf_dmabuf *dma,int direction,unsigned long data,unsigned long size) { +struct range_rwlock range;int ret; -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); ret =videobuf_dma_init_user_locked (dma, direction, data, size); -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);return ret; }
drivers/misc/cxl/fault.c +4 -2
@@ -338,6 +338,7 @@ static void cxl_prefault_vma(struct cxl_context *ctx)struct vm_area_struct *vma;int rc;struct mm_struct *mm; +struct range_rwlock range; mm =get_mem_context (ctx);if (mm ==NULL ) {
@@ -346,7 +347,8 @@ static void cxl_prefault_vma(struct cxl_context *ctx)return ; } -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (vma = mm->mmap; vma; vma = vma->vm_next) {for (ea = vma->vm_start; ea < vma->vm_end; ea =next_segment (ea, slb.vsid)) {
@@ -361,7 +363,7 @@ static void cxl_prefault_vma(struct cxl_context *ctx) last_esid = slb.esid; } } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);mmput (mm); }
drivers/misc/mic/scif/scif_rma.c +11 -6
@@ -275,19 +275,22 @@ static inline int __scif_dec_pinned_vm_lock(struct mm_struct *mm, int nr_pages, bool try_lock) { + struct range_rwlock range; +if (!mm || !nr_pages || !scif_ulimit_check)return 0 ; +range_rwlock_init_full (&range);if (try_lock) { - if (!down_write_trylock(&mm->mmap_sem)) { +if (!range_write_trylock (&mm->mmap_rw_tree, &range)) {dev_err (scif_info.mdev.this_device,"%s %d err\n" , __func__, __LINE__);return -1 ; } }else { - down_write(&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &range); } mm->pinned_vm -= nr_pages; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return 0 ; }
@@ -1333,6 +1336,7 @@ int __scif_pin_pages(void *addr, size_t len, int *out_prot,int prot = *out_prot;int ulimit =0 ;struct mm_struct *mm =NULL ; +struct range_rwlock range;/* Unsupported flags */ if (map_flags & ~(SCIF_MAP_KERNEL |SCIF_MAP_ULIMIT ))
@@ -1386,11 +1390,12 @@ int __scif_pin_pages(void *addr, size_t len, int *out_prot, prot |=SCIF_PROT_WRITE ; retry: mm = current->mm; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range);if (ulimit) { err =__scif_check_inc_pinned_vm (mm, nr_pages);if (err) { -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); pinned_pages->nr_pages =0 ; goto error_unmap; }
@@ -1402,7 +1407,7 @@ int __scif_pin_pages(void *addr, size_t len, int *out_prot, (prot &SCIF_PROT_WRITE ) ?FOLL_WRITE :0 , pinned_pages->pages, NULL, NULL); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (nr_pages != pinned_pages->nr_pages) {if (try_upgrade) {if (ulimit)
drivers/oprofile/buffer_sync.c +9 -5
@@ -90,12 +90,14 @@ munmap_notify(struct notifier_block *self, unsigned long val, void *data)unsigned long addr = (unsigned long )data;struct mm_struct *mm = current->mm;struct vm_area_struct *mpnt; +struct range_rwlock range; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); mpnt =find_vma (mm, addr);if (mpnt && mpnt->vm_file && (mpnt->vm_flags &VM_EXEC )) { -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);/* To avoid latency problems, we only process the current CPU, * hoping that most samples for the task are on this CPU */
@@ -103,7 +105,7 @@ munmap_notify(struct notifier_block *self, unsigned long val, void *data)return 0 ; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return 0 ; }
@@ -255,8 +257,10 @@ lookup_dcookie(struct mm_struct *mm, unsigned long addr, off_t *offset) {unsigned long cookie =NO_COOKIE ;struct vm_area_struct *vma; +struct range_rwlock range; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (vma =find_vma (mm, addr); vma; vma = vma->vm_next) {if (addr < vma->vm_start || addr >= vma->vm_end)
@@ -276,7 +280,7 @@ lookup_dcookie(struct mm_struct *mm, unsigned long addr, off_t *offset)if (!vma) cookie =INVALID_COOKIE ; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return cookie; }
drivers/staging/lustre/lustre/llite/llite_mmap.c +3 -1
@@ -61,9 +61,11 @@ struct vm_area_struct *our_vma(struct mm_struct *mm, unsigned long addr, size_t count) {struct vm_area_struct *vma, *ret =NULL ; +struct range_rwlock range;/* mmap_sem must have been held by caller. */ -LASSERT (!down_write_trylock (&mm->mmap_sem)); +range_rwlock_init_full (&range); +LASSERT (!range_write_trylock (&mm->mmap_rw_tree, &range));for (vma =find_vma (mm, addr); vma && vma->vm_start < (addr + count); vma = vma->vm_next) {
drivers/staging/lustre/lustre/llite/vvp_io.c +4 -2
@@ -376,6 +376,7 @@ static int vvp_mmap_locks(const struct lu_env *env,int result =0 ;struct iov_iter i;struct iovec iov; +struct range_rwlock range;LASSERT (io->ci_type ==CIT_READ || io->ci_type ==CIT_WRITE );
@@ -395,7 +396,8 @@ static int vvp_mmap_locks(const struct lu_env *env, count += addr & (~PAGE_MASK ); addr &=PAGE_MASK ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);while ((vma =our_vma (mm, addr, count)) !=NULL ) {struct inode *inode =file_inode (vma->vm_file);int flags =CEF_MUST ;
@@ -436,7 +438,7 @@ static int vvp_mmap_locks(const struct lu_env *env, count -= vma->vm_end - addr; addr = vma->vm_end; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);if (result <0 )break ; }
drivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.c +5 -2
@@ -467,14 +467,17 @@ create_pagelist(char __user *buf, size_t count, unsigned short type, }/* do not try and release vmalloc pages */ }else { -down_read (&task->mm->mmap_sem); +struct range_rwlock range; + +range_rwlock_init_full (&range); +range_read_lock (&task->mm->mmap_rw_tree, &range); actual_pages =get_user_pages ( (unsigned long )buf & ~(PAGE_SIZE -1 ), num_pages, (type ==PAGELIST_READ ) ?FOLL_WRITE :0 , pages,NULL /*vmas */ ); -up_read (&task->mm->mmap_sem); +range_read_unlock (&task->mm->mmap_rw_tree, &range);if (actual_pages != num_pages) { vchiq_log_info(vchiq_arm_log_level,
drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c +4 -2
@@ -1560,6 +1560,7 @@ dump_phys_mem(void *virt_addr, u32 num_bytes)struct page *page;struct page **pages;u8 *kmapped_virt_ptr; +struct range_rwlock range;/* Align virtAddr and endVirtAddr to 16 byte boundaries. */
@@ -1580,14 +1581,15 @@ dump_phys_mem(void *virt_addr, u32 num_bytes)return ; } -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); rc =get_user_pages ( (unsigned long )virt_addr,/* start */ num_pages,/* len */ 0 ,/* gup_flags */ pages,/* pages (array of page pointers) */ NULL );/* vmas */ -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); prev_idx =-1 ; page =NULL ;
drivers/vfio/vfio_iommu_spapr_tce.c +9 -4
@@ -37,6 +37,7 @@ static void tce_iommu_detach_group(void *iommu_data,static long try_increment_locked_vm (struct mm_struct *mm,long npages) {long ret =0 , locked, lock_limit; +struct range_rwlock range;if (WARN_ON_ONCE (!mm))return -EPERM ;
@@ -44,7 +45,8 @@ static long try_increment_locked_vm(struct mm_struct *mm, long npages)if (!npages)return 0 ; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); locked = mm->locked_vm + npages; lock_limit =rlimit (RLIMIT_MEMLOCK ) >>PAGE_SHIFT ;if (locked > lock_limit && !capable (CAP_IPC_LOCK ))
@@ -58,17 +60,20 @@ static long try_increment_locked_vm(struct mm_struct *mm, long npages)rlimit (RLIMIT_MEMLOCK ), ret ? " - exceeded" : ""); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }static void decrement_locked_vm (struct mm_struct *mm,long npages) { +struct range_rwlock range; +if (!mm || !npages)return ; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range);if (WARN_ON_ONCE (npages > mm->locked_vm)) npages = mm->locked_vm; mm->locked_vm -= npages;
@@ -76,7 +81,7 @@ static void decrement_locked_vm(struct mm_struct *mm, long npages) npages <<PAGE_SHIFT , mm->locked_vm <<PAGE_SHIFT , rlimit(RLIMIT_MEMLOCK)); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); }/*
drivers/vfio/vfio_iommu_type1.c +15 -9
@@ -257,11 +257,13 @@ static void vfio_lock_acct_bg(struct work_struct *work) { struct vwork *vwork = container_of(work, struct vwork, work);struct mm_struct *mm; +struct range_rwlock range; mm = vwork->mm; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); mm->locked_vm += vwork->npage; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm);kfree (vwork); }
@@ -270,6 +272,7 @@ static void vfio_lock_acct(struct task_struct *task, long npage) {struct vwork *vwork;struct mm_struct *mm; +struct range_rwlock range;bool is_current;if (!npage)
@@ -281,9 +284,10 @@ static void vfio_lock_acct(struct task_struct *task, long npage)if (!mm)return ;/* process exited */ -if (down_write_trylock (&mm->mmap_sem)) { +range_rwlock_init_full (&range); +if (range_write_trylock (&mm->mmap_rw_tree, &range)) { mm->locked_vm += npage; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (!is_current)mmput (mm);return ;
@@ -361,8 +365,10 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr, {struct page *page[1 ];struct vm_area_struct *vma; +struct range_rwlock range;int ret; +range_rwlock_init_full (&range);if (mm == current->mm) { ret =get_user_pages_fast (vaddr,1 , !!(prot &IOMMU_WRITE ), page);
@@ -372,10 +378,10 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,if (prot &IOMMU_WRITE ) flags |=FOLL_WRITE ; -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); ret =get_user_pages_remote (NULL , mm, vaddr,1 , flags, page, -NULL ,NULL ); -up_read (&mm->mmap_sem); +NULL ,NULL ,NULL ); +range_read_unlock (&mm->mmap_rw_tree, &range); }if (ret ==1 ) {
@@ -383,7 +389,7 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,return 0 ; } -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_vma_intersection (mm, vaddr, vaddr +1 );
@@ -393,7 +399,7 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr, ret =0 ; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return ret; }
drivers/virt/fsl_hypervisor.c +4 -2
@@ -146,6 +146,7 @@ static long ioctl_stop(struct fsl_hv_ioctl_stop __user *p)*/ static long ioctl_memcpy (struct fsl_hv_ioctl_memcpy __user *p) { +struct range_rwlock range;struct fsl_hv_ioctl_memcpy param;struct page **pages =NULL ;
@@ -243,11 +244,12 @@ static long ioctl_memcpy(struct fsl_hv_ioctl_memcpy __user *p) sg_list =PTR_ALIGN (sg_list_unaligned,sizeof (struct fh_sg_list ));/* Get the physical addresses of the source buffer */ -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); num_pinned =get_user_pages (param.local_vaddr - lb_offset, num_pages, (param.source ==-1 ) ?0 :FOLL_WRITE , pages,NULL ); -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (num_pinned != num_pages) {/* get_user_pages() failed */
drivers/xen/gntdev.c +4 -2
@@ -657,13 +657,15 @@ static long gntdev_ioctl_get_offset_for_vaddr(struct gntdev_priv *priv,struct ioctl_gntdev_get_offset_for_vaddr op;struct vm_area_struct *vma;struct grant_map *map; +struct range_rwlock range;int rv = -EINVAL ;if (copy_from_user (&op, u,sizeof (op)) !=0 )return -EFAULT ;pr_debug ("priv %p, offset for vaddr %lx\n" , priv, (unsigned long )op.vaddr); -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); vma =find_vma (current->mm, op.vaddr);if (!vma || vma->vm_ops != &gntdev_vmops) goto out_unlock;
@@ -677,7 +679,7 @@ static long gntdev_ioctl_get_offset_for_vaddr(struct gntdev_priv *priv, rv =0 ; out_unlock: -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (rv ==0 &©_to_user (u, &op,sizeof (op)) !=0 )return -EFAULT ;
drivers/xen/privcmd.c +9 -5
@@ -260,6 +260,7 @@ static long privcmd_ioctl_mmap(struct file *file, void __user *udata)int rc;LIST_HEAD (pagelist);struct mmap_gfn_state state; +struct range_rwlock range;/* We only support privcmd_ioctl_mmap_batch for auto translated. */ if (xen_feature (XENFEAT_auto_translated_physmap))
@@ -279,7 +280,8 @@ static long privcmd_ioctl_mmap(struct file *file, void __user *udata)if (rc ||list_empty (&pagelist)) goto out; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); { struct page *page = list_first_entry(&pagelist,
@@ -304,7 +306,7 @@ static long privcmd_ioctl_mmap(struct file *file, void __user *udata) out_up: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); out:free_page_list (&pagelist);
@@ -454,6 +456,7 @@ static long privcmd_ioctl_mmap_batch(unsigned long nr_pages;LIST_HEAD (pagelist);struct mmap_batch_state state; +struct range_rwlock range;switch (version) {case 1 :
@@ -500,7 +503,8 @@ static long privcmd_ioctl_mmap_batch( } } -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, m.addr); if (!vma ||
@@ -556,7 +560,7 @@ static long privcmd_ioctl_mmap_batch(BUG_ON (traverse_pages_block (m.num,sizeof (xen_pfn_t), &pagelist, mmap_batch_fn, &state)); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (state.global_error) {/* Write back errors in second pass. */
@@ -577,7 +581,7 @@ static long privcmd_ioctl_mmap_batch(return ret; out_unlock: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); goto out; }
fs/aio.c +5 -2
@@ -450,6 +450,9 @@ static int aio_setup_ring(struct kioctx *ctx)int nr_pages;int i;struct file *file; +struct range_rwlock range; + +range_rwlock_init_full (&range);/* Compensate for the ring buffer's head/tail overlap entry */ nr_events +=2 ;/* 1 is required, 2 for good luck */
@@ -504,7 +507,7 @@ static int aio_setup_ring(struct kioctx *ctx) ctx->mmap_size = nr_pages *PAGE_SIZE ;pr_debug ("attempting mmap of %lu bytes\n" , ctx->mmap_size); -if (down_write_killable (&mm->mmap_sem)) { +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range)) { ctx->mmap_size =0 ;aio_free_ring (ctx);return -EINTR ;
@@ -513,7 +516,7 @@ static int aio_setup_ring(struct kioctx *ctx) ctx->mmap_base =do_mmap_pgoff (ctx->aio_ring_file,0 , ctx->mmap_size,PROT_READ |PROT_WRITE ,MAP_SHARED ,0 , &unused,NULL ); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (IS_ERR ((void *)ctx->mmap_base)) { ctx->mmap_size =0 ;aio_free_ring (ctx);
fs/coredump.c +4 -2
@@ -411,17 +411,19 @@ static int coredump_wait(int exit_code, struct core_state *core_state)struct task_struct *tsk = current;struct mm_struct *mm = tsk->mm;int core_waiters = -EBUSY ; +struct range_rwlock range; +range_rwlock_init_full (&range);init_completion (&core_state->startup); core_state->dumper.task = tsk; core_state->dumper.next =NULL ; -if (down_write_killable (&mm->mmap_sem)) +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return -EINTR ;if (!mm->core_state) core_waiters =zap_threads (tsk, mm, core_state, exit_code); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (core_waiters >0 ) {struct core_thread *ptr;
fs/exec.c +16 -8
@@ -268,12 +268,14 @@ static int __bprm_mm_init(struct linux_binprm *bprm)int err;struct vm_area_struct *vma =NULL ;struct mm_struct *mm = bprm->mm; +struct range_rwlock range; bprm->vma = vma =kmem_cache_zalloc (vm_area_cachep,GFP_KERNEL );if (!vma)return -ENOMEM ; -if (down_write_killable (&mm->mmap_sem)) { +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range)) { err = -EINTR ; goto err_free; }
@@ -298,11 +300,11 @@ static int __bprm_mm_init(struct linux_binprm *bprm) mm->stack_vm = mm->total_vm =1 ;arch_bprm_mm_init (mm, vma); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); bprm->p = vma->vm_end -sizeof (void *);return 0 ; err: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); err_free: bprm->vma =NULL ;kmem_cache_free (vm_area_cachep, vma);
@@ -673,6 +675,7 @@ int setup_arg_pages(struct linux_binprm *bprm,unsigned long stack_size;unsigned long stack_expand;unsigned long rlim_stack; +struct range_rwlock range;#ifdef CONFIG_STACK_GROWSUP /* Limit stack size */
@@ -710,7 +713,8 @@ int setup_arg_pages(struct linux_binprm *bprm, bprm->loader -= stack_shift; bprm->exec -= stack_shift; -if (down_write_killable (&mm->mmap_sem)) +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return -EINTR ; vm_flags =VM_STACK_FLAGS ;
@@ -767,7 +771,7 @@ int setup_arg_pages(struct linux_binprm *bprm, ret = -EFAULT ; out_unlock: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }EXPORT_SYMBOL (setup_arg_pages);
@@ -1001,6 +1005,9 @@ static int exec_mmap(struct mm_struct *mm) {struct task_struct *tsk;struct mm_struct *old_mm, *active_mm; +struct range_rwlock range; + +range_rwlock_init_full (&range);/* Notify parent that we're no longer interested in the old VM */ tsk = current;
@@ -1015,9 +1022,10 @@ static int exec_mmap(struct mm_struct *mm)* through with the exec. We must hold mmap_sem around * checking core_state and changing tsk->mm. */ -down_read (&old_mm->mmap_sem); + +range_read_lock (&old_mm->mmap_rw_tree, &range);if (unlikely (old_mm->core_state)) { -up_read (&old_mm->mmap_sem); +range_read_unlock (&old_mm->mmap_rw_tree, &range);return -EINTR ; } }
@@ -1030,7 +1038,7 @@ static int exec_mmap(struct mm_struct *mm)vmacache_flush (tsk);task_unlock (tsk);if (old_mm) { -up_read (&old_mm->mmap_sem); +range_read_unlock (&old_mm->mmap_rw_tree, &range);BUG_ON (active_mm != old_mm);setmax_mm_hiwater_rss (&tsk->signal->maxrss, old_mm);mm_update_next_owner (old_mm);
fs/proc/base.c +25 -13
@@ -216,7 +216,9 @@ static ssize_t proc_pid_cmdline_read(struct file *file, char __user *buf,unsigned long p;char c;ssize_t rv; +struct range_rwlock range; +range_rwlock_init_full (&range);BUG_ON (*pos <0 ); tsk =get_proc_task (file_inode (file));
@@ -238,12 +240,12 @@ static ssize_t proc_pid_cmdline_read(struct file *file, char __user *buf, goto out_mmput; } -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); arg_start = mm->arg_start; arg_end = mm->arg_end; env_start = mm->env_start; env_end = mm->env_end; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);BUG_ON (arg_start > arg_end);BUG_ON (env_start > env_end);
@@ -916,7 +918,9 @@ static ssize_t environ_read(struct file *file, char __user *buf,int ret =0 ;struct mm_struct *mm = file->private_data;unsigned long env_start, env_end; +struct range_rwlock range; +range_rwlock_init_full (&range);/* Ensure the process spawned far enough to have an environment. */ if (!mm || !mm->env_end)return 0 ;
@@ -929,10 +933,10 @@ static ssize_t environ_read(struct file *file, char __user *buf,if (!mmget_not_zero (mm)) goto free; -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); env_start = mm->env_start; env_end = mm->env_end; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);while (count >0 ) {size_t this_len, max_len;
@@ -1880,6 +1884,7 @@ static int map_files_d_revalidate(struct dentry *dentry, unsigned int flags)struct task_struct *task;struct inode *inode;int status =0 ; +struct range_rwlock range;if (flags &LOOKUP_RCU )return -ECHILD ;
@@ -1894,9 +1899,10 @@ static int map_files_d_revalidate(struct dentry *dentry, unsigned int flags) goto out;if (!dname_to_vma_addr (dentry, &vm_start, &vm_end)) { -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); exact_vma_exists = !!find_exact_vma (mm, vm_start, vm_end); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); }mmput (mm);
@@ -1927,6 +1933,7 @@ static int map_files_get_link(struct dentry *dentry, struct path *path)struct task_struct *task;struct mm_struct *mm;int rc; +struct range_rwlock range; rc = -ENOENT ; task =get_proc_task (d_inode (dentry));
@@ -1943,14 +1950,15 @@ static int map_files_get_link(struct dentry *dentry, struct path *path) goto out_mmput; rc = -ENOENT ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_exact_vma (mm, vm_start, vm_end);if (vma && vma->vm_file) { *path = vma->vm_file->f_path;path_get (path); rc =0 ; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); out_mmput:mmput (mm);
@@ -2023,6 +2031,7 @@ static struct dentry *proc_map_files_lookup(struct inode *dir,struct task_struct *task;int result;struct mm_struct *mm; +struct range_rwlock range; result = -ENOENT ; task =get_proc_task (dir);
@@ -2041,7 +2050,8 @@ static struct dentry *proc_map_files_lookup(struct inode *dir,if (!mm) goto out_put_task; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_exact_vma (mm, vm_start, vm_end);if (!vma) goto out_no_vma;
@@ -2051,7 +2061,7 @@ static struct dentry *proc_map_files_lookup(struct inode *dir, (void *)(unsigned long)vma->vm_file->f_mode); out_no_vma: -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);mmput (mm); out_put_task:put_task_struct (task);
@@ -2076,7 +2086,9 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx)struct map_files_info info;struct map_files_info *p;int ret; +struct range_rwlock range; +range_rwlock_init_full (&range); ret = -ENOENT ; task =get_proc_task (file_inode (file));if (!task)
@@ -2093,7 +2105,7 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx) mm =get_task_mm (task);if (!mm) goto out_put_task; -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); nr_files =0 ;
@@ -2120,7 +2132,7 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx) ret = -ENOMEM ;if (fa)flex_array_free (fa); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);mmput (mm); goto out_put_task; }
@@ -2139,7 +2151,7 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx)BUG (); } } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);for (i =0 ; i < nr_files; i++) { p =flex_array_get (fa, i);
fs/proc/internal.h +1 -0
@@ -279,6 +279,7 @@ struct proc_maps_private {#ifdef CONFIG_NUMA struct mempolicy *task_mempolicy;#endif +struct range_rwlock range; };struct mm_struct *proc_mem_open (struct inode *inode,unsigned int mode);
fs/proc/task_mmu.c +19 -11
@@ -133,7 +133,7 @@ static void vma_stop(struct proc_maps_private *priv)struct mm_struct *mm = priv->mm;release_task_mempolicy (priv); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &priv->range);mmput (mm); }
@@ -171,7 +171,8 @@ static void *m_start(struct seq_file *m, loff_t *ppos)if (!mm || !mmget_not_zero (mm))return NULL ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&priv->range); +range_read_lock (&mm->mmap_rw_tree, &priv->range);hold_task_mempolicy (priv); priv->tail_vma =get_gate_vma (mm);
@@ -1009,7 +1010,9 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf,enum clear_refs_types type;int itype;int rv; +struct range_rwlock range; +range_rwlock_init_full (&range);memset (buffer,0 ,sizeof (buffer));if (count >sizeof (buffer) -1 ) count =sizeof (buffer) -1 ;
@@ -1038,7 +1041,8 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, };if (type ==CLEAR_REFS_MM_HIWATER_RSS ) { -if (down_write_killable (&mm->mmap_sem)) { +if (range_write_lock_interruptible (&mm->mmap_rw_tree, + &range)) { count = -EINTR ; goto out_mm; }
@@ -1048,17 +1052,18 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf,* resident set size to this mm's current rss value. */ reset_mm_hiwater_rss (mm); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); goto out_mm; } -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range);if (type ==CLEAR_REFS_SOFT_DIRTY ) {for (vma = mm->mmap; vma; vma = vma->vm_next) {if (!(vma->vm_flags &VM_SOFTDIRTY ))continue ; -up_read (&mm->mmap_sem); -if (down_write_killable (&mm->mmap_sem)) { +range_read_unlock (&mm->mmap_rw_tree, &range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, + &range)) { count = -EINTR ; goto out_mm; }
@@ -1066,7 +1071,8 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, vma->vm_flags &= ~VM_SOFTDIRTY ;vma_set_page_prot (vma); } -downgrade_write (&mm->mmap_sem); +range_downgrade_write (&mm->mmap_rw_tree, + &range);break ; }mmu_notifier_invalidate_range_start (mm,0 ,-1 );
@@ -1075,7 +1081,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf,if (type ==CLEAR_REFS_SOFT_DIRTY )mmu_notifier_invalidate_range_end (mm,0 ,-1 );flush_tlb_mm (mm); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); out_mm:mmput (mm); }
@@ -1359,6 +1365,7 @@ static ssize_t pagemap_read(struct file *file, char __user *buf,unsigned long start_vaddr;unsigned long end_vaddr;int ret =0 , copied =0 ; +struct range_rwlock range;if (!mm || !mmget_not_zero (mm)) goto out;
@@ -1414,9 +1421,10 @@ static ssize_t pagemap_read(struct file *file, char __user *buf,/* overflow ? */ if (end < start_vaddr || end > end_vaddr) end = end_vaddr; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); ret =walk_page_range (start_vaddr, end, &pagemap_walk); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); start_vaddr = end; len =min (count,PM_ENTRY_BYTES * pm.pos);
fs/proc/task_nommu.c +17 -10
@@ -23,8 +23,10 @@ void task_mem(struct seq_file *m, struct mm_struct *mm)struct vm_region *region;struct rb_node *p;unsigned long bytes =0 , sbytes =0 , slack =0 , size; - -down_read (&mm->mmap_sem); +struct range_rwlock range; + +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (p =rb_first (&mm->mm_rb); p; p =rb_next (p)) { vma = rb_entry(p, struct vm_area_struct, vm_rb);
@@ -76,7 +78,7 @@ void task_mem(struct seq_file *m, struct mm_struct *mm)"Shared:\t%8lu bytes\n" , bytes, slack, sbytes); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); }unsigned long task_vsize (struct mm_struct *mm)
@@ -84,13 +86,15 @@ unsigned long task_vsize(struct mm_struct *mm)struct vm_area_struct *vma;struct rb_node *p;unsigned long vsize =0 ; +struct range_rwlock range; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (p =rb_first (&mm->mm_rb); p; p =rb_next (p)) { vma = rb_entry(p, struct vm_area_struct, vm_rb); vsize += vma->vm_end - vma->vm_start; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return vsize; }
@@ -102,8 +106,10 @@ unsigned long task_statm(struct mm_struct *mm,struct vm_region *region;struct rb_node *p;unsigned long size =kobjsize (mm); +struct range_rwlock range; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (p =rb_first (&mm->mm_rb); p; p =rb_next (p)) { vma = rb_entry(p, struct vm_area_struct, vm_rb); size +=kobjsize (vma);
@@ -118,7 +124,7 @@ unsigned long task_statm(struct mm_struct *mm, >> PAGE_SHIFT; *data = (PAGE_ALIGN (mm->start_stack) - (mm->start_data &PAGE_MASK )) >>PAGE_SHIFT ; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); size >>=PAGE_SHIFT ; size += *text + *data; *resident = size;
@@ -224,13 +230,14 @@ static void *m_start(struct seq_file *m, loff_t *pos)if (!mm || !mmget_not_zero (mm))return NULL ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&priv->range); +range_read_lock (&mm->mmap_rw_tree, &priv->range);/* start from the Nth VMA */ for (p =rb_first (&mm->mm_rb); p; p =rb_next (p))if (n-- ==0 )return p; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &priv->range);mmput (mm);return NULL ; }
@@ -240,7 +247,7 @@ static void m_stop(struct seq_file *m, void *_vml)struct proc_maps_private *priv = m->private;if (!IS_ERR_OR_NULL (_vml)) { -up_read (&priv->mm->mmap_sem); +range_read_unlock (&priv->mm->mmap_rw_tree, &priv->range);mmput (priv->mm); }if (priv->task) {
fs/userfaultfd.c +15 -9
@@ -431,7 +431,7 @@ int handle_userfault(struct vm_fault *vmf, unsigned long reason)else must_wait =userfaultfd_huge_must_wait (ctx, vmf->address, vmf->flags, reason); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, vmf->lockrange); if (likely(must_wait && !ACCESS_ONCE(ctx->released) && (return_to_userland ? !signal_pending(current) :
@@ -485,7 +485,7 @@ int handle_userfault(struct vm_fault *vmf, unsigned long reason)* and there's no need to retake the mmap_sem * in such case. */ -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, vmf->lockrange); ret =VM_FAULT_NOPAGE ; } }
@@ -704,7 +704,7 @@ bool userfaultfd_remove(struct vm_area_struct *vma,return true;userfaultfd_ctx_get (ctx); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, range);msg_init (&ewq.msg);
@@ -783,7 +783,9 @@ static int userfaultfd_release(struct inode *inode, struct file *file)/* len == 0 means wake all */ struct userfaultfd_wake_range range = { .len =0 , };unsigned long new_flags; +struct range_rwlock lockrange; +range_rwlock_init_full (&lockrange);ACCESS_ONCE (ctx->released) = true;if (!mmget_not_zero (mm))
@@ -797,7 +799,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file)* it's critical that released is set to true (above), before * taking the mmap_sem for writing. */ -down_write (&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &lockrange); prev =NULL ;for (vma = mm->mmap; vma; vma = vma->vm_next) {cond_resched ();
@@ -820,7 +822,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ; } -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &lockrange);mmput (mm); wakeup:/*
@@ -1180,6 +1182,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,bool found;bool non_anon_pages;unsigned long start, end, vma_end; +struct range_rwlock range; user_uffdio_register = (struct uffdio_register __user *) arg;
@@ -1219,7 +1222,8 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,if (!mmget_not_zero (mm)) goto out; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); vma =find_vma_prev (mm, start, &prev);if (!vma) goto out_unlock;
@@ -1347,7 +1351,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, vma = vma->vm_next; }while (vma && vma->vm_start < end); out_unlock: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm);if (!ret) {/*
@@ -1375,6 +1379,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,bool found;unsigned long start, end, vma_end;const void __user *buf = (void __user *)arg; +struct range_rwlock range; ret = -EFAULT ;if (copy_from_user (&uffdio_unregister, buf,sizeof (uffdio_unregister)))
@@ -1392,7 +1397,8 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,if (!mmget_not_zero (mm)) goto out; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); vma =find_vma_prev (mm, start, &prev);if (!vma) goto out_unlock;
@@ -1505,7 +1511,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, vma = vma->vm_next; }while (vma && vma->vm_start < end); out_unlock: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm); out:return ret;
include/linux/mm_types.h +2 -1
@@ -8,6 +8,7 @@#include <linux/spinlock.h> #include <linux/rbtree.h> #include <linux/rwsem.h> +#include <linux/range_rwlock.h> #include <linux/completion.h> #include <linux/cpumask.h> #include <linux/uprobes.h>
@@ -398,7 +399,7 @@ struct mm_struct {int map_count;/* number of VMAs */ spinlock_t page_table_lock;/* Protects page tables and some counters */ -struct rw_semaphore mmap_sem; +struct range_rwlock_tree mmap_rw_tree;/* formerly mmap_sem */ struct list_head mmlist;/* List of maybe swapped mm's. These are globally strung * together off init_mm.mmlist, and are protected
ipc/shm.c +9 -4
@@ -1107,6 +1107,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg,struct path path;fmode_t f_mode;unsigned long populate =0 ; +struct range_rwlock range; err = -EINVAL ;if (shmid <0 )
@@ -1213,7 +1214,9 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg,if (err) goto out_fput; -if (down_write_killable (¤t->mm->mmap_sem)) { +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (¤t->mm->mmap_rw_tree, + &range)) { err = -EINTR ; goto out_fput; }
@@ -1233,7 +1236,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg,if (IS_ERR_VALUE (addr)) err = (long )addr; invalid: -up_write (¤t->mm->mmap_sem); +range_write_unlock (¤t->mm->mmap_rw_tree, &range);if (populate)mm_populate (addr, populate);
@@ -1284,11 +1287,13 @@ SYSCALL_DEFINE1(shmdt, char __user *, shmaddr)struct file *file;struct vm_area_struct *next;#endif +struct range_rwlock range;if (addr & ~PAGE_MASK )return retval; -if (down_write_killable (&mm->mmap_sem)) +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return -EINTR ;/*
@@ -1376,7 +1381,7 @@ SYSCALL_DEFINE1(shmdt, char __user *, shmaddr)#endif -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return retval; }
kernel/acct.c +4 -2
@@ -534,17 +534,19 @@ void acct_collect(long exitcode, int group_dead)struct pacct_struct *pacct = ¤t->signal->pacct;u64 utime, stime;unsigned long vsize =0 ; +struct range_rwlock range; +range_rwlock_init_full (&range);if (group_dead && current->mm) {struct vm_area_struct *vma; -down_read (¤t->mm->mmap_sem); +range_read_lock (¤t->mm->mmap_rw_tree, &range); vma = current->mm->mmap;while (vma) { vsize += vma->vm_end - vma->vm_start; vma = vma->vm_next; } -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); }spin_lock_irq (¤t->sighand->siglock);
kernel/events/core.c +4 -2
@@ -8091,7 +8091,9 @@ static void perf_event_addr_filters_apply(struct perf_event *event)struct mm_struct *mm =NULL ;unsigned int count =0 ;unsigned long flags; +struct range_rwlock range; +range_rwlock_init_full (&range);/* * We may observe TASK_TOMBSTONE, which means that the event tear-down * will stop on the parent's child_mutex that our caller is also holding
@@ -8106,7 +8108,7 @@ static void perf_event_addr_filters_apply(struct perf_event *event)if (!mm) goto restart; -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range);raw_spin_lock_irqsave (&ifh->lock, flags);list_for_each_entry (filter, &ifh->list, entry) {
@@ -8126,7 +8128,7 @@ static void perf_event_addr_filters_apply(struct perf_event *event) event->addr_filters_gen++;raw_spin_unlock_irqrestore (&ifh->lock, flags); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);mmput (mm);
kernel/events/uprobes.c +16 -8
@@ -806,11 +806,13 @@ register_for_each_vma(struct uprobe *uprobe, struct uprobe_consumer *new)while (info) {struct mm_struct *mm = info->mm;struct vm_area_struct *vma; +struct range_rwlock range;if (err && is_register) goto free; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, info->vaddr);if (!vma || !valid_vma (vma, is_register) ||file_inode (vma->vm_file) != uprobe->inode)
@@ -832,7 +834,7 @@ register_for_each_vma(struct uprobe *uprobe, struct uprobe_consumer *new) } unlock: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); free:mmput (mm); info =free_map_info (info);
@@ -971,9 +973,11 @@ EXPORT_SYMBOL_GPL(uprobe_unregister);static int unapply_uprobe (struct uprobe *uprobe,struct mm_struct *mm) {struct vm_area_struct *vma; +struct range_rwlock range;int err =0 ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (vma = mm->mmap; vma; vma = vma->vm_next) {unsigned long vaddr;loff_t offset;
@@ -990,7 +994,7 @@ static int unapply_uprobe(struct uprobe *uprobe, struct mm_struct *mm) vaddr =offset_to_vaddr (vma, uprobe->offset); err |=remove_breakpoint (uprobe, mm, vaddr); } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return err; }
@@ -1138,9 +1142,11 @@ void uprobe_munmap(struct vm_area_struct *vma, unsigned long start, unsigned lonstatic int xol_add_vma (struct mm_struct *mm,struct xol_area *area) {struct vm_area_struct *vma; +struct range_rwlock range;int ret; -if (down_write_killable (&mm->mmap_sem)) +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return -EINTR ;if (mm->uprobes_state.xol_area) {
@@ -1170,7 +1176,7 @@ static int xol_add_vma(struct mm_struct *mm, struct xol_area *area)smp_wmb ();/* pairs with get_xol_area() */ mm->uprobes_state.xol_area = area; fail: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }
@@ -1736,8 +1742,10 @@ static struct uprobe *find_active_uprobe(unsigned long bp_vaddr, int *is_swbp)struct mm_struct *mm = current->mm;struct uprobe *uprobe =NULL ;struct vm_area_struct *vma; +struct range_rwlock range; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, bp_vaddr);if (vma && vma->vm_start <= bp_vaddr) {if (valid_vma (vma, false)) {
@@ -1755,7 +1763,7 @@ static struct uprobe *find_active_uprobe(unsigned long bp_vaddr, int *is_swbp)if (!uprobe &&test_and_clear_bit (MMF_RECALC_UPROBES , &mm->flags))mmf_recalc_uprobes (mm); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return uprobe; }
kernel/exit.c +6 -4
@@ -508,6 +508,7 @@ static void exit_mm(void) {struct mm_struct *mm = current->mm;struct core_state *core_state; +struct range_rwlock range;mm_release (current, mm);if (!mm)
@@ -520,12 +521,13 @@ static void exit_mm(void)* will increment ->nr_threads for each thread in the * group with ->mm != NULL. */ -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); core_state = mm->core_state;if (core_state) {struct core_thread self; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); self.task = current; self.next =xchg (&core_state->dumper.next, &self);
@@ -543,14 +545,14 @@ static void exit_mm(void)freezable_schedule (); }__set_current_state (TASK_RUNNING ); -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); }mmgrab (mm);BUG_ON (mm != current->active_mm);/* more a memory barrier than a real lock */ task_lock (current); current->mm =NULL ; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);enter_lazy_tlb (mm, current);task_unlock (current);mm_update_next_owner (mm);
kernel/fork.c +14 -7
@@ -573,9 +573,13 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,int retval;unsigned long charge;LIST_HEAD (uf); +struct range_rwlock range, oldrange; + +range_rwlock_init_full (&range); +range_rwlock_init_full (&oldrange);uprobe_start_dup_mmap (); -if (down_write_killable (&oldmm->mmap_sem)) { +if (range_write_lock_interruptible (&oldmm->mmap_rw_tree, &oldrange)) { retval = -EINTR ; goto fail_uprobe_end; }
@@ -584,7 +588,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,/* * Not linked in yet - no deadlock potential: */ -down_write_nested (&mm->mmap_sem,SINGLE_DEPTH_NESTING ); +range_write_lock (&mm->mmap_rw_tree, &range);/* No ordering required: file already has been exposed. */ RCU_INIT_POINTER (mm->exe_file,get_mm_exe_file (oldmm));
@@ -688,9 +692,9 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,arch_dup_mmap (oldmm, mm); retval =0 ; out: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);flush_tlb_mm (oldmm); -up_write (&oldmm->mmap_sem); +range_write_unlock (&oldmm->mmap_rw_tree, &oldrange);dup_userfaultfd_complete (&uf); fail_uprobe_end:uprobe_end_dup_mmap ();
@@ -720,9 +724,12 @@ static inline void mm_free_pgd(struct mm_struct *mm)#else static int dup_mmap (struct mm_struct *mm,struct mm_struct *oldmm) { -down_write (&oldmm->mmap_sem); +struct range_rwlock oldrange; + +range_rwlock_init_full (&oldrange); +range_write_lock (&oldmm->mmap_rw_tree, &oldrange);RCU_INIT_POINTER (mm->exe_file,get_mm_exe_file (oldmm)); -up_write (&oldmm->mmap_sem); +range_write_unlock (&oldmm->mmap_rw_tree, &oldrange);return 0 ; }#define mm_alloc_pgd (mm) (0)
@@ -771,7 +778,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm->vmacache_seqnum =0 ;atomic_set (&mm->mm_users,1 );atomic_set (&mm->mm_count,1 ); -init_rwsem (&mm->mmap_sem); +range_rwlock_tree_init (&mm->mmap_rw_tree);INIT_LIST_HEAD (&mm->mmlist); mm->core_state =NULL ;atomic_long_set (&mm->nr_ptes,0 );
kernel/futex.c +5 -3
@@ -723,12 +723,14 @@ static inline void put_futex_key(union futex_key *key)static int fault_in_user_writeable (u32 __user *uaddr) {struct mm_struct *mm = current->mm; +struct range_rwlock range;int ret; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range);/* XXX finer grain required here */ +range_read_lock (&mm->mmap_rw_tree, &range); ret =fixup_user_fault (current, mm, (unsigned long )uaddr, -FAULT_FLAG_WRITE ,NULL ,NULL ); -up_read (&mm->mmap_sem); +FAULT_FLAG_WRITE ,NULL , &range); +range_read_unlock (&mm->mmap_rw_tree, &range);return ret <0 ? ret :0 ; }
kernel/sched/fair.c +4 -3
@@ -2425,6 +2425,7 @@ void task_numa_work(struct callback_head *work)unsigned long start, end;unsigned long nr_pte_updates =0 ;long pages, virtpages; +struct range_rwlock range; SCHED_WARN_ON(p != container_of(work, struct task_struct, numa_work));
@@ -2474,8 +2475,8 @@ void task_numa_work(struct callback_head *work)if (!pages)return ; - -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, start);if (!vma) {reset_ptenuma_scan (p);
@@ -2542,7 +2543,7 @@ void task_numa_work(struct callback_head *work) mm->numa_scan_offset = start; elsereset_ptenuma_scan (p ); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);/* * Make sure tasks use at least 32x as much time to run other code
kernel/sys.c +22 -9
@@ -1663,6 +1663,9 @@ static int prctl_set_mm_exe_file(struct mm_struct *mm, unsigned int fd)struct file *old_exe, *exe_file;struct inode *inode;int err; +struct range_rwlock range; + +range_rwlock_init_full (&range); exe =fdget (fd);if (!exe.file)
@@ -1691,7 +1694,7 @@ static int prctl_set_mm_exe_file(struct mm_struct *mm, unsigned int fd)if (exe_file) {struct vm_area_struct *vma; -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range);for (vma = mm->mmap; vma; vma = vma->vm_next) {if (!vma->vm_file)continue ;
@@ -1700,7 +1703,7 @@ static int prctl_set_mm_exe_file(struct mm_struct *mm, unsigned int fd) goto exit_err; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);fput (exe_file); }
@@ -1714,7 +1717,7 @@ static int prctl_set_mm_exe_file(struct mm_struct *mm, unsigned int fd)fdput (exe);return err; exit_err: -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);fput (exe_file); goto exit; }
@@ -1821,6 +1824,9 @@ static int prctl_set_mm_map(int opt, const void __user *addr, unsigned long dataunsigned long user_auxv[AT_VECTOR_SIZE ];struct mm_struct *mm = current->mm;int error; +struct range_rwlock range; + +range_rwlock_init_full (&range);BUILD_BUG_ON (sizeof (user_auxv) !=sizeof (mm->saved_auxv));BUILD_BUG_ON (sizeof (struct prctl_mm_map ) >256 );
@@ -1857,7 +1863,7 @@ static int prctl_set_mm_map(int opt, const void __user *addr, unsigned long datareturn error; } -down_write (&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &range);/* * We don't validate if these members are pointing to
@@ -1894,7 +1900,7 @@ static int prctl_set_mm_map(int opt, const void __user *addr, unsigned long dataif (prctl_map.auxv_size)memcpy (mm->saved_auxv, user_auxv,sizeof (user_auxv)); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); return 0; }#endif /* CONFIG_CHECKPOINT_RESTORE */
@@ -1936,6 +1942,9 @@ static int prctl_set_mm(int opt, unsigned long addr,struct prctl_mm_map prctl_map;struct vm_area_struct *vma;int error; +struct range_rwlock range; + +range_rwlock_init_full (&range); if (arg5 || (arg4 && (opt != PR_SET_MM_AUXV && opt != PR_SET_MM_MAP &&
@@ -1961,7 +1970,7 @@ static int prctl_set_mm(int opt, unsigned long addr, error = -EINVAL ; -down_write (&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, addr); prctl_map.start_code = mm->start_code;
@@ -2054,7 +2063,7 @@ static int prctl_set_mm(int opt, unsigned long addr, error =0 ; out: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return error; }
@@ -2094,6 +2103,9 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,struct task_struct *me = current;unsigned char comm[sizeof (me->comm)];long error; +struct range_rwlock range; + +range_rwlock_init_full (&range); error =security_task_prctl (option, arg2, arg3, arg4, arg5);if (error != -ENOSYS )
@@ -2266,13 +2278,14 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,case PR_SET_THP_DISABLE :if (arg3 || arg4 || arg5)return -EINVAL ; -if (down_write_killable (&me->mm->mmap_sem)) +if (range_write_lock_interruptible (&me->mm->mmap_rw_tree, + &range))return -EINTR ;if (arg2) me->mm->def_flags |=VM_NOHUGEPAGE ;else me->mm->def_flags &= ~VM_NOHUGEPAGE ; -up_write (&me->mm->mmap_sem); +range_write_unlock (&me->mm->mmap_rw_tree, &range);break ;case PR_MPX_ENABLE_MANAGEMENT :if (arg2 || arg3 || arg4 || arg5)
kernel/trace/trace_output.c +4 -2
@@ -380,6 +380,7 @@ static int seq_print_user_ip(struct trace_seq *s, struct mm_struct *mm,struct file *file =NULL ;unsigned long vmstart =0 ;int ret =1 ; +struct range_rwlock range;if (s->full)return 0 ;
@@ -387,7 +388,8 @@ static int seq_print_user_ip(struct trace_seq *s, struct mm_struct *mm,if (mm) {const struct vm_area_struct *vma; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, ip);if (vma) { file = vma->vm_file;
@@ -399,7 +401,7 @@ static int seq_print_user_ip(struct trace_seq *s, struct mm_struct *mm,trace_seq_printf (s,"[+0x%lx]" , ip - vmstart); } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); }if (ret && ((sym_flags &TRACE_ITER_SYM_ADDR ) || !file))trace_seq_printf (s," <" IP_FMT ">" , ip);
mm/filemap.c +2 -2
@@ -1063,7 +1063,7 @@ int __lock_page_or_retry(struct page *page, struct mm_struct *mm,if (flags &FAULT_FLAG_RETRY_NOWAIT )return 0 ; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, range);if (flags &FAULT_FLAG_KILLABLE )wait_on_page_locked_killable (page);else
@@ -1075,7 +1075,7 @@ int __lock_page_or_retry(struct page *page, struct mm_struct *mm, ret =__lock_page_killable (page);if (ret) { -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, range);return 0 ; } }else
mm/frame_vector.c +6 -3
@@ -35,6 +35,7 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames, {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; +struct range_rwlock range;int ret =0 ;int err;int locked;
@@ -45,7 +46,8 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames,if (WARN_ON_ONCE (nr_frames > vec->nr_allocated)) nr_frames = vec->nr_allocated; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); locked =1 ; vma =find_vma_intersection (mm, start, start +1 );if (!vma) {
@@ -56,7 +58,8 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames, vec->got_ref = true; vec->is_pfns = false; ret =get_user_pages_locked (start, nr_frames, - gup_flags, (struct page **)(vec->ptrs), &locked,NULL ); + gup_flags, (struct page **)(vec->ptrs), + &locked, &range); goto out; }
@@ -85,7 +88,7 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames, }while (vma && vma->vm_flags & (VM_IO |VM_PFNMAP )); out:if (locked) -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);if (!ret) ret = -EFAULT ;if (ret >0 )
mm/gup.c +13 -9
@@ -735,7 +735,7 @@ int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm, }if (ret &VM_FAULT_RETRY ) { -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, range);if (!(fault_flags &FAULT_FLAG_TRIED )) { *unlocked = true; fault_flags &= ~FAULT_FLAG_ALLOW_RETRY ;
@@ -819,7 +819,7 @@ static __always_inline long __get_user_pages_locked(struct task_struct *tsk,*/ *locked =1 ; lock_dropped = true; -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, range); ret =__get_user_pages (tsk, mm, start,1 , flags |FOLL_TRIED , pages,NULL ,NULL , range);if (ret !=1 ) {
@@ -840,7 +840,7 @@ static __always_inline long __get_user_pages_locked(struct task_struct *tsk,* We must let the caller know we temporarily dropped the lock * and so the critical section protected by it was lost. */ -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, range); *locked =0 ; }return pages_done;
@@ -892,12 +892,14 @@ static __always_inline long __get_user_pages_unlocked(struct task_struct *tsk, {long ret;int locked =1 ; +struct range_rwlock range; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); ret =__get_user_pages_locked (tsk, mm, start, nr_pages, pages,NULL , - &locked, false,NULL , gup_flags); + &locked, false, &range, gup_flags);if (locked) -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return ret; }
@@ -1081,6 +1083,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)struct vm_area_struct *vma =NULL ;int locked =0 ;long ret =0 ; +struct range_rwlock range;VM_BUG_ON (start & ~PAGE_MASK );VM_BUG_ON (len !=PAGE_ALIGN (len));
@@ -1093,7 +1096,8 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)*/ if (!locked) { locked =1 ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, nstart); }else if (nstart >= vma->vm_end) vma = vma->vm_next;
@@ -1114,7 +1118,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)* if the vma was already munlocked. */ ret =populate_vma_page_range (vma, nstart, nend, &locked, -NULL ); + &range);if (ret <0 ) {if (ignore_errors) { ret =0 ;
@@ -1126,7 +1130,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors) ret =0 ; }if (locked) -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return ret;/* 0 or negative error code */ }
mm/init-mm.c +1 -1
@@ -19,7 +19,7 @@ struct mm_struct init_mm = { .pgd = swapper_pg_dir, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), - .mmap_sem = __RWSEM_INITIALIZER(init_mm.mmap_sem), + .mmap_rw_tree = __RANGE_RWLOCK_TREE_INITIALIZER(init_mm.mmap_rw_tree), .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), .user_ns = &init_user_ns,
mm/khugepaged.c +27 -17
32 lines not on this page · raw message
mm/ksm.c +30 -15
45 lines not on this page · raw message
mm/mmap.c +30 -13
43 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.7.4
Part of the diff is not shown here. View the raw message.
Matthew Wilcox
maintainerQuoted 3 lines · Laurent Dufour
Following the series pushed by Davidlohr Bueso based on the Jan Kara's work [1] which introduces range locks, this series implements the first step of the attempt to replace the
mmap_semby a range lock.
Have you previously documented attempts to replace the mmap_sem by an existing lock type before introducing a new (and frankly weird) lock? My initial question is "Why not use RCU for this?" -- the rxrpc code uses an rbtree protected by RCU.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Matthew Wilcox
Quoted 3 lines · Laurent Dufour
Following the series pushed by Davidlohr Bueso based on the Jan Kara's work [1] which introduces range locks, this series implements the first step of the attempt to replace the
mmap_semby a range lock.Have you previously documented attempts to replace the
mmap_semby an existing lock type before introducing a new (and frankly weird) lock?
No :/
Quoted 2 lines
My initial question is "Why not use RCU for this?" -- the rxrpc code uses an rbtree protected by RCU.
I'm also working on forward-porting work done by Peter Zijlstra :
https://marc.info/?l=linux-mm&m=141384492326748
I'll send a series on top 4.10 soon.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 4/4 Change mmap_sem to range lock
In reply to Laurent Dufour[resent this patch which seems to have not reached the mailing lists]
Change the mmap_sem to a range lock to allow finer grain locking on the memory layout of a task.
This patch rename mmap_sem into mmap_rw_tree to avoid confusion and replace any locking (read or write) by complete range locking. So there is no functional change except in the way the underlying locking is achieved.
Currently, this patch only supports x86 and PowerPc architectures, furthermore it should break the build of any others.
Signed-off-by Laurent Dufour <[email protected]>
--- arch/powerpc/kernel/vdso.c | 8 ++-- arch/powerpc/kvm/book3s_64_mmu_hv.c | 6 ++- arch/powerpc/kvm/book3s_64_mmu_radix.c | 6 ++- arch/powerpc/kvm/book3s_64_vio.c | 6 ++- arch/powerpc/kvm/book3s_hv.c | 8 ++-- arch/powerpc/kvm/e500_mmu_host.c | 7 +++- arch/powerpc/mm/copro_fault.c | 6 ++- arch/powerpc/mm/fault.c | 12 +++--- arch/powerpc/mm/mmu_context_iommu.c | 6 ++- arch/powerpc/mm/subpage-prot.c | 16 +++++--- arch/powerpc/oprofile/cell/spu_task_sync.c | 8 ++-- arch/powerpc/platforms/cell/spufs/file.c | 4 +- arch/x86/entry/vdso/vma.c | 14 ++++--- arch/x86/kernel/tboot.c | 2 +- arch/x86/kernel/vm86_32.c | 6 ++- arch/x86/mm/fault.c | 39 +++++++++++-------- arch/x86/mm/mpx.c | 18 ++++++--- drivers/android/binder.c | 8 ++-- drivers/firmware/efi/arm-runtime.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 9 +++-- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 8 ++-- drivers/gpu/drm/amd/amdkfd/kfd_events.c | 6 ++- drivers/gpu/drm/amd/amdkfd/kfd_process.c | 6 ++- drivers/gpu/drm/etnaviv/etnaviv_gem.c | 6 ++- drivers/gpu/drm/i915/i915_gem.c | 6 ++- drivers/gpu/drm/i915/i915_gem_userptr.c | 12 ++++-- drivers/gpu/drm/radeon/radeon_cs.c | 9 +++-- drivers/gpu/drm/radeon/radeon_gem.c | 8 ++-- drivers/gpu/drm/radeon/radeon_mn.c | 8 ++-- drivers/gpu/drm/ttm/ttm_bo_vm.c | 6 ++- drivers/gpu/drm/via/via_dmablit.c | 6 ++- drivers/infiniband/core/umem.c | 20 ++++++---- drivers/infiniband/core/umem_odp.c | 6 ++- drivers/infiniband/hw/hfi1/user_pages.c | 18 ++++++--- drivers/infiniband/hw/mlx4/main.c | 6 ++- drivers/infiniband/hw/mlx5/main.c | 6 ++- drivers/infiniband/hw/qib/qib_user_pages.c | 16 +++++--- drivers/infiniband/hw/usnic/usnic_uiom.c | 20 ++++++---- drivers/iommu/amd_iommu_v2.c | 8 ++-- drivers/iommu/intel-svm.c | 6 ++- drivers/media/v4l2-core/videobuf-core.c | 9 +++-- drivers/media/v4l2-core/videobuf-dma-contig.c | 6 ++- drivers/media/v4l2-core/videobuf-dma-sg.c | 6 ++- drivers/misc/cxl/fault.c | 6 ++- drivers/misc/mic/scif/scif_rma.c | 17 +++++--- drivers/oprofile/buffer_sync.c | 14 ++++--- drivers/staging/lustre/lustre/llite/llite_mmap.c | 4 +- drivers/staging/lustre/lustre/llite/vvp_io.c | 6 ++- .../interface/vchiq_arm/vchiq_2835_arm.c | 7 +++- .../vc04_services/interface/vchiq_arm/vchiq_arm.c | 6 ++- drivers/vfio/vfio_iommu_spapr_tce.c | 13 +++++-- drivers/vfio/vfio_iommu_type1.c | 24 +++++++----- drivers/virt/fsl_hypervisor.c | 6 ++- drivers/xen/gntdev.c | 6 ++- drivers/xen/privcmd.c | 14 ++++--- fs/aio.c | 7 +++- fs/coredump.c | 6 ++- fs/exec.c | 24 ++++++++---- fs/proc/base.c | 38 +++++++++++------- fs/proc/internal.h | 1 + fs/proc/task_mmu.c | 30 +++++++++------ fs/proc/task_nommu.c | 27 ++++++++----- fs/userfaultfd.c | 24 +++++++----- include/linux/mm_types.h | 3 +- ipc/shm.c | 13 +++++-- kernel/acct.c | 6 ++- kernel/events/core.c | 6 ++- kernel/events/uprobes.c | 24 ++++++++---- kernel/exit.c | 10 +++-- kernel/fork.c | 21 ++++++---- kernel/futex.c | 8 ++-- kernel/sched/fair.c | 7 ++-- kernel/sys.c | 31 ++++++++++----- kernel/trace/trace_output.c | 6 ++- mm/filemap.c | 4 +- mm/frame_vector.c | 9 +++-- mm/gup.c | 22 ++++++----- mm/init-mm.c | 2 +- mm/khugepaged.c | 44 +++++++++++++-------- mm/ksm.c | 45 ++++++++++++++-------- mm/madvise.c | 28 ++++++++------ mm/memcontrol.c | 14 ++++--- mm/memory.c | 21 +++++++--- mm/mempolicy.c | 30 +++++++++------ mm/migrate.c | 12 ++++-- mm/mincore.c | 6 ++- mm/mlock.c | 25 ++++++++---- mm/mmap.c | 43 ++++++++++++++------- mm/mmu_notifier.c | 6 ++- mm/mprotect.c | 19 ++++++--- mm/mremap.c | 6 ++- mm/msync.c | 10 +++-- mm/nommu.c | 31 ++++++++++----- mm/oom_kill.c | 9 +++-- mm/process_vm_access.c | 8 ++-- mm/shmem.c | 3 +- mm/swapfile.c | 8 ++-- mm/userfaultfd.c | 19 +++++---- mm/util.c | 15 ++++++-- virt/kvm/async_pf.c | 8 ++-- virt/kvm/kvm_main.c | 21 ++++++---- 102 files changed, 838 insertions(+), 452 deletions(-)
arch/powerpc/kernel/vdso.c +5 -3
@@ -154,6 +154,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)struct page **vdso_pagelist;unsigned long vdso_pages;unsigned long vdso_base; +struct range_rwlock range;int rc;if (!vdso_ready)
@@ -196,7 +197,8 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)* and end up putting it elsewhere. * Add enough to the size so that the result can be aligned. */ -if (down_write_killable (&mm->mmap_sem)) +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return -EINTR ; vdso_base = get_unmapped_area(NULL, vdso_base, (vdso_pages << PAGE_SHIFT) +
@@ -236,11 +238,11 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) goto fail_mmapsem; } -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return 0 ; fail_mmapsem: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return rc; }
arch/powerpc/kvm/book3s_64_mmu_hv.c +4 -2
@@ -485,6 +485,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,struct vm_area_struct *vma;unsigned long rcbits;long mmio_update; +struct range_rwlock range;if (kvm_is_radix (kvm))return kvmppc_book3s_radix_page_fault (run, vcpu, ea, dsisr);
@@ -568,7 +569,8 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, npages =get_user_pages_fast (hva,1 , writing, pages);if (npages <1 ) {/* Check if it's an I/O mapping */ -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); vma =find_vma (current->mm, hva);if (vma && vma->vm_start <= hva && hva + psize <= vma->vm_end && (vma->vm_flags &VM_PFNMAP )) {
@@ -578,7 +580,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, is_ci =pte_ci (__pte ((pgprot_val (vma->vm_page_prot)))); write_ok = vma->vm_flags &VM_WRITE ; } -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (!pfn) goto out_put; }else {
arch/powerpc/kvm/book3s_64_mmu_radix.c +4 -2
@@ -305,6 +305,7 @@ int kvmppc_book3s_radix_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,pte_t pte, *ptep;unsigned long pgflags;unsigned int shift, level; +struct range_rwlock range;/* Check for unusual errors */ if (dsisr &DSISR_UNSUPP_MMU ) {
@@ -394,7 +395,8 @@ int kvmppc_book3s_radix_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, npages =get_user_pages_fast (hva,1 , writing, pages);if (npages <1 ) {/* Check if it's an I/O mapping */ -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); vma =find_vma (current->mm, hva);if (vma && vma->vm_start <= hva && hva < vma->vm_end && (vma->vm_flags &VM_PFNMAP )) {
@@ -402,7 +404,7 @@ int kvmppc_book3s_radix_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, ((hva - vma->vm_start) >>PAGE_SHIFT ); pgflags =pgprot_val (vma->vm_page_prot); } -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (!pfn)return -EFAULT ; }else {
arch/powerpc/kvm/book3s_64_vio.c +4 -2
@@ -57,11 +57,13 @@ static unsigned long kvmppc_stt_pages(unsigned long tce_pages)static long kvmppc_account_memlimit (unsigned long stt_pages,bool inc) {long ret =0 ; +struct range_rwlock range;if (!current || !current->mm)return ret;/* process exited */ -down_write (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (¤t->mm->mmap_rw_tree, &range);if (inc) {unsigned long locked, lock_limit;
@@ -86,7 +88,7 @@ static long kvmppc_account_memlimit(unsigned long stt_pages, bool inc)rlimit (RLIMIT_MEMLOCK ), ret ? " - exceeded" : ""); -up_write (¤t->mm->mmap_sem); +range_write_unlock (¤t->mm->mmap_rw_tree, &range);return ret; }
arch/powerpc/kvm/book3s_hv.c +5 -3
@@ -3192,6 +3192,7 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu)unsigned long lpcr =0 , senc;unsigned long psize, porder;int srcu_idx; +struct range_rwlock range;mutex_lock (&kvm->lock);if (kvm->arch.hpte_setup_done)
@@ -3228,7 +3229,8 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu)/* Look up the VMA for the start of this memory slot */ hva = memslot->userspace_addr; -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); vma =find_vma (current->mm, hva);if (!vma || vma->vm_start > hva || (vma->vm_flags &VM_IO )) goto up_out;
@@ -3236,7 +3238,7 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu) psize =vma_kernel_pagesize (vma); porder =__ilog2 (psize); -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);/* We can handle 4k, 64k or 16M pages in the VRMA */ err = -EINVAL ;
@@ -3270,7 +3272,7 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu)return err; up_out: -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); goto out_srcu; }
arch/powerpc/kvm/e500_mmu_host.c +5 -2
@@ -357,7 +357,10 @@ static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500,if (tlbsel ==1 ) {struct vm_area_struct *vma; -down_read (¤t->mm->mmap_sem); +struct range_rwlock range; + +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); vma =find_vma (current->mm, hva); if (vma && hva >= vma->vm_start &&
@@ -443,7 +446,7 @@ static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500, tsize =max (BOOK3E_PAGESZ_4K , tsize & ~1 ); } -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); }if (likely (!pfnmap)) {
arch/powerpc/mm/copro_fault.c +4 -2
@@ -37,6 +37,7 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea, unsigned long dsisr, unsigned *flt) { struct vm_area_struct *vma; +struct range_rwlock range;unsigned long is_write;int ret;
@@ -46,7 +47,8 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,if (mm->pgd ==NULL )return -EFAULT ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); ret = -EFAULT ; vma =find_vma (mm, ea);if (!vma)
@@ -95,7 +97,7 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea, current->min_flt++; out_unlock: -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return ret; }EXPORT_SYMBOL_GPL (copro_handle_mm_fault);
arch/powerpc/mm/fault.c +7 -5
@@ -208,6 +208,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,int is_exec = trap ==0x400 ;int fault;int rc =0 , store_update_sp =0 ; +struct range_rwlock range;#if !(defined(CONFIG_4xx) || defined(CONFIG_BOOKE ))/*
@@ -308,12 +309,13 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,* source. If this is invalid we can skip the address space check, * thus avoiding the deadlock. */ -if (!down_read_trylock (&mm->mmap_sem)) { +range_rwlock_init_full (&range);/* XXX finer grain required here */ +if (!range_read_trylock (&mm->mmap_rw_tree, &range)) {if (!user_mode (regs) && !search_exception_tables (regs->nip)) goto bad_area_nosemaphore; retry: -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); }else {/* * The above down_read_trylock() might have succeeded in
@@ -446,7 +448,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, &range);/* * Handle the retry right now, the mmap_sem has been released in that
@@ -466,7 +468,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address, }/* We will enter mm_fault_error() below */ }else -up_read (¤t->mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);if (unlikely (fault & (VM_FAULT_RETRY |VM_FAULT_ERROR ))) {if (fault &VM_FAULT_SIGSEGV )
@@ -505,7 +507,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address, goto bail; bad_area: -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); bad_area_nosemaphore:/* User mode accesses cause a SIGSEGV */
arch/powerpc/mm/mmu_context_iommu.c +4 -2
@@ -36,11 +36,13 @@ static long mm_iommu_adjust_locked_vm(struct mm_struct *mm, unsigned long npages, bool incr) { long ret = 0, locked, lock_limit; +struct range_rwlock range;if (!npages)return 0 ; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range);if (incr) { locked = mm->locked_vm + npages;
@@ -61,7 +63,7 @@ static long mm_iommu_adjust_locked_vm(struct mm_struct *mm, npages <<PAGE_SHIFT , mm->locked_vm <<PAGE_SHIFT , rlimit(RLIMIT_MEMLOCK)); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }
arch/powerpc/mm/subpage-prot.c +10 -6
@@ -98,8 +98,10 @@ static void subpage_prot_clear(unsigned long addr, unsigned long len)unsigned long i;size_t nw;unsigned long next, limit; +struct range_rwlock range; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); limit = addr + len;if (limit > spt->maxaddr) limit = spt->maxaddr;
@@ -127,7 +129,7 @@ static void subpage_prot_clear(unsigned long addr, unsigned long len)/* now flush any existing HPTEs for the range */ hpte_flush_range (mm, addr, nw); } -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); }#ifdef CONFIG_TRANSPARENT_HUGEPAGE
@@ -194,6 +196,7 @@ long sys_subpage_prot(unsigned long addr, unsigned long len, u32 __user *map)size_t nw;unsigned long next, limit;int err; +struct range_rwlock range;/* Check parameters */ if ((addr & ~PAGE_MASK) || (len & ~PAGE_MASK) ||
@@ -212,7 +215,8 @@ long sys_subpage_prot(unsigned long addr, unsigned long len, u32 __user *map)if (!access_ok (VERIFY_READ , map, (len >>PAGE_SHIFT ) *sizeof (u32)))return -EFAULT ; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range);subpage_mark_vma_nohuge (mm, addr, len);for (limit = addr + len; addr < limit; addr = next) { next =pmd_addr_end (addr, limit);
@@ -247,11 +251,11 @@ long sys_subpage_prot(unsigned long addr, unsigned long len, u32 __user *map)if (addr + (nw <<PAGE_SHIFT ) > next) nw = (next - addr) >>PAGE_SHIFT ; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (__copy_from_user (spp, map, nw *sizeof (u32)))return -EFAULT ; map += nw; -down_write (&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &range);/* now flush any existing HPTEs for the range */ hpte_flush_range (mm, addr, nw);
@@ -260,6 +264,6 @@ long sys_subpage_prot(unsigned long addr, unsigned long len, u32 __user *map) spt->maxaddr = limit; err =0 ; out: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return err; }
arch/powerpc/oprofile/cell/spu_task_sync.c +5 -3
@@ -325,6 +325,7 @@ get_exec_dcookie_and_offset(struct spu *spu, unsigned int *offsetp,struct vm_area_struct *vma;struct file *exe_file;struct mm_struct *mm = spu->mm; +struct range_rwlock range;if (!mm) goto out;
@@ -336,7 +337,8 @@ get_exec_dcookie_and_offset(struct spu *spu, unsigned int *offsetp,fput (exe_file); } -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (vma = mm->mmap; vma; vma = vma->vm_next) {if (vma->vm_start > spu_ref || vma->vm_end <= spu_ref)continue ;
@@ -353,13 +355,13 @@ get_exec_dcookie_and_offset(struct spu *spu, unsigned int *offsetp, *spu_bin_dcookie =fast_get_dcookie (&vma->vm_file->f_path);pr_debug ("got dcookie for %pD\n" , vma->vm_file); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); out:return app_cookie; fail_no_image_cookie: -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); printk(KERN_ERR "SPU_PROF: " "%s, line %d: Cannot find dcookie for SPU binary\n",
arch/powerpc/platforms/cell/spufs/file.c +2 -2
@@ -347,11 +347,11 @@ static int spufs_ps_fault(struct vm_fault *vmf, goto refault;if (ctx->state ==SPU_STATE_SAVED ) { -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, vmf->lockrange);spu_context_nospu_trace (spufs_ps_fault__sleep, ctx); ret =spufs_wait (ctx->run_wq, ctx->state ==SPU_STATE_RUNNABLE );spu_context_trace (spufs_ps_fault__wake, ctx, ctx->spu); -down_read (¤t->mm->mmap_sem); +range_read_lock (¤t->mm->mmap_rw_tree, vmf->lockrange); }else { area = ctx->spu->problem_phys + ps_offs;vm_insert_pfn (vmf->vma, vmf->address, (area + offset) >>PAGE_SHIFT );
arch/x86/entry/vdso/vma.c +9 -5
@@ -148,10 +148,12 @@ static int map_vdso(const struct vdso_image *image, unsigned long addr) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; +struct range_rwlock range;unsigned long text_start;int ret =0 ; -if (down_write_killable (&mm->mmap_sem)) +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return -EINTR ; addr = get_unmapped_area(NULL, addr,
@@ -194,7 +196,7 @@ static int map_vdso(const struct vdso_image *image, unsigned long addr) } up_fail: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }
@@ -255,8 +257,10 @@ int map_vdso_once(const struct vdso_image *image, unsigned long addr) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; +struct range_rwlock range; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range);/* * Check if we have already mapped vdso blob - fail to prevent * abusing from userspace install_speciall_mapping, which may
@@ -267,11 +271,11 @@ int map_vdso_once(const struct vdso_image *image, unsigned long addr)for (vma = mm->mmap; vma; vma = vma->vm_next) {if (vma_is_special_mapping (vma, &vdso_mapping) ||vma_is_special_mapping (vma, &vvar_mapping)) { -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return -EEXIST ; } } -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return map_vdso (image, addr); }
arch/x86/kernel/tboot.c +1 -1
@@ -104,7 +104,7 @@ static struct mm_struct tboot_mm = { .pgd = swapper_pg_dir, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), - .mmap_sem = __RWSEM_INITIALIZER(init_mm.mmap_sem), + .mmap_rw_tree = __RANGE_RWLOCK_TREE_INITIALIZER(init_mm.mmap_rw_tree), .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), };
arch/x86/kernel/vm86_32.c +4 -2
@@ -162,6 +162,7 @@ void save_v86_state(struct kernel_vm86_regs *regs, int retval)static void mark_screen_rdonly (struct mm_struct *mm) {struct vm_area_struct *vma; +struct range_rwlock range;spinlock_t *ptl;pgd_t *pgd;pud_t *pud;
@@ -169,7 +170,8 @@ static void mark_screen_rdonly(struct mm_struct *mm)pte_t *pte;int i; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); pgd =pgd_offset (mm,0xA0000 );if (pgd_none_or_clear_bad (pgd)) goto out;
@@ -192,7 +194,7 @@ static void mark_screen_rdonly(struct mm_struct *mm) }pte_unmap_unlock (pte, ptl); out: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);flush_tlb (); }
arch/x86/mm/fault.c +22 -17
@@ -922,7 +922,8 @@ bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code,static void __bad_area (struct pt_regs *regs,unsigned long error_code, -unsigned long address,struct vm_area_struct *vma,int si_code) +unsigned long address,struct vm_area_struct *vma,int si_code, +struct range_rwlock *range) {struct mm_struct *mm = current->mm;
@@ -930,15 +931,16 @@ __bad_area(struct pt_regs *regs, unsigned long error_code,* Something tried to access memory that isn't in our memory map.. * Fix it, but check if it's kernel or user first.. */ -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, range);__bad_area_nosemaphore (regs, error_code, address, vma, si_code); } static noinline void -bad_area (struct pt_regs *regs,unsigned long error_code,unsigned long address) +bad_area (struct pt_regs *regs,unsigned long error_code,unsigned long address, +struct range_rwlock *range) { -__bad_area (regs, error_code, address,NULL ,SEGV_MAPERR ); +__bad_area (regs, error_code, address,NULL ,SEGV_MAPERR , range); } static inline bool bad_area_access_from_pkeys(unsigned long error_code,
@@ -960,7 +962,8 @@ static inline bool bad_area_access_from_pkeys(unsigned long error_code, static noinline voidbad_area_access_error (struct pt_regs *regs,unsigned long error_code, -unsigned long address,struct vm_area_struct *vma) +unsigned long address,struct vm_area_struct *vma, +struct range_rwlock *range) {/* * This OSPKE check is not strictly necessary at runtime.
@@ -968,9 +971,9 @@ bad_area_access_error(struct pt_regs *regs, unsigned long error_code,* if pkeys are compiled out. */ if (bad_area_access_from_pkeys (error_code, vma)) -__bad_area (regs, error_code, address, vma,SEGV_PKUERR ); +__bad_area (regs, error_code, address, vma,SEGV_PKUERR , range);else -__bad_area (regs, error_code, address, vma,SEGV_ACCERR ); +__bad_area (regs, error_code, address, vma,SEGV_ACCERR , range); } static void
@@ -1218,6 +1221,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,struct vm_area_struct *vma;struct task_struct *tsk;struct mm_struct *mm; +struct range_rwlock range;int fault, major =0 ;unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ;
@@ -1230,7 +1234,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,*/ if (kmemcheck_active (regs))kmemcheck_hide (regs); -prefetchw (&mm->mmap_sem); +prefetchw (&mm->mmap_rw_tree);if (unlikely (kmmio_fault (regs, address)))return ;
@@ -1333,14 +1337,15 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,* validate the source. If this is invalid we can skip the address * space check, thus avoiding the deadlock: */ -if (unlikely (!down_read_trylock (&mm->mmap_sem))) { +range_rwlock_init_full (&range); +if (unlikely (!range_read_trylock (&mm->mmap_rw_tree, &range))) {if ((error_code &PF_USER ) ==0 && !search_exception_tables (regs->ip)) {bad_area_nosemaphore (regs, error_code, address,NULL );return ; } retry: -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); }else {/* * The above down_read_trylock() might have succeeded in
@@ -1352,13 +1357,13 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, vma =find_vma (mm, address);if (unlikely (!vma)) { -bad_area (regs, error_code, address); +bad_area (regs, error_code, address, &range);return ; }if (likely (vma->vm_start <= address)) goto good_area;if (unlikely (!(vma->vm_flags &VM_GROWSDOWN ))) { -bad_area (regs, error_code, address); +bad_area (regs, error_code, address, &range);return ; }if (error_code &PF_USER ) {
@@ -1369,12 +1374,12 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,* 32 pointers and then decrements %sp by 65535.) */ if (unlikely (address +65536 +32 *sizeof (unsigned long ) < regs->sp)) { -bad_area (regs, error_code, address); +bad_area (regs, error_code, address, &range);return ; } }if (unlikely (expand_stack (vma, address))) { -bad_area (regs, error_code, address); +bad_area (regs, error_code, address, &range);return ; }
@@ -1384,7 +1389,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,*/ good_area:if (unlikely (access_error (error_code, vma))) { -bad_area_access_error (regs, error_code, address, vma); +bad_area_access_error (regs, error_code, address, vma, &range);return ; }
@@ -1394,7 +1399,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,* the fault. Since we never set FAULT_FLAG_RETRY_NOWAIT, if * we get VM_FAULT_RETRY back, the mmap_sem has been unlocked. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, &range); major |= fault &VM_FAULT_MAJOR ;/*
@@ -1420,7 +1425,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,return ; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);if (unlikely (fault &VM_FAULT_ERROR )) {mm_fault_error (regs, error_code, address, vma, fault);return ;
arch/x86/mm/mpx.c +12 -6
@@ -44,16 +44,18 @@ static inline unsigned long mpx_bt_size_bytes(struct mm_struct *mm)static unsigned long mpx_mmap (unsigned long len) {struct mm_struct *mm = current->mm; +struct range_rwlock range;unsigned long addr, populate;/* Only bounds table can be allocated here */ if (len !=mpx_bt_size_bytes (mm))return -EINVAL ; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); addr =do_mmap (NULL ,0 , len,PROT_READ |PROT_WRITE ,MAP_ANONYMOUS |MAP_PRIVATE ,VM_MPX ,0 , &populate,NULL ); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (populate)mm_populate (addr, populate);
@@ -340,6 +342,7 @@ int mpx_enable_management(void) {void __user *bd_base =MPX_INVALID_BOUNDS_DIR ;struct mm_struct *mm = current->mm; +struct range_rwlock range;int ret =0 ;/*
@@ -354,25 +357,28 @@ int mpx_enable_management(void)* unmap path; we can just use mm->context.bd_addr instead. */ bd_base =mpx_get_bounds_dir (); -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); mm->context.bd_addr = bd_base;if (mm->context.bd_addr ==MPX_INVALID_BOUNDS_DIR ) ret = -ENXIO ; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }int mpx_disable_management (void ) {struct mm_struct *mm = current->mm; +struct range_rwlock range;if (!cpu_feature_enabled (X86_FEATURE_MPX ))return -ENXIO ; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); mm->context.bd_addr =MPX_INVALID_BOUNDS_DIR ; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return 0 ; }
drivers/android/binder.c +5 -3
@@ -581,6 +581,7 @@ static int binder_update_page_range(struct binder_proc *proc, int allocate,unsigned long user_page_addr;struct page **page;struct mm_struct *mm; +struct range_rwlock range; binder_debug(BINDER_DEBUG_BUFFER_ALLOC, "%d: %s pages %p-%p\n", proc->pid,
@@ -597,7 +598,8 @@ static int binder_update_page_range(struct binder_proc *proc, int allocate, mm =get_task_mm (proc->tsk);if (mm) { -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); vma = proc->vma;if (vma && mm != proc->vma_vm_mm) { pr_err("%d: vma mm and task mm mismatch\n",
@@ -647,7 +649,7 @@ static int binder_update_page_range(struct binder_proc *proc, int allocate,/* vm_insert_page does not seem to increment the refcount */ }if (mm) { -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm); }return 0 ;
@@ -669,7 +671,7 @@ static int binder_update_page_range(struct binder_proc *proc, int allocate, } err_no_vma:if (mm) { -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm); }return -ENOMEM ;
drivers/firmware/efi/arm-runtime.c +1 -1
@@ -34,7 +34,7 @@ static struct mm_struct efi_mm = { .mm_rb = RB_ROOT, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), - .mmap_sem = __RWSEM_INITIALIZER(efi_mm.mmap_sem), + .mmap_rw_tree = __RANGE_RWLOCK_TREE_INITIALIZER(efi_mm.mmap_rw_tree), .page_table_lock = __SPIN_LOCK_UNLOCKED(efi_mm.page_table_lock), .mmlist = LIST_HEAD_INIT(efi_mm.mmlist), };
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +6 -3
@@ -509,6 +509,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,struct amdgpu_fpriv *fpriv = p->filp->driver_priv;struct amdgpu_bo_list_entry *e;struct list_head duplicates; +struct range_rwlock range;bool need_mmap_lock = false;unsigned i, tries =10 ;int r;
@@ -528,8 +529,10 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,if (p->uf_entry.robj)list_add (&p->uf_entry.tv.head, &p->validated); -if (need_mmap_lock) -down_read (¤t->mm->mmap_sem); +if (need_mmap_lock) { +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); + }while (1 ) {struct list_head need_pages;
@@ -686,7 +689,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, error_free_pages:if (need_mmap_lock) -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (p->bo_list) { for (i = p->bo_list->first_userptr;
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +5 -3
@@ -269,6 +269,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,struct drm_amdgpu_gem_userptr *args = data;struct drm_gem_object *gobj;struct amdgpu_bo *bo; +struct range_rwlock range;uint32_t handle;int r;
@@ -309,7 +310,8 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data, }if (args->flags &AMDGPU_GEM_USERPTR_VALIDATE ) { -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); r =amdgpu_ttm_tt_get_user_pages (bo->tbo.ttm, bo->tbo.ttm->pages);
@@ -326,7 +328,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,if (r) goto free_pages; -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); } r =drm_gem_handle_create (filp, gobj, &handle);
@@ -342,7 +344,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,release_pages (bo->tbo.ttm->pages, bo->tbo.ttm->num_pages, false); unlock_mmap_sem: -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); release_object:drm_gem_object_unreference_unlocked (gobj);
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +5 -3
@@ -229,10 +229,12 @@ static struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev) {struct mm_struct *mm = current->mm;struct amdgpu_mn *rmn; +struct range_rwlock range;int r; +range_rwlock_init_full (&range);mutex_lock (&adev->mn_lock); -if (down_write_killable (&mm->mmap_sem)) { +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range)) {mutex_unlock (&adev->mn_lock);return ERR_PTR (-EINTR ); }
@@ -260,13 +262,13 @@ static struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev)hash_add (adev->mn_hash, &rmn->node, (unsigned long )mm); release_locks: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mutex_unlock (&adev->mn_lock);return rmn; free_rmn: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mutex_unlock (&adev->mn_lock);kfree (rmn);
drivers/gpu/drm/amd/amdkfd/kfd_events.c +4 -2
@@ -897,6 +897,7 @@ void kfd_signal_iommu_event(struct kfd_dev *dev, unsigned int pasid, {struct kfd_hsa_memory_exception_data memory_exception_data;struct vm_area_struct *vma; +struct range_rwlock range;/* * Because we are called from arbitrary context (workqueue) as opposed
@@ -910,7 +911,8 @@ void kfd_signal_iommu_event(struct kfd_dev *dev, unsigned int pasid,memset (&memory_exception_data,0 ,sizeof (memory_exception_data)); -down_read (&p->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&p->mm->mmap_rw_tree, &range); vma =find_vma (p->mm, address); memory_exception_data.gpu_id = dev->id;
@@ -937,7 +939,7 @@ void kfd_signal_iommu_event(struct kfd_dev *dev, unsigned int pasid, } } -up_read (&p->mm->mmap_sem); +range_read_unlock (&p->mm->mmap_rw_tree, &range);mutex_lock (&p->event_mutex);
drivers/gpu/drm/amd/amdkfd/kfd_process.c +4 -2
@@ -78,6 +78,7 @@ void kfd_process_destroy_wq(void)struct kfd_process *kfd_create_process (const struct task_struct *thread) {struct kfd_process *process; +struct range_rwlock range;BUG_ON (!kfd_process_wq);
@@ -89,7 +90,8 @@ struct kfd_process *kfd_create_process(const struct task_struct *thread)return ERR_PTR (-EINVAL );/* Take mmap_sem because we call __mmu_notifier_register inside */ -down_write (&thread->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&thread->mm->mmap_rw_tree, &range);/* * take kfd processes mutex before starting of process creation
@@ -108,7 +110,7 @@ struct kfd_process *kfd_create_process(const struct task_struct *thread)mutex_unlock (&kfd_processes_mutex); -up_write (&thread->mm->mmap_sem); +range_write_unlock (&thread->mm->mmap_rw_tree, &range);return process; }
drivers/gpu/drm/etnaviv/etnaviv_gem.c +4 -2
@@ -745,6 +745,7 @@ static struct page **etnaviv_gem_userptr_do_get_pages( {int ret =0 , pinned, npages = etnaviv_obj->base.size >>PAGE_SHIFT ;struct page **pvec; +struct range_rwlock range;uintptr_t ptr;unsigned int flags =0 ;
@@ -758,7 +759,8 @@ static struct page **etnaviv_gem_userptr_do_get_pages( pinned =0 ; ptr = etnaviv_obj->userptr.ptr; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);while (pinned < npages) { ret = get_user_pages_remote(task, mm, ptr, npages - pinned, flags, pvec + pinned, NULL, NULL,
@@ -769,7 +771,7 @@ static struct page **etnaviv_gem_userptr_do_get_pages( ptr += ret *PAGE_SIZE ; pinned += ret; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);if (ret <0 ) {release_pages (pvec, pinned,0 );
drivers/gpu/drm/i915/i915_gem.c +4 -2
@@ -1673,8 +1673,10 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,if (args->flags &I915_MMAP_WC ) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; +struct range_rwlock range; -if (down_write_killable (&mm->mmap_sem)) { +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range)) {i915_gem_object_put (obj);return -EINTR ; }
@@ -1684,7 +1686,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,pgprot_writecombine (vm_get_page_prot (vma->vm_flags)); else addr = -ENOMEM ; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);/* This may race, but that's ok, it only gets set */ WRITE_ONCE (obj->frontbuffer_ggtt_origin,ORIGIN_CPU );
drivers/gpu/drm/i915/i915_gem_userptr.c +8 -4
@@ -203,12 +203,14 @@ static struct i915_mmu_notifier * i915_mmu_notifier_find(struct i915_mm_struct *mm) {struct i915_mmu_notifier *mn = mm->mn; +struct range_rwlock range; mn = mm->mn;if (mn)return mn; -down_write (&mm->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mm->mmap_rw_tree, &range);mutex_lock (&mm->i915->mm_lock);if ((mn = mm->mn) ==NULL ) { mn =i915_mmu_notifier_create (mm->mm);
@@ -216,7 +218,7 @@ i915_mmu_notifier_find(struct i915_mm_struct *mm) mm->mn = mn; }mutex_unlock (&mm->i915->mm_lock); -up_write (&mm->mm->mmap_sem); +range_write_unlock (&mm->mm->mmap_rw_tree, &range);return mn; }
@@ -501,6 +503,7 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work) pvec =drm_malloc_gfp (npages,sizeof (struct page *),GFP_TEMPORARY );if (pvec !=NULL ) { +struct range_rwlock range;struct mm_struct *mm = obj->userptr.mm->mm;unsigned int flags =0 ;
@@ -509,7 +512,8 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work) ret = -EFAULT ;if (mmget_not_zero (mm)) { -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);while (pinned < npages) { ret = get_user_pages_remote (work->task, mm,
@@ -522,7 +526,7 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work) pinned += ret; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);mmput (mm); } }
drivers/gpu/drm/radeon/radeon_cs.c +6 -3
@@ -76,6 +76,7 @@ static int radeon_cs_parser_relocs(struct radeon_cs_parser *p) {struct radeon_cs_chunk *chunk;struct radeon_cs_buckets buckets; +struct range_rwlock range;unsigned i;bool need_mmap_lock = false;int r;
@@ -176,13 +177,15 @@ static int radeon_cs_parser_relocs(struct radeon_cs_parser *p)if (p->cs_flags &RADEON_CS_USE_VM ) p->vm_bos =radeon_vm_get_bos (p->rdev, p->ib.vm, &p->validated); -if (need_mmap_lock) -down_read (¤t->mm->mmap_sem); +if (need_mmap_lock) { +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); + } r =radeon_bo_list_validate (p->rdev, &p->ticket, &p->validated, p->ring);if (need_mmap_lock) -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);return r; }
drivers/gpu/drm/radeon/radeon_gem.c +5 -3
@@ -285,6 +285,7 @@ int radeon_gem_userptr_ioctl(struct drm_device *dev, void *data,struct drm_radeon_gem_userptr *args = data;struct drm_gem_object *gobj;struct radeon_bo *bo; +struct range_rwlock range;uint32_t handle;int r;
@@ -331,17 +332,18 @@ int radeon_gem_userptr_ioctl(struct drm_device *dev, void *data, }if (args->flags &RADEON_GEM_USERPTR_VALIDATE ) { -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); r =radeon_bo_reserve (bo, true);if (r) { -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); goto release_object; }radeon_ttm_placement_from_domain (bo,RADEON_GEM_DOMAIN_GTT ); r =ttm_bo_validate (&bo->tbo, &bo->placement, true, false);radeon_bo_unreserve (bo); -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (r) goto release_object; }
drivers/gpu/drm/radeon/radeon_mn.c +5 -3
@@ -184,9 +184,11 @@ static struct radeon_mn *radeon_mn_get(struct radeon_device *rdev) {struct mm_struct *mm = current->mm;struct radeon_mn *rmn; +struct range_rwlock range;int r; -if (down_write_killable (&mm->mmap_sem)) +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return ERR_PTR (-EINTR );mutex_lock (&rdev->mn_lock);
@@ -215,13 +217,13 @@ static struct radeon_mn *radeon_mn_get(struct radeon_device *rdev) release_locks:mutex_unlock (&rdev->mn_lock); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return rmn; free_rmn:mutex_unlock (&rdev->mn_lock); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);kfree (rmn);return ERR_PTR (r);
drivers/gpu/drm/ttm/ttm_bo_vm.c +4 -2
@@ -66,7 +66,8 @@ static int ttm_bo_vm_fault_idle(struct ttm_buffer_object *bo, goto out_unlock;ttm_bo_reference (bo); -up_read (&vmf->vma->vm_mm->mmap_sem); +range_read_unlock (&vmf->vma->vm_mm->mmap_rw_tree, + vmf->lockrange); (void )dma_fence_wait (bo->moving, true);ttm_bo_unreserve (bo);ttm_bo_unref (&bo);
@@ -124,7 +125,8 @@ static int ttm_bo_vm_fault(struct vm_fault *vmf)if (vmf->flags &FAULT_FLAG_ALLOW_RETRY ) {if (!(vmf->flags &FAULT_FLAG_RETRY_NOWAIT )) {ttm_bo_reference (bo); -up_read (&vmf->vma->vm_mm->mmap_sem); +range_read_unlock (&vmf->vma->vm_mm->mmap_rw_tree, + vmf->lockrange); (void )ttm_bo_wait_unreserved (bo);ttm_bo_unref (&bo); }
drivers/gpu/drm/via/via_dmablit.c +4 -2
@@ -230,6 +230,7 @@ via_fire_dmablit(struct drm_device *dev, drm_via_sg_info_t *vsg, int engine)static int via_lock_all_dma_pages (drm_via_sg_info_t *vsg,drm_via_dmablit_t *xfer) { +struct range_rwlock range;int ret;unsigned long first_pfn =VIA_PFN (xfer->mem_addr); vsg->num_pages = VIA_PFN(xfer->mem_addr + (xfer->num_lines * xfer->mem_stride - 1)) -
@@ -238,13 +239,14 @@ via_lock_all_dma_pages(drm_via_sg_info_t *vsg, drm_via_dmablit_t *xfer) vsg->pages =vzalloc (sizeof (struct page *) * vsg->num_pages);if (NULL == vsg->pages)return -ENOMEM ; -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); ret =get_user_pages ((unsigned long )xfer->mem_addr, vsg->num_pages, (vsg->direction ==DMA_FROM_DEVICE ) ?FOLL_WRITE :0 , vsg->pages,NULL ); -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (ret != vsg->num_pages) {if (ret <0 )return ret;
drivers/infiniband/core/umem.c +13 -7
@@ -86,6 +86,7 @@ struct ib_umem *ib_umem_get(struct ib_ucontext *context, unsigned long addr,struct ib_umem *umem;struct page **page_list;struct vm_area_struct **vma_list; +struct range_rwlock range;unsigned long locked;unsigned long lock_limit;unsigned long cur_base;
@@ -163,7 +164,8 @@ struct ib_umem *ib_umem_get(struct ib_ucontext *context, unsigned long addr, npages =ib_umem_num_pages (umem); -down_write (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (¤t->mm->mmap_rw_tree, &range); locked = npages + current->mm->pinned_vm; lock_limit =rlimit (RLIMIT_MEMLOCK ) >>PAGE_SHIFT ;
@@ -236,7 +238,7 @@ struct ib_umem *ib_umem_get(struct ib_ucontext *context, unsigned long addr, }else current->mm->pinned_vm = locked; -up_write (¤t->mm->mmap_sem); +range_write_unlock (¤t->mm->mmap_rw_tree, &range);if (vma_list)free_page ((unsigned long ) vma_list);free_page ((unsigned long ) page_list);
@@ -248,10 +250,12 @@ EXPORT_SYMBOL(ib_umem_get);static void ib_umem_account (struct work_struct *work) { struct ib_umem *umem = container_of(work, struct ib_umem, work); +struct range_rwlock range; -down_write (&umem->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&umem->mm->mmap_rw_tree, &range); umem->mm->pinned_vm -= umem->diff; -up_write (&umem->mm->mmap_sem); +range_write_unlock (&umem->mm->mmap_rw_tree, &range);mmput (umem->mm);kfree (umem); }
@@ -265,6 +269,7 @@ void ib_umem_release(struct ib_umem *umem)struct ib_ucontext *context = umem->context;struct mm_struct *mm;struct task_struct *task; +struct range_rwlock range;unsigned long diff;if (umem->odp_data) {
@@ -293,8 +298,9 @@ void ib_umem_release(struct ib_umem *umem)* up here and not be able to take the mmap_sem. In that case * we defer the vm_locked accounting to the system workqueue. */ +range_rwlock_init_full (&range);if (context->closing) { -if (!down_write_trylock (&mm->mmap_sem)) { +if (!range_write_trylock (&mm->mmap_rw_tree, &range)) {INIT_WORK (&umem->work, ib_umem_account); umem->mm = mm; umem->diff = diff;
@@ -303,10 +309,10 @@ void ib_umem_release(struct ib_umem *umem)return ; } }else -down_write (&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &range); mm->pinned_vm -= diff; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm); out:kfree (umem);
drivers/infiniband/core/umem_odp.c +4 -2
@@ -595,6 +595,7 @@ int ib_umem_odp_map_dma_pages(struct ib_umem *umem, u64 user_virt, u64 bcnt,struct task_struct *owning_process =NULL ;struct mm_struct *owning_mm =NULL ;struct page **local_page_list =NULL ; +struct range_rwlock range;u64 off;int j, k, ret =0 , start_idx, npages =0 ;u64 base_virt_addr;
@@ -639,7 +640,8 @@ int ib_umem_odp_map_dma_pages(struct ib_umem *umem, u64 user_virt, u64 bcnt,min_t (size_t,ALIGN (bcnt,PAGE_SIZE ) /PAGE_SIZE ,PAGE_SIZE /sizeof (struct page *)); -down_read (&owning_mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&owning_mm->mmap_rw_tree, &range);/* * Note: this might result in redundent page getting. We can * avoid this by checking dma_list to be 0 before calling
@@ -650,7 +652,7 @@ int ib_umem_odp_map_dma_pages(struct ib_umem *umem, u64 user_virt, u64 bcnt, npages =get_user_pages_remote (owning_process, owning_mm, user_virt, gup_num_pages, flags, local_page_list,NULL ,NULL ,NULL ); -up_read (&owning_mm->mmap_sem); +range_read_unlock (&owning_mm->mmap_rw_tree, &range);if (npages <0 )break ;
drivers/infiniband/hw/hfi1/user_pages.c +12 -6
@@ -71,6 +71,7 @@ MODULE_PARM_DESC(cache_size, "Send and receive side cache size limit (in MB)");bool hfi1_can_pin_pages (struct hfi1_devdata *dd,struct mm_struct *mm,u32 nlocked,u32 npages) { +struct range_rwlock range;unsigned long ulimit =rlimit (RLIMIT_MEMLOCK ), pinned, cache_limit, size = (cache_size * (1UL <<20 ));/* convert to bytes */ unsigned usr_ctxts = dd->num_rcv_contexts - dd->first_user_ctxt;
@@ -90,9 +91,10 @@ bool hfi1_can_pin_pages(struct hfi1_devdata *dd, struct mm_struct *mm,/* Convert to number of pages */ size =DIV_ROUND_UP (size,PAGE_SIZE ); -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); pinned = mm->pinned_vm; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);/* First, check the absolute limit against all pinned pages. */ if (pinned + npages >= ulimit && !can_lock)
@@ -104,15 +106,17 @@ bool hfi1_can_pin_pages(struct hfi1_devdata *dd, struct mm_struct *mm,int hfi1_acquire_user_pages (struct mm_struct *mm,unsigned long vaddr,size_t npages,bool writable,struct page **pages) { +struct range_rwlock range;int ret; ret =get_user_pages_fast (vaddr, npages, writable, pages);if (ret <0 )return ret; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); mm->pinned_vm += ret; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }
@@ -120,6 +124,7 @@ int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t npvoid hfi1_release_user_pages (struct mm_struct *mm,struct page **p,size_t npages,bool dirty) { +struct range_rwlock range;size_t i;for (i =0 ; i < npages; i++) {
@@ -129,8 +134,9 @@ void hfi1_release_user_pages(struct mm_struct *mm, struct page **p, }if (mm) {/* during close after signal, mm can be NULL */ -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); mm->pinned_vm -= npages; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); } }
drivers/infiniband/hw/mlx4/main.c +4 -2
@@ -1142,6 +1142,7 @@ static void mlx4_ib_disassociate_ucontext(struct ib_ucontext *ibcontext)struct mlx4_ib_ucontext *context =to_mucontext (ibcontext);struct task_struct *owning_process =NULL ;struct mm_struct *owning_mm =NULL ; +struct range_rwlock range; owning_process =get_pid_task (ibcontext->tgid,PIDTYPE_PID );if (!owning_process)
@@ -1173,7 +1174,8 @@ static void mlx4_ib_disassociate_ucontext(struct ib_ucontext *ibcontext)/* need to protect from a race on closing the vma as part of * mlx4_ib_vma_close(). */ -down_read (&owning_mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&owning_mm->mmap_rw_tree, &range);for (i =0 ; i <HW_BAR_COUNT ; i++) { vma = context->hw_bar_info[i].vma;if (!vma)
@@ -1191,7 +1193,7 @@ static void mlx4_ib_disassociate_ucontext(struct ib_ucontext *ibcontext) context->hw_bar_info[i].vma->vm_ops =NULL ; } -up_read (&owning_mm->mmap_sem); +range_read_unlock (&owning_mm->mmap_rw_tree, &range);mmput (owning_mm);put_task_struct (owning_process); }
drivers/infiniband/hw/mlx5/main.c +4 -2
@@ -1449,6 +1449,7 @@ static void mlx5_ib_disassociate_ucontext(struct ib_ucontext *ibcontext)struct mlx5_ib_ucontext *context =to_mucontext (ibcontext);struct task_struct *owning_process =NULL ;struct mm_struct *owning_mm =NULL ; +struct range_rwlock range; owning_process =get_pid_task (ibcontext->tgid,PIDTYPE_PID );if (!owning_process)
@@ -1478,7 +1479,8 @@ static void mlx5_ib_disassociate_ucontext(struct ib_ucontext *ibcontext)/* need to protect from a race on closing the vma as part of * mlx5_ib_vma_close. */ -down_read (&owning_mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&owning_mm->mmap_rw_tree, &range);list_for_each_entry_safe (vma_private, n, &context->vma_private_list, list) { vma = vma_private->vma;
@@ -1492,7 +1494,7 @@ static void mlx5_ib_disassociate_ucontext(struct ib_ucontext *ibcontext)list_del (&vma_private->list);kfree (vma_private); } -up_read (&owning_mm->mmap_sem); +range_read_unlock (&owning_mm->mmap_rw_tree, &range);mmput (owning_mm);put_task_struct (owning_process); }
drivers/infiniband/hw/qib/qib_user_pages.c +11 -5
@@ -133,26 +133,32 @@ dma_addr_t qib_map_page(struct pci_dev *hwdev, struct page *page,int qib_get_user_pages (unsigned long start_page,size_t num_pages,struct page **p) { +struct range_rwlock range;int ret; -down_write (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (¤t->mm->mmap_rw_tree, &range); ret =__qib_get_user_pages (start_page, num_pages, p); -up_write (¤t->mm->mmap_sem); +range_write_unlock (¤t->mm->mmap_rw_tree, &range);return ret; }void qib_release_user_pages (struct page **p,size_t num_pages) { -if (current->mm)/* during close after signal, mm can be NULL */ -down_write (¤t->mm->mmap_sem); +struct range_rwlock range; + +if (current->mm) {/* during close after signal, mm can be NULL */ +range_rwlock_init_full (&range); +range_write_lock (¤t->mm->mmap_rw_tree, &range); + }__qib_release_user_pages (p, num_pages,1 );if (current->mm) { current->mm->pinned_vm -= num_pages; -up_write (¤t->mm->mmap_sem); +range_write_unlock (¤t->mm->mmap_rw_tree, &range); } }
drivers/infiniband/hw/usnic/usnic_uiom.c +13 -7
@@ -55,12 +55,14 @@ static struct workqueue_struct *usnic_uiom_wq;static void usnic_uiom_reg_account (struct work_struct *work) { +struct range_rwlock range;struct usnic_uiom_reg *umem =container_of (work, struct usnic_uiom_reg, work); -down_write (&umem->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&umem->mm->mmap_rw_tree, &range); umem->mm->locked_vm -= umem->diff; -up_write (&umem->mm->mmap_sem); +range_write_unlock (&umem->mm->mmap_rw_tree, &range);mmput (umem->mm);kfree (umem); }
@@ -103,6 +105,7 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable,struct page **page_list;struct scatterlist *sg;struct usnic_uiom_chunk *chunk; +struct range_rwlock range;unsigned long locked;unsigned long lock_limit;unsigned long cur_base;
@@ -125,7 +128,8 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, npages =PAGE_ALIGN (size + (addr & ~PAGE_MASK )) >>PAGE_SHIFT ; -down_write (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (¤t->mm->mmap_rw_tree, &range); locked = npages + current->mm->locked_vm; lock_limit =rlimit (RLIMIT_MEMLOCK ) >>PAGE_SHIFT ;
@@ -188,7 +192,7 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable,else current->mm->locked_vm = locked; -up_write (¤t->mm->mmap_sem); +range_write_unlock (¤t->mm->mmap_rw_tree, &range);free_page ((unsigned long ) page_list);return ret; }
@@ -423,6 +427,7 @@ struct usnic_uiom_reg *usnic_uiom_reg_get(struct usnic_uiom_pd *pd,void usnic_uiom_reg_release (struct usnic_uiom_reg *uiomr,int closing) {struct mm_struct *mm; +struct range_rwlock range;unsigned long diff;__usnic_uiom_reg_release (uiomr->pd, uiomr,1 );
@@ -443,8 +448,9 @@ void usnic_uiom_reg_release(struct usnic_uiom_reg *uiomr, int closing)* up here and not be able to take the mmap_sem. In that case * we defer the vm_locked accounting to the system workqueue. */ +range_rwlock_init_full (&range);if (closing) { -if (!down_write_trylock (&mm->mmap_sem)) { +if (!range_write_trylock (&mm->mmap_rw_tree, &range)) {INIT_WORK (&uiomr->work, usnic_uiom_reg_account); uiomr->mm = mm; uiomr->diff = diff;
@@ -453,10 +459,10 @@ void usnic_uiom_reg_release(struct usnic_uiom_reg *uiomr, int closing)return ; } }else -down_write (&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &range); current->mm->locked_vm -= diff; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm);kfree (uiomr); }
drivers/iommu/amd_iommu_v2.c +5 -3
@@ -518,6 +518,7 @@ static void do_fault(struct work_struct *work)int ret =VM_FAULT_ERROR ;unsigned int flags =0 ;struct mm_struct *mm; +struct range_rwlock range;u64 address; mm = fault->state->mm;
@@ -529,7 +530,8 @@ static void do_fault(struct work_struct *work) flags |=FAULT_FLAG_WRITE ; flags |=FAULT_FLAG_REMOTE ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_extend_vma (mm, address);if (!vma || address < vma->vm_start)/* failed to get a vma in the right range */
@@ -539,9 +541,9 @@ static void do_fault(struct work_struct *work)if (access_error (vma, fault)) goto out; - ret =handle_mm_fault (vma, address, flags); + ret =handle_mm_fault (vma, address, flags,NULL ); out: -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);if (ret &VM_FAULT_ERROR )/* failed to service fault */
drivers/iommu/intel-svm.c +4 -2
@@ -542,6 +542,7 @@ static irqreturn_t prq_event_thread(int irq, void *d)struct vm_area_struct *vma;struct page_req_dsc *req;struct qi_desc resp; +struct range_rwlock range;int ret, result;u64 address;
@@ -582,7 +583,8 @@ static irqreturn_t prq_event_thread(int irq, void *d)/* If the mm is already defunct, don't handle faults. */ if (!mmget_not_zero (svm->mm)) goto bad_req; -down_read (&svm->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&svm->mm->mmap_rw_tree, &range); vma =find_extend_vma (svm->mm, address);if (!vma || address < vma->vm_start) goto invalid;
@@ -597,7 +599,7 @@ static irqreturn_t prq_event_thread(int irq, void *d) result =QI_RESP_SUCCESS ; invalid: -up_read (&svm->mm->mmap_sem); +range_read_unlock (&svm->mm->mmap_rw_tree, &range);mmput (svm->mm); bad_req:/* Accounting for major/minor faults? */
drivers/media/v4l2-core/videobuf-core.c +6 -3
@@ -530,14 +530,17 @@ EXPORT_SYMBOL_GPL(videobuf_querybuf);int videobuf_qbuf (struct videobuf_queue *q,struct v4l2_buffer *b) {struct videobuf_buffer *buf; +struct range_rwlock range;enum v4l2_field field;unsigned long flags =0 ;int retval;MAGIC_CHECK (q->int_ops->magic,MAGIC_QTYPE_OPS ); -if (b->memory ==V4L2_MEMORY_MMAP ) -down_read (¤t->mm->mmap_sem); +if (b->memory ==V4L2_MEMORY_MMAP ) { +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); + }videobuf_queue_lock (q); retval = -EBUSY ;
@@ -624,7 +627,7 @@ int videobuf_qbuf(struct videobuf_queue *q, struct v4l2_buffer *b)videobuf_queue_unlock (q);if (b->memory ==V4L2_MEMORY_MMAP ) -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);return retval; }
drivers/media/v4l2-core/videobuf-dma-contig.c +4 -2
@@ -162,6 +162,7 @@ static int videobuf_dma_contig_user_get(struct videobuf_dma_contig_memory *mem, {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; +struct range_rwlock range;unsigned long prev_pfn, this_pfn;unsigned long pages_done, user_address;unsigned int offset;
@@ -171,7 +172,8 @@ static int videobuf_dma_contig_user_get(struct videobuf_dma_contig_memory *mem, mem->size =PAGE_ALIGN (vb->size + offset); ret = -EINVAL ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, vb->baddr);if (!vma)
@@ -203,7 +205,7 @@ static int videobuf_dma_contig_user_get(struct videobuf_dma_contig_memory *mem, } out_up: -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);return ret; }
drivers/media/v4l2-core/videobuf-dma-sg.c +4 -2
@@ -199,11 +199,13 @@ static int videobuf_dma_init_user_locked(struct videobuf_dmabuf *dma,static int videobuf_dma_init_user (struct videobuf_dmabuf *dma,int direction,unsigned long data,unsigned long size) { +struct range_rwlock range;int ret; -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); ret =videobuf_dma_init_user_locked (dma, direction, data, size); -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);return ret; }
drivers/misc/cxl/fault.c +4 -2
@@ -338,6 +338,7 @@ static void cxl_prefault_vma(struct cxl_context *ctx)struct vm_area_struct *vma;int rc;struct mm_struct *mm; +struct range_rwlock range; mm =get_mem_context (ctx);if (mm ==NULL ) {
@@ -346,7 +347,8 @@ static void cxl_prefault_vma(struct cxl_context *ctx)return ; } -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (vma = mm->mmap; vma; vma = vma->vm_next) {for (ea = vma->vm_start; ea < vma->vm_end; ea =next_segment (ea, slb.vsid)) {
@@ -361,7 +363,7 @@ static void cxl_prefault_vma(struct cxl_context *ctx) last_esid = slb.esid; } } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);mmput (mm); }
drivers/misc/mic/scif/scif_rma.c +11 -6
@@ -275,19 +275,22 @@ static inline int __scif_dec_pinned_vm_lock(struct mm_struct *mm, int nr_pages, bool try_lock) { + struct range_rwlock range; +if (!mm || !nr_pages || !scif_ulimit_check)return 0 ; +range_rwlock_init_full (&range);if (try_lock) { - if (!down_write_trylock(&mm->mmap_sem)) { +if (!range_write_trylock (&mm->mmap_rw_tree, &range)) {dev_err (scif_info.mdev.this_device,"%s %d err\n" , __func__, __LINE__);return -1 ; } }else { - down_write(&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &range); } mm->pinned_vm -= nr_pages; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return 0 ; }
@@ -1333,6 +1336,7 @@ int __scif_pin_pages(void *addr, size_t len, int *out_prot,int prot = *out_prot;int ulimit =0 ;struct mm_struct *mm =NULL ; +struct range_rwlock range;/* Unsupported flags */ if (map_flags & ~(SCIF_MAP_KERNEL |SCIF_MAP_ULIMIT ))
@@ -1386,11 +1390,12 @@ int __scif_pin_pages(void *addr, size_t len, int *out_prot, prot |=SCIF_PROT_WRITE ; retry: mm = current->mm; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range);if (ulimit) { err =__scif_check_inc_pinned_vm (mm, nr_pages);if (err) { -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); pinned_pages->nr_pages =0 ; goto error_unmap; }
@@ -1402,7 +1407,7 @@ int __scif_pin_pages(void *addr, size_t len, int *out_prot, (prot &SCIF_PROT_WRITE ) ?FOLL_WRITE :0 , pinned_pages->pages, NULL, NULL); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (nr_pages != pinned_pages->nr_pages) {if (try_upgrade) {if (ulimit)
drivers/oprofile/buffer_sync.c +9 -5
@@ -90,12 +90,14 @@ munmap_notify(struct notifier_block *self, unsigned long val, void *data)unsigned long addr = (unsigned long )data;struct mm_struct *mm = current->mm;struct vm_area_struct *mpnt; +struct range_rwlock range; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); mpnt =find_vma (mm, addr);if (mpnt && mpnt->vm_file && (mpnt->vm_flags &VM_EXEC )) { -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);/* To avoid latency problems, we only process the current CPU, * hoping that most samples for the task are on this CPU */
@@ -103,7 +105,7 @@ munmap_notify(struct notifier_block *self, unsigned long val, void *data)return 0 ; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return 0 ; }
@@ -255,8 +257,10 @@ lookup_dcookie(struct mm_struct *mm, unsigned long addr, off_t *offset) {unsigned long cookie =NO_COOKIE ;struct vm_area_struct *vma; +struct range_rwlock range; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (vma =find_vma (mm, addr); vma; vma = vma->vm_next) {if (addr < vma->vm_start || addr >= vma->vm_end)
@@ -276,7 +280,7 @@ lookup_dcookie(struct mm_struct *mm, unsigned long addr, off_t *offset)if (!vma) cookie =INVALID_COOKIE ; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return cookie; }
drivers/staging/lustre/lustre/llite/llite_mmap.c +3 -1
@@ -61,9 +61,11 @@ struct vm_area_struct *our_vma(struct mm_struct *mm, unsigned long addr, size_t count) {struct vm_area_struct *vma, *ret =NULL ; +struct range_rwlock range;/* mmap_sem must have been held by caller. */ -LASSERT (!down_write_trylock (&mm->mmap_sem)); +range_rwlock_init_full (&range); +LASSERT (!range_write_trylock (&mm->mmap_rw_tree, &range));for (vma =find_vma (mm, addr); vma && vma->vm_start < (addr + count); vma = vma->vm_next) {
drivers/staging/lustre/lustre/llite/vvp_io.c +4 -2
@@ -376,6 +376,7 @@ static int vvp_mmap_locks(const struct lu_env *env,int result =0 ;struct iov_iter i;struct iovec iov; +struct range_rwlock range;LASSERT (io->ci_type ==CIT_READ || io->ci_type ==CIT_WRITE );
@@ -395,7 +396,8 @@ static int vvp_mmap_locks(const struct lu_env *env, count += addr & (~PAGE_MASK ); addr &=PAGE_MASK ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);while ((vma =our_vma (mm, addr, count)) !=NULL ) {struct inode *inode =file_inode (vma->vm_file);int flags =CEF_MUST ;
@@ -436,7 +438,7 @@ static int vvp_mmap_locks(const struct lu_env *env, count -= vma->vm_end - addr; addr = vma->vm_end; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);if (result <0 )break ; }
drivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.c +5 -2
@@ -467,14 +467,17 @@ create_pagelist(char __user *buf, size_t count, unsigned short type, }/* do not try and release vmalloc pages */ }else { -down_read (&task->mm->mmap_sem); +struct range_rwlock range; + +range_rwlock_init_full (&range); +range_read_lock (&task->mm->mmap_rw_tree, &range); actual_pages =get_user_pages ( (unsigned long )buf & ~(PAGE_SIZE -1 ), num_pages, (type ==PAGELIST_READ ) ?FOLL_WRITE :0 , pages,NULL /*vmas */ ); -up_read (&task->mm->mmap_sem); +range_read_unlock (&task->mm->mmap_rw_tree, &range);if (actual_pages != num_pages) { vchiq_log_info(vchiq_arm_log_level,
drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c +4 -2
@@ -1560,6 +1560,7 @@ dump_phys_mem(void *virt_addr, u32 num_bytes)struct page *page;struct page **pages;u8 *kmapped_virt_ptr; +struct range_rwlock range;/* Align virtAddr and endVirtAddr to 16 byte boundaries. */
@@ -1580,14 +1581,15 @@ dump_phys_mem(void *virt_addr, u32 num_bytes)return ; } -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); rc =get_user_pages ( (unsigned long )virt_addr,/* start */ num_pages,/* len */ 0 ,/* gup_flags */ pages,/* pages (array of page pointers) */ NULL );/* vmas */ -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); prev_idx =-1 ; page =NULL ;
drivers/vfio/vfio_iommu_spapr_tce.c +9 -4
@@ -37,6 +37,7 @@ static void tce_iommu_detach_group(void *iommu_data,static long try_increment_locked_vm (struct mm_struct *mm,long npages) {long ret =0 , locked, lock_limit; +struct range_rwlock range;if (WARN_ON_ONCE (!mm))return -EPERM ;
@@ -44,7 +45,8 @@ static long try_increment_locked_vm(struct mm_struct *mm, long npages)if (!npages)return 0 ; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); locked = mm->locked_vm + npages; lock_limit =rlimit (RLIMIT_MEMLOCK ) >>PAGE_SHIFT ;if (locked > lock_limit && !capable (CAP_IPC_LOCK ))
@@ -58,17 +60,20 @@ static long try_increment_locked_vm(struct mm_struct *mm, long npages)rlimit (RLIMIT_MEMLOCK ), ret ? " - exceeded" : ""); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }static void decrement_locked_vm (struct mm_struct *mm,long npages) { +struct range_rwlock range; +if (!mm || !npages)return ; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range);if (WARN_ON_ONCE (npages > mm->locked_vm)) npages = mm->locked_vm; mm->locked_vm -= npages;
@@ -76,7 +81,7 @@ static void decrement_locked_vm(struct mm_struct *mm, long npages) npages <<PAGE_SHIFT , mm->locked_vm <<PAGE_SHIFT , rlimit(RLIMIT_MEMLOCK)); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); }/*
drivers/vfio/vfio_iommu_type1.c +15 -9
@@ -257,11 +257,13 @@ static void vfio_lock_acct_bg(struct work_struct *work) { struct vwork *vwork = container_of(work, struct vwork, work);struct mm_struct *mm; +struct range_rwlock range; mm = vwork->mm; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); mm->locked_vm += vwork->npage; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm);kfree (vwork); }
@@ -270,6 +272,7 @@ static void vfio_lock_acct(struct task_struct *task, long npage) {struct vwork *vwork;struct mm_struct *mm; +struct range_rwlock range;bool is_current;if (!npage)
@@ -281,9 +284,10 @@ static void vfio_lock_acct(struct task_struct *task, long npage)if (!mm)return ;/* process exited */ -if (down_write_trylock (&mm->mmap_sem)) { +range_rwlock_init_full (&range); +if (range_write_trylock (&mm->mmap_rw_tree, &range)) { mm->locked_vm += npage; -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (!is_current)mmput (mm);return ;
@@ -361,8 +365,10 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr, {struct page *page[1 ];struct vm_area_struct *vma; +struct range_rwlock range;int ret; +range_rwlock_init_full (&range);if (mm == current->mm) { ret =get_user_pages_fast (vaddr,1 , !!(prot &IOMMU_WRITE ), page);
@@ -372,10 +378,10 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,if (prot &IOMMU_WRITE ) flags |=FOLL_WRITE ; -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); ret =get_user_pages_remote (NULL , mm, vaddr,1 , flags, page, -NULL ,NULL ); -up_read (&mm->mmap_sem); +NULL ,NULL ,NULL ); +range_read_unlock (&mm->mmap_rw_tree, &range); }if (ret ==1 ) {
@@ -383,7 +389,7 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,return 0 ; } -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_vma_intersection (mm, vaddr, vaddr +1 );
@@ -393,7 +399,7 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr, ret =0 ; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return ret; }
drivers/virt/fsl_hypervisor.c +4 -2
@@ -146,6 +146,7 @@ static long ioctl_stop(struct fsl_hv_ioctl_stop __user *p)*/ static long ioctl_memcpy (struct fsl_hv_ioctl_memcpy __user *p) { +struct range_rwlock range;struct fsl_hv_ioctl_memcpy param;struct page **pages =NULL ;
@@ -243,11 +244,12 @@ static long ioctl_memcpy(struct fsl_hv_ioctl_memcpy __user *p) sg_list =PTR_ALIGN (sg_list_unaligned,sizeof (struct fh_sg_list ));/* Get the physical addresses of the source buffer */ -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); num_pinned =get_user_pages (param.local_vaddr - lb_offset, num_pages, (param.source ==-1 ) ?0 :FOLL_WRITE , pages,NULL ); -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (num_pinned != num_pages) {/* get_user_pages() failed */
drivers/xen/gntdev.c +4 -2
@@ -657,13 +657,15 @@ static long gntdev_ioctl_get_offset_for_vaddr(struct gntdev_priv *priv,struct ioctl_gntdev_get_offset_for_vaddr op;struct vm_area_struct *vma;struct grant_map *map; +struct range_rwlock range;int rv = -EINVAL ;if (copy_from_user (&op, u,sizeof (op)) !=0 )return -EFAULT ;pr_debug ("priv %p, offset for vaddr %lx\n" , priv, (unsigned long )op.vaddr); -down_read (¤t->mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (¤t->mm->mmap_rw_tree, &range); vma =find_vma (current->mm, op.vaddr);if (!vma || vma->vm_ops != &gntdev_vmops) goto out_unlock;
@@ -677,7 +679,7 @@ static long gntdev_ioctl_get_offset_for_vaddr(struct gntdev_priv *priv, rv =0 ; out_unlock: -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range);if (rv ==0 &©_to_user (u, &op,sizeof (op)) !=0 )return -EFAULT ;
drivers/xen/privcmd.c +9 -5
@@ -260,6 +260,7 @@ static long privcmd_ioctl_mmap(struct file *file, void __user *udata)int rc;LIST_HEAD (pagelist);struct mmap_gfn_state state; +struct range_rwlock range;/* We only support privcmd_ioctl_mmap_batch for auto translated. */ if (xen_feature (XENFEAT_auto_translated_physmap))
@@ -279,7 +280,8 @@ static long privcmd_ioctl_mmap(struct file *file, void __user *udata)if (rc ||list_empty (&pagelist)) goto out; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); { struct page *page = list_first_entry(&pagelist,
@@ -304,7 +306,7 @@ static long privcmd_ioctl_mmap(struct file *file, void __user *udata) out_up: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); out:free_page_list (&pagelist);
@@ -454,6 +456,7 @@ static long privcmd_ioctl_mmap_batch(unsigned long nr_pages;LIST_HEAD (pagelist);struct mmap_batch_state state; +struct range_rwlock range;switch (version) {case 1 :
@@ -500,7 +503,8 @@ static long privcmd_ioctl_mmap_batch( } } -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, m.addr); if (!vma ||
@@ -556,7 +560,7 @@ static long privcmd_ioctl_mmap_batch(BUG_ON (traverse_pages_block (m.num,sizeof (xen_pfn_t), &pagelist, mmap_batch_fn, &state)); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (state.global_error) {/* Write back errors in second pass. */
@@ -577,7 +581,7 @@ static long privcmd_ioctl_mmap_batch(return ret; out_unlock: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); goto out; }
fs/aio.c +5 -2
@@ -450,6 +450,9 @@ static int aio_setup_ring(struct kioctx *ctx)int nr_pages;int i;struct file *file; +struct range_rwlock range; + +range_rwlock_init_full (&range);/* Compensate for the ring buffer's head/tail overlap entry */ nr_events +=2 ;/* 1 is required, 2 for good luck */
@@ -504,7 +507,7 @@ static int aio_setup_ring(struct kioctx *ctx) ctx->mmap_size = nr_pages *PAGE_SIZE ;pr_debug ("attempting mmap of %lu bytes\n" , ctx->mmap_size); -if (down_write_killable (&mm->mmap_sem)) { +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range)) { ctx->mmap_size =0 ;aio_free_ring (ctx);return -EINTR ;
@@ -513,7 +516,7 @@ static int aio_setup_ring(struct kioctx *ctx) ctx->mmap_base =do_mmap_pgoff (ctx->aio_ring_file,0 , ctx->mmap_size,PROT_READ |PROT_WRITE ,MAP_SHARED ,0 , &unused,NULL ); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (IS_ERR ((void *)ctx->mmap_base)) { ctx->mmap_size =0 ;aio_free_ring (ctx);
fs/coredump.c +4 -2
@@ -411,17 +411,19 @@ static int coredump_wait(int exit_code, struct core_state *core_state)struct task_struct *tsk = current;struct mm_struct *mm = tsk->mm;int core_waiters = -EBUSY ; +struct range_rwlock range; +range_rwlock_init_full (&range);init_completion (&core_state->startup); core_state->dumper.task = tsk; core_state->dumper.next =NULL ; -if (down_write_killable (&mm->mmap_sem)) +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return -EINTR ;if (!mm->core_state) core_waiters =zap_threads (tsk, mm, core_state, exit_code); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);if (core_waiters >0 ) {struct core_thread *ptr;
fs/exec.c +16 -8
@@ -268,12 +268,14 @@ static int __bprm_mm_init(struct linux_binprm *bprm)int err;struct vm_area_struct *vma =NULL ;struct mm_struct *mm = bprm->mm; +struct range_rwlock range; bprm->vma = vma =kmem_cache_zalloc (vm_area_cachep,GFP_KERNEL );if (!vma)return -ENOMEM ; -if (down_write_killable (&mm->mmap_sem)) { +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range)) { err = -EINTR ; goto err_free; }
@@ -298,11 +300,11 @@ static int __bprm_mm_init(struct linux_binprm *bprm) mm->stack_vm = mm->total_vm =1 ;arch_bprm_mm_init (mm, vma); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); bprm->p = vma->vm_end -sizeof (void *);return 0 ; err: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); err_free: bprm->vma =NULL ;kmem_cache_free (vm_area_cachep, vma);
@@ -673,6 +675,7 @@ int setup_arg_pages(struct linux_binprm *bprm,unsigned long stack_size;unsigned long stack_expand;unsigned long rlim_stack; +struct range_rwlock range;#ifdef CONFIG_STACK_GROWSUP /* Limit stack size */
@@ -710,7 +713,8 @@ int setup_arg_pages(struct linux_binprm *bprm, bprm->loader -= stack_shift; bprm->exec -= stack_shift; -if (down_write_killable (&mm->mmap_sem)) +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return -EINTR ; vm_flags =VM_STACK_FLAGS ;
@@ -767,7 +771,7 @@ int setup_arg_pages(struct linux_binprm *bprm, ret = -EFAULT ; out_unlock: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }EXPORT_SYMBOL (setup_arg_pages);
@@ -1001,6 +1005,9 @@ static int exec_mmap(struct mm_struct *mm) {struct task_struct *tsk;struct mm_struct *old_mm, *active_mm; +struct range_rwlock range; + +range_rwlock_init_full (&range);/* Notify parent that we're no longer interested in the old VM */ tsk = current;
@@ -1015,9 +1022,10 @@ static int exec_mmap(struct mm_struct *mm)* through with the exec. We must hold mmap_sem around * checking core_state and changing tsk->mm. */ -down_read (&old_mm->mmap_sem); + +range_read_lock (&old_mm->mmap_rw_tree, &range);if (unlikely (old_mm->core_state)) { -up_read (&old_mm->mmap_sem); +range_read_unlock (&old_mm->mmap_rw_tree, &range);return -EINTR ; } }
@@ -1030,7 +1038,7 @@ static int exec_mmap(struct mm_struct *mm)vmacache_flush (tsk);task_unlock (tsk);if (old_mm) { -up_read (&old_mm->mmap_sem); +range_read_unlock (&old_mm->mmap_rw_tree, &range);BUG_ON (active_mm != old_mm);setmax_mm_hiwater_rss (&tsk->signal->maxrss, old_mm);mm_update_next_owner (old_mm);
fs/proc/base.c +25 -13
@@ -216,7 +216,9 @@ static ssize_t proc_pid_cmdline_read(struct file *file, char __user *buf,unsigned long p;char c;ssize_t rv; +struct range_rwlock range; +range_rwlock_init_full (&range);BUG_ON (*pos <0 ); tsk =get_proc_task (file_inode (file));
@@ -238,12 +240,12 @@ static ssize_t proc_pid_cmdline_read(struct file *file, char __user *buf, goto out_mmput; } -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); arg_start = mm->arg_start; arg_end = mm->arg_end; env_start = mm->env_start; env_end = mm->env_end; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);BUG_ON (arg_start > arg_end);BUG_ON (env_start > env_end);
@@ -916,7 +918,9 @@ static ssize_t environ_read(struct file *file, char __user *buf,int ret =0 ;struct mm_struct *mm = file->private_data;unsigned long env_start, env_end; +struct range_rwlock range; +range_rwlock_init_full (&range);/* Ensure the process spawned far enough to have an environment. */ if (!mm || !mm->env_end)return 0 ;
@@ -929,10 +933,10 @@ static ssize_t environ_read(struct file *file, char __user *buf,if (!mmget_not_zero (mm)) goto free; -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); env_start = mm->env_start; env_end = mm->env_end; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);while (count >0 ) {size_t this_len, max_len;
@@ -1880,6 +1884,7 @@ static int map_files_d_revalidate(struct dentry *dentry, unsigned int flags)struct task_struct *task;struct inode *inode;int status =0 ; +struct range_rwlock range;if (flags &LOOKUP_RCU )return -ECHILD ;
@@ -1894,9 +1899,10 @@ static int map_files_d_revalidate(struct dentry *dentry, unsigned int flags) goto out;if (!dname_to_vma_addr (dentry, &vm_start, &vm_end)) { -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); exact_vma_exists = !!find_exact_vma (mm, vm_start, vm_end); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); }mmput (mm);
@@ -1927,6 +1933,7 @@ static int map_files_get_link(struct dentry *dentry, struct path *path)struct task_struct *task;struct mm_struct *mm;int rc; +struct range_rwlock range; rc = -ENOENT ; task =get_proc_task (d_inode (dentry));
@@ -1943,14 +1950,15 @@ static int map_files_get_link(struct dentry *dentry, struct path *path) goto out_mmput; rc = -ENOENT ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_exact_vma (mm, vm_start, vm_end);if (vma && vma->vm_file) { *path = vma->vm_file->f_path;path_get (path); rc =0 ; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); out_mmput:mmput (mm);
@@ -2023,6 +2031,7 @@ static struct dentry *proc_map_files_lookup(struct inode *dir,struct task_struct *task;int result;struct mm_struct *mm; +struct range_rwlock range; result = -ENOENT ; task =get_proc_task (dir);
@@ -2041,7 +2050,8 @@ static struct dentry *proc_map_files_lookup(struct inode *dir,if (!mm) goto out_put_task; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_exact_vma (mm, vm_start, vm_end);if (!vma) goto out_no_vma;
@@ -2051,7 +2061,7 @@ static struct dentry *proc_map_files_lookup(struct inode *dir, (void *)(unsigned long)vma->vm_file->f_mode); out_no_vma: -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);mmput (mm); out_put_task:put_task_struct (task);
@@ -2076,7 +2086,9 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx)struct map_files_info info;struct map_files_info *p;int ret; +struct range_rwlock range; +range_rwlock_init_full (&range); ret = -ENOENT ; task =get_proc_task (file_inode (file));if (!task)
@@ -2093,7 +2105,7 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx) mm =get_task_mm (task);if (!mm) goto out_put_task; -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); nr_files =0 ;
@@ -2120,7 +2132,7 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx) ret = -ENOMEM ;if (fa)flex_array_free (fa); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);mmput (mm); goto out_put_task; }
@@ -2139,7 +2151,7 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx)BUG (); } } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);for (i =0 ; i < nr_files; i++) { p =flex_array_get (fa, i);
fs/proc/internal.h +1 -0
@@ -279,6 +279,7 @@ struct proc_maps_private {#ifdef CONFIG_NUMA struct mempolicy *task_mempolicy;#endif +struct range_rwlock range; };struct mm_struct *proc_mem_open (struct inode *inode,unsigned int mode);
fs/proc/task_mmu.c +19 -11
@@ -133,7 +133,7 @@ static void vma_stop(struct proc_maps_private *priv)struct mm_struct *mm = priv->mm;release_task_mempolicy (priv); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &priv->range);mmput (mm); }
@@ -171,7 +171,8 @@ static void *m_start(struct seq_file *m, loff_t *ppos)if (!mm || !mmget_not_zero (mm))return NULL ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&priv->range); +range_read_lock (&mm->mmap_rw_tree, &priv->range);hold_task_mempolicy (priv); priv->tail_vma =get_gate_vma (mm);
@@ -1009,7 +1010,9 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf,enum clear_refs_types type;int itype;int rv; +struct range_rwlock range; +range_rwlock_init_full (&range);memset (buffer,0 ,sizeof (buffer));if (count >sizeof (buffer) -1 ) count =sizeof (buffer) -1 ;
@@ -1038,7 +1041,8 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, };if (type ==CLEAR_REFS_MM_HIWATER_RSS ) { -if (down_write_killable (&mm->mmap_sem)) { +if (range_write_lock_interruptible (&mm->mmap_rw_tree, + &range)) { count = -EINTR ; goto out_mm; }
@@ -1048,17 +1052,18 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf,* resident set size to this mm's current rss value. */ reset_mm_hiwater_rss (mm); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); goto out_mm; } -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range);if (type ==CLEAR_REFS_SOFT_DIRTY ) {for (vma = mm->mmap; vma; vma = vma->vm_next) {if (!(vma->vm_flags &VM_SOFTDIRTY ))continue ; -up_read (&mm->mmap_sem); -if (down_write_killable (&mm->mmap_sem)) { +range_read_unlock (&mm->mmap_rw_tree, &range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, + &range)) { count = -EINTR ; goto out_mm; }
@@ -1066,7 +1071,8 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, vma->vm_flags &= ~VM_SOFTDIRTY ;vma_set_page_prot (vma); } -downgrade_write (&mm->mmap_sem); +range_downgrade_write (&mm->mmap_rw_tree, + &range);break ; }mmu_notifier_invalidate_range_start (mm,0 ,-1 );
@@ -1075,7 +1081,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf,if (type ==CLEAR_REFS_SOFT_DIRTY )mmu_notifier_invalidate_range_end (mm,0 ,-1 );flush_tlb_mm (mm); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); out_mm:mmput (mm); }
@@ -1359,6 +1365,7 @@ static ssize_t pagemap_read(struct file *file, char __user *buf,unsigned long start_vaddr;unsigned long end_vaddr;int ret =0 , copied =0 ; +struct range_rwlock range;if (!mm || !mmget_not_zero (mm)) goto out;
@@ -1414,9 +1421,10 @@ static ssize_t pagemap_read(struct file *file, char __user *buf,/* overflow ? */ if (end < start_vaddr || end > end_vaddr) end = end_vaddr; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); ret =walk_page_range (start_vaddr, end, &pagemap_walk); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); start_vaddr = end; len =min (count,PM_ENTRY_BYTES * pm.pos);
fs/proc/task_nommu.c +17 -10
@@ -23,8 +23,10 @@ void task_mem(struct seq_file *m, struct mm_struct *mm)struct vm_region *region;struct rb_node *p;unsigned long bytes =0 , sbytes =0 , slack =0 , size; - -down_read (&mm->mmap_sem); +struct range_rwlock range; + +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (p =rb_first (&mm->mm_rb); p; p =rb_next (p)) { vma = rb_entry(p, struct vm_area_struct, vm_rb);
@@ -76,7 +78,7 @@ void task_mem(struct seq_file *m, struct mm_struct *mm)"Shared:\t%8lu bytes\n" , bytes, slack, sbytes); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); }unsigned long task_vsize (struct mm_struct *mm)
@@ -84,13 +86,15 @@ unsigned long task_vsize(struct mm_struct *mm)struct vm_area_struct *vma;struct rb_node *p;unsigned long vsize =0 ; +struct range_rwlock range; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (p =rb_first (&mm->mm_rb); p; p =rb_next (p)) { vma = rb_entry(p, struct vm_area_struct, vm_rb); vsize += vma->vm_end - vma->vm_start; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return vsize; }
@@ -102,8 +106,10 @@ unsigned long task_statm(struct mm_struct *mm,struct vm_region *region;struct rb_node *p;unsigned long size =kobjsize (mm); +struct range_rwlock range; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (p =rb_first (&mm->mm_rb); p; p =rb_next (p)) { vma = rb_entry(p, struct vm_area_struct, vm_rb); size +=kobjsize (vma);
@@ -118,7 +124,7 @@ unsigned long task_statm(struct mm_struct *mm, >> PAGE_SHIFT; *data = (PAGE_ALIGN (mm->start_stack) - (mm->start_data &PAGE_MASK )) >>PAGE_SHIFT ; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); size >>=PAGE_SHIFT ; size += *text + *data; *resident = size;
@@ -224,13 +230,14 @@ static void *m_start(struct seq_file *m, loff_t *pos)if (!mm || !mmget_not_zero (mm))return NULL ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&priv->range); +range_read_lock (&mm->mmap_rw_tree, &priv->range);/* start from the Nth VMA */ for (p =rb_first (&mm->mm_rb); p; p =rb_next (p))if (n-- ==0 )return p; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &priv->range);mmput (mm);return NULL ; }
@@ -240,7 +247,7 @@ static void m_stop(struct seq_file *m, void *_vml)struct proc_maps_private *priv = m->private;if (!IS_ERR_OR_NULL (_vml)) { -up_read (&priv->mm->mmap_sem); +range_read_unlock (&priv->mm->mmap_rw_tree, &priv->range);mmput (priv->mm); }if (priv->task) {
fs/userfaultfd.c +15 -9
@@ -431,7 +431,7 @@ int handle_userfault(struct vm_fault *vmf, unsigned long reason)else must_wait =userfaultfd_huge_must_wait (ctx, vmf->address, vmf->flags, reason); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, vmf->lockrange); if (likely(must_wait && !ACCESS_ONCE(ctx->released) && (return_to_userland ? !signal_pending(current) :
@@ -485,7 +485,7 @@ int handle_userfault(struct vm_fault *vmf, unsigned long reason)* and there's no need to retake the mmap_sem * in such case. */ -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, vmf->lockrange); ret =VM_FAULT_NOPAGE ; } }
@@ -704,7 +704,7 @@ bool userfaultfd_remove(struct vm_area_struct *vma,return true;userfaultfd_ctx_get (ctx); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, range);msg_init (&ewq.msg);
@@ -783,7 +783,9 @@ static int userfaultfd_release(struct inode *inode, struct file *file)/* len == 0 means wake all */ struct userfaultfd_wake_range range = { .len =0 , };unsigned long new_flags; +struct range_rwlock lockrange; +range_rwlock_init_full (&lockrange);ACCESS_ONCE (ctx->released) = true;if (!mmget_not_zero (mm))
@@ -797,7 +799,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file)* it's critical that released is set to true (above), before * taking the mmap_sem for writing. */ -down_write (&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &lockrange); prev =NULL ;for (vma = mm->mmap; vma; vma = vma->vm_next) {cond_resched ();
@@ -820,7 +822,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ; } -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &lockrange);mmput (mm); wakeup:/*
@@ -1180,6 +1182,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,bool found;bool non_anon_pages;unsigned long start, end, vma_end; +struct range_rwlock range; user_uffdio_register = (struct uffdio_register __user *) arg;
@@ -1219,7 +1222,8 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,if (!mmget_not_zero (mm)) goto out; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); vma =find_vma_prev (mm, start, &prev);if (!vma) goto out_unlock;
@@ -1347,7 +1351,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, vma = vma->vm_next; }while (vma && vma->vm_start < end); out_unlock: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm);if (!ret) {/*
@@ -1375,6 +1379,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,bool found;unsigned long start, end, vma_end;const void __user *buf = (void __user *)arg; +struct range_rwlock range; ret = -EFAULT ;if (copy_from_user (&uffdio_unregister, buf,sizeof (uffdio_unregister)))
@@ -1392,7 +1397,8 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,if (!mmget_not_zero (mm)) goto out; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); vma =find_vma_prev (mm, start, &prev);if (!vma) goto out_unlock;
@@ -1505,7 +1511,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, vma = vma->vm_next; }while (vma && vma->vm_start < end); out_unlock: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);mmput (mm); out:return ret;
include/linux/mm_types.h +2 -1
@@ -8,6 +8,7 @@#include <linux/spinlock.h> #include <linux/rbtree.h> #include <linux/rwsem.h> +#include <linux/range_rwlock.h> #include <linux/completion.h> #include <linux/cpumask.h> #include <linux/uprobes.h>
@@ -398,7 +399,7 @@ struct mm_struct {int map_count;/* number of VMAs */ spinlock_t page_table_lock;/* Protects page tables and some counters */ -struct rw_semaphore mmap_sem; +struct range_rwlock_tree mmap_rw_tree;/* formerly mmap_sem */ struct list_head mmlist;/* List of maybe swapped mm's. These are globally strung * together off init_mm.mmlist, and are protected
ipc/shm.c +9 -4
@@ -1107,6 +1107,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg,struct path path;fmode_t f_mode;unsigned long populate =0 ; +struct range_rwlock range; err = -EINVAL ;if (shmid <0 )
@@ -1213,7 +1214,9 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg,if (err) goto out_fput; -if (down_write_killable (¤t->mm->mmap_sem)) { +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (¤t->mm->mmap_rw_tree, + &range)) { err = -EINTR ; goto out_fput; }
@@ -1233,7 +1236,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg,if (IS_ERR_VALUE (addr)) err = (long )addr; invalid: -up_write (¤t->mm->mmap_sem); +range_write_unlock (¤t->mm->mmap_rw_tree, &range);if (populate)mm_populate (addr, populate);
@@ -1284,11 +1287,13 @@ SYSCALL_DEFINE1(shmdt, char __user *, shmaddr)struct file *file;struct vm_area_struct *next;#endif +struct range_rwlock range;if (addr & ~PAGE_MASK )return retval; -if (down_write_killable (&mm->mmap_sem)) +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return -EINTR ;/*
@@ -1376,7 +1381,7 @@ SYSCALL_DEFINE1(shmdt, char __user *, shmaddr)#endif -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return retval; }
kernel/acct.c +4 -2
@@ -534,17 +534,19 @@ void acct_collect(long exitcode, int group_dead)struct pacct_struct *pacct = ¤t->signal->pacct;u64 utime, stime;unsigned long vsize =0 ; +struct range_rwlock range; +range_rwlock_init_full (&range);if (group_dead && current->mm) {struct vm_area_struct *vma; -down_read (¤t->mm->mmap_sem); +range_read_lock (¤t->mm->mmap_rw_tree, &range); vma = current->mm->mmap;while (vma) { vsize += vma->vm_end - vma->vm_start; vma = vma->vm_next; } -up_read (¤t->mm->mmap_sem); +range_read_unlock (¤t->mm->mmap_rw_tree, &range); }spin_lock_irq (¤t->sighand->siglock);
kernel/events/core.c +4 -2
@@ -8091,7 +8091,9 @@ static void perf_event_addr_filters_apply(struct perf_event *event)struct mm_struct *mm =NULL ;unsigned int count =0 ;unsigned long flags; +struct range_rwlock range; +range_rwlock_init_full (&range);/* * We may observe TASK_TOMBSTONE, which means that the event tear-down * will stop on the parent's child_mutex that our caller is also holding
@@ -8106,7 +8108,7 @@ static void perf_event_addr_filters_apply(struct perf_event *event)if (!mm) goto restart; -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range);raw_spin_lock_irqsave (&ifh->lock, flags);list_for_each_entry (filter, &ifh->list, entry) {
@@ -8126,7 +8128,7 @@ static void perf_event_addr_filters_apply(struct perf_event *event) event->addr_filters_gen++;raw_spin_unlock_irqrestore (&ifh->lock, flags); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);mmput (mm);
kernel/events/uprobes.c +16 -8
@@ -806,11 +806,13 @@ register_for_each_vma(struct uprobe *uprobe, struct uprobe_consumer *new)while (info) {struct mm_struct *mm = info->mm;struct vm_area_struct *vma; +struct range_rwlock range;if (err && is_register) goto free; -down_write (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_write_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, info->vaddr);if (!vma || !valid_vma (vma, is_register) ||file_inode (vma->vm_file) != uprobe->inode)
@@ -832,7 +834,7 @@ register_for_each_vma(struct uprobe *uprobe, struct uprobe_consumer *new) } unlock: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); free:mmput (mm); info =free_map_info (info);
@@ -971,9 +973,11 @@ EXPORT_SYMBOL_GPL(uprobe_unregister);static int unapply_uprobe (struct uprobe *uprobe,struct mm_struct *mm) {struct vm_area_struct *vma; +struct range_rwlock range;int err =0 ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range);for (vma = mm->mmap; vma; vma = vma->vm_next) {unsigned long vaddr;loff_t offset;
@@ -990,7 +994,7 @@ static int unapply_uprobe(struct uprobe *uprobe, struct mm_struct *mm) vaddr =offset_to_vaddr (vma, uprobe->offset); err |=remove_breakpoint (uprobe, mm, vaddr); } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return err; }
@@ -1138,9 +1142,11 @@ void uprobe_munmap(struct vm_area_struct *vma, unsigned long start, unsigned lonstatic int xol_add_vma (struct mm_struct *mm,struct xol_area *area) {struct vm_area_struct *vma; +struct range_rwlock range;int ret; -if (down_write_killable (&mm->mmap_sem)) +range_rwlock_init_full (&range); +if (range_write_lock_interruptible (&mm->mmap_rw_tree, &range))return -EINTR ;if (mm->uprobes_state.xol_area) {
@@ -1170,7 +1176,7 @@ static int xol_add_vma(struct mm_struct *mm, struct xol_area *area)smp_wmb ();/* pairs with get_xol_area() */ mm->uprobes_state.xol_area = area; fail: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return ret; }
@@ -1736,8 +1742,10 @@ static struct uprobe *find_active_uprobe(unsigned long bp_vaddr, int *is_swbp)struct mm_struct *mm = current->mm;struct uprobe *uprobe =NULL ;struct vm_area_struct *vma; +struct range_rwlock range; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, bp_vaddr);if (vma && vma->vm_start <= bp_vaddr) {if (valid_vma (vma, false)) {
@@ -1755,7 +1763,7 @@ static struct uprobe *find_active_uprobe(unsigned long bp_vaddr, int *is_swbp)if (!uprobe &&test_and_clear_bit (MMF_RECALC_UPROBES , &mm->flags))mmf_recalc_uprobes (mm); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return uprobe; }
kernel/exit.c +6 -4
@@ -508,6 +508,7 @@ static void exit_mm(void) {struct mm_struct *mm = current->mm;struct core_state *core_state; +struct range_rwlock range;mm_release (current, mm);if (!mm)
@@ -520,12 +521,13 @@ static void exit_mm(void)* will increment ->nr_threads for each thread in the * group with ->mm != NULL. */ -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); core_state = mm->core_state;if (core_state) {struct core_thread self; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); self.task = current; self.next =xchg (&core_state->dumper.next, &self);
@@ -543,14 +545,14 @@ static void exit_mm(void)freezable_schedule (); }__set_current_state (TASK_RUNNING ); -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range); }mmgrab (mm);BUG_ON (mm != current->active_mm);/* more a memory barrier than a real lock */ task_lock (current); current->mm =NULL ; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);enter_lazy_tlb (mm, current);task_unlock (current);mm_update_next_owner (mm);
kernel/fork.c +14 -7
@@ -573,9 +573,13 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,int retval;unsigned long charge;LIST_HEAD (uf); +struct range_rwlock range, oldrange; + +range_rwlock_init_full (&range); +range_rwlock_init_full (&oldrange);uprobe_start_dup_mmap (); -if (down_write_killable (&oldmm->mmap_sem)) { +if (range_write_lock_interruptible (&oldmm->mmap_rw_tree, &oldrange)) { retval = -EINTR ; goto fail_uprobe_end; }
@@ -584,7 +588,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,/* * Not linked in yet - no deadlock potential: */ -down_write_nested (&mm->mmap_sem,SINGLE_DEPTH_NESTING ); +range_write_lock (&mm->mmap_rw_tree, &range);/* No ordering required: file already has been exposed. */ RCU_INIT_POINTER (mm->exe_file,get_mm_exe_file (oldmm));
@@ -688,9 +692,9 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,arch_dup_mmap (oldmm, mm); retval =0 ; out: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);flush_tlb_mm (oldmm); -up_write (&oldmm->mmap_sem); +range_write_unlock (&oldmm->mmap_rw_tree, &oldrange);dup_userfaultfd_complete (&uf); fail_uprobe_end:uprobe_end_dup_mmap ();
@@ -720,9 +724,12 @@ static inline void mm_free_pgd(struct mm_struct *mm)#else static int dup_mmap (struct mm_struct *mm,struct mm_struct *oldmm) { -down_write (&oldmm->mmap_sem); +struct range_rwlock oldrange; + +range_rwlock_init_full (&oldrange); +range_write_lock (&oldmm->mmap_rw_tree, &oldrange);RCU_INIT_POINTER (mm->exe_file,get_mm_exe_file (oldmm)); -up_write (&oldmm->mmap_sem); +range_write_unlock (&oldmm->mmap_rw_tree, &oldrange);return 0 ; }#define mm_alloc_pgd (mm) (0)
@@ -771,7 +778,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm->vmacache_seqnum =0 ;atomic_set (&mm->mm_users,1 );atomic_set (&mm->mm_count,1 ); -init_rwsem (&mm->mmap_sem); +range_rwlock_tree_init (&mm->mmap_rw_tree);INIT_LIST_HEAD (&mm->mmlist); mm->core_state =NULL ;atomic_long_set (&mm->nr_ptes,0 );
kernel/futex.c +5 -3
@@ -723,12 +723,14 @@ static inline void put_futex_key(union futex_key *key)static int fault_in_user_writeable (u32 __user *uaddr) {struct mm_struct *mm = current->mm; +struct range_rwlock range;int ret; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range);/* XXX finer grain required here */ +range_read_lock (&mm->mmap_rw_tree, &range); ret =fixup_user_fault (current, mm, (unsigned long )uaddr, -FAULT_FLAG_WRITE ,NULL ,NULL ); -up_read (&mm->mmap_sem); +FAULT_FLAG_WRITE ,NULL , &range); +range_read_unlock (&mm->mmap_rw_tree, &range);return ret <0 ? ret :0 ; }
kernel/sched/fair.c +4 -3
@@ -2425,6 +2425,7 @@ void task_numa_work(struct callback_head *work)unsigned long start, end;unsigned long nr_pte_updates =0 ;long pages, virtpages; +struct range_rwlock range; SCHED_WARN_ON(p != container_of(work, struct task_struct, numa_work));
@@ -2474,8 +2475,8 @@ void task_numa_work(struct callback_head *work)if (!pages)return ; - -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, start);if (!vma) {reset_ptenuma_scan (p);
@@ -2542,7 +2543,7 @@ void task_numa_work(struct callback_head *work) mm->numa_scan_offset = start; elsereset_ptenuma_scan (p ); -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);/* * Make sure tasks use at least 32x as much time to run other code
kernel/sys.c +22 -9
@@ -1663,6 +1663,9 @@ static int prctl_set_mm_exe_file(struct mm_struct *mm, unsigned int fd)struct file *old_exe, *exe_file;struct inode *inode;int err; +struct range_rwlock range; + +range_rwlock_init_full (&range); exe =fdget (fd);if (!exe.file)
@@ -1691,7 +1694,7 @@ static int prctl_set_mm_exe_file(struct mm_struct *mm, unsigned int fd)if (exe_file) {struct vm_area_struct *vma; -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, &range);for (vma = mm->mmap; vma; vma = vma->vm_next) {if (!vma->vm_file)continue ;
@@ -1700,7 +1703,7 @@ static int prctl_set_mm_exe_file(struct mm_struct *mm, unsigned int fd) goto exit_err; } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);fput (exe_file); }
@@ -1714,7 +1717,7 @@ static int prctl_set_mm_exe_file(struct mm_struct *mm, unsigned int fd)fdput (exe);return err; exit_err: -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);fput (exe_file); goto exit; }
@@ -1821,6 +1824,9 @@ static int prctl_set_mm_map(int opt, const void __user *addr, unsigned long dataunsigned long user_auxv[AT_VECTOR_SIZE ];struct mm_struct *mm = current->mm;int error; +struct range_rwlock range; + +range_rwlock_init_full (&range);BUILD_BUG_ON (sizeof (user_auxv) !=sizeof (mm->saved_auxv));BUILD_BUG_ON (sizeof (struct prctl_mm_map ) >256 );
@@ -1857,7 +1863,7 @@ static int prctl_set_mm_map(int opt, const void __user *addr, unsigned long datareturn error; } -down_write (&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &range);/* * We don't validate if these members are pointing to
@@ -1894,7 +1900,7 @@ static int prctl_set_mm_map(int opt, const void __user *addr, unsigned long dataif (prctl_map.auxv_size)memcpy (mm->saved_auxv, user_auxv,sizeof (user_auxv)); -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range); return 0; }#endif /* CONFIG_CHECKPOINT_RESTORE */
@@ -1936,6 +1942,9 @@ static int prctl_set_mm(int opt, unsigned long addr,struct prctl_mm_map prctl_map;struct vm_area_struct *vma;int error; +struct range_rwlock range; + +range_rwlock_init_full (&range); if (arg5 || (arg4 && (opt != PR_SET_MM_AUXV && opt != PR_SET_MM_MAP &&
@@ -1961,7 +1970,7 @@ static int prctl_set_mm(int opt, unsigned long addr, error = -EINVAL ; -down_write (&mm->mmap_sem); +range_write_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, addr); prctl_map.start_code = mm->start_code;
@@ -2054,7 +2063,7 @@ static int prctl_set_mm(int opt, unsigned long addr, error =0 ; out: -up_write (&mm->mmap_sem); +range_write_unlock (&mm->mmap_rw_tree, &range);return error; }
@@ -2094,6 +2103,9 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,struct task_struct *me = current;unsigned char comm[sizeof (me->comm)];long error; +struct range_rwlock range; + +range_rwlock_init_full (&range); error =security_task_prctl (option, arg2, arg3, arg4, arg5);if (error != -ENOSYS )
@@ -2266,13 +2278,14 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,case PR_SET_THP_DISABLE :if (arg3 || arg4 || arg5)return -EINVAL ; -if (down_write_killable (&me->mm->mmap_sem)) +if (range_write_lock_interruptible (&me->mm->mmap_rw_tree, + &range))return -EINTR ;if (arg2) me->mm->def_flags |=VM_NOHUGEPAGE ;else me->mm->def_flags &= ~VM_NOHUGEPAGE ; -up_write (&me->mm->mmap_sem); +range_write_unlock (&me->mm->mmap_rw_tree, &range);break ;case PR_MPX_ENABLE_MANAGEMENT :if (arg2 || arg3 || arg4 || arg5)
kernel/trace/trace_output.c +4 -2
@@ -380,6 +380,7 @@ static int seq_print_user_ip(struct trace_seq *s, struct mm_struct *mm,struct file *file =NULL ;unsigned long vmstart =0 ;int ret =1 ; +struct range_rwlock range;if (s->full)return 0 ;
@@ -387,7 +388,8 @@ static int seq_print_user_ip(struct trace_seq *s, struct mm_struct *mm,if (mm) {const struct vm_area_struct *vma; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, ip);if (vma) { file = vma->vm_file;
@@ -399,7 +401,7 @@ static int seq_print_user_ip(struct trace_seq *s, struct mm_struct *mm,trace_seq_printf (s,"[+0x%lx]" , ip - vmstart); } -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range); }if (ret && ((sym_flags &TRACE_ITER_SYM_ADDR ) || !file))trace_seq_printf (s," <" IP_FMT ">" , ip);
mm/filemap.c +2 -2
@@ -1063,7 +1063,7 @@ int __lock_page_or_retry(struct page *page, struct mm_struct *mm,if (flags &FAULT_FLAG_RETRY_NOWAIT )return 0 ; -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, range);if (flags &FAULT_FLAG_KILLABLE )wait_on_page_locked_killable (page);else
@@ -1075,7 +1075,7 @@ int __lock_page_or_retry(struct page *page, struct mm_struct *mm, ret =__lock_page_killable (page);if (ret) { -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, range);return 0 ; } }else
mm/frame_vector.c +6 -3
@@ -35,6 +35,7 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames, {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; +struct range_rwlock range;int ret =0 ;int err;int locked;
@@ -45,7 +46,8 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames,if (WARN_ON_ONCE (nr_frames > vec->nr_allocated)) nr_frames = vec->nr_allocated; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); locked =1 ; vma =find_vma_intersection (mm, start, start +1 );if (!vma) {
@@ -56,7 +58,8 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames, vec->got_ref = true; vec->is_pfns = false; ret =get_user_pages_locked (start, nr_frames, - gup_flags, (struct page **)(vec->ptrs), &locked,NULL ); + gup_flags, (struct page **)(vec->ptrs), + &locked, &range); goto out; }
@@ -85,7 +88,7 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames, }while (vma && vma->vm_flags & (VM_IO |VM_PFNMAP )); out:if (locked) -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);if (!ret) ret = -EFAULT ;if (ret >0 )
mm/gup.c +13 -9
@@ -735,7 +735,7 @@ int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm, }if (ret &VM_FAULT_RETRY ) { -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, range);if (!(fault_flags &FAULT_FLAG_TRIED )) { *unlocked = true; fault_flags &= ~FAULT_FLAG_ALLOW_RETRY ;
@@ -819,7 +819,7 @@ static __always_inline long __get_user_pages_locked(struct task_struct *tsk,*/ *locked =1 ; lock_dropped = true; -down_read (&mm->mmap_sem); +range_read_lock (&mm->mmap_rw_tree, range); ret =__get_user_pages (tsk, mm, start,1 , flags |FOLL_TRIED , pages,NULL ,NULL , range);if (ret !=1 ) {
@@ -840,7 +840,7 @@ static __always_inline long __get_user_pages_locked(struct task_struct *tsk,* We must let the caller know we temporarily dropped the lock * and so the critical section protected by it was lost. */ -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, range); *locked =0 ; }return pages_done;
@@ -892,12 +892,14 @@ static __always_inline long __get_user_pages_unlocked(struct task_struct *tsk, {long ret;int locked =1 ; +struct range_rwlock range; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); ret =__get_user_pages_locked (tsk, mm, start, nr_pages, pages,NULL , - &locked, false,NULL , gup_flags); + &locked, false, &range, gup_flags);if (locked) -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return ret; }
@@ -1081,6 +1083,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)struct vm_area_struct *vma =NULL ;int locked =0 ;long ret =0 ; +struct range_rwlock range;VM_BUG_ON (start & ~PAGE_MASK );VM_BUG_ON (len !=PAGE_ALIGN (len));
@@ -1093,7 +1096,8 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)*/ if (!locked) { locked =1 ; -down_read (&mm->mmap_sem); +range_rwlock_init_full (&range); +range_read_lock (&mm->mmap_rw_tree, &range); vma =find_vma (mm, nstart); }else if (nstart >= vma->vm_end) vma = vma->vm_next;
@@ -1114,7 +1118,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)* if the vma was already munlocked. */ ret =populate_vma_page_range (vma, nstart, nend, &locked, -NULL ); + &range);if (ret <0 ) {if (ignore_errors) { ret =0 ;
@@ -1126,7 +1130,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors) ret =0 ; }if (locked) -up_read (&mm->mmap_sem); +range_read_unlock (&mm->mmap_rw_tree, &range);return ret;/* 0 or negative error code */ }
mm/init-mm.c +1 -1
@@ -19,7 +19,7 @@ struct mm_struct init_mm = { .pgd = swapper_pg_dir, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), - .mmap_sem = __RWSEM_INITIALIZER(init_mm.mmap_sem), + .mmap_rw_tree = __RANGE_RWLOCK_TREE_INITIALIZER(init_mm.mmap_rw_tree), .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), .user_ns = &init_user_ns,
mm/khugepaged.c +27 -17
32 lines not on this page · raw message
mm/ksm.c +30 -15
45 lines not on this page · raw message
mm/mmap.c +30 -13
43 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 7 lines
2.7.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Part of the diff is not shown here. View the raw message.
Peter Zijlstra
maintainerRe: [RFC 4/4] Change mmap_sem to range lock
Quoted 1 line · Laurent Dufour
[resent this patch which seems to have not reached the mailing lists]
Probably because its too big at ~180k ?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 4/4] Change mmap_sem to range lock
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Laurent Dufour
[resent this patch which seems to have not reached the mailing lists]
Probably because its too big at ~180k ?
Probably, but this time it has reached linux-mm ... at least.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 4/4] Change mmap_sem to range lock
In reply to Laurent DufourQuoted 5 lines · Laurent Dufour
[resent this patch which seems to have not reached the mailing lists]
Change the
mmap_semto a range lock to allow finer grain locking on the memory layout of a task.
Quoted 7 lines
This patch rename
mmap_semintommap_rw_treeto avoid confusion and replace any locking (read or write) by complete range locking. So there is no functional change except in the way the underlying locking is achieved.Currently, this patch only supports x86 and PowerPc architectures, furthermore it should break the build of any others.
Thanks for working on this.
However as commented before I think the first step to make progress here is a description of everything mmap_sem protects.
Surely the init full case could be done shorter with some wrapper that combines the init_full and lock operation?
Then it would be likely a simple search'n'replace to move the whole tree in one atomic step to the new wrappers. Initially they could be just defined to use rwsems too to not change anything at all.
It would be a good idea to merge such a patch as quickly as possible beause it will be a nightmare to maintain longer term.
Then you could add a config to use a range lock through the wrappers.
Then after that you could add real ranges step by step, after doing the proper analysis.
-Andi
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 4/4] Change mmap_sem to range lock
Quoted 12 lines · Andi Kleen
Quoted 10 lines · Laurent Dufour
[resent this patch which seems to have not reached the mailing lists]
Change the
mmap_semto a range lock to allow finer grain locking on the memory layout of a task.This patch rename
mmap_semintommap_rw_treeto avoid confusion and replace any locking (read or write) by complete range locking. So there is no functional change except in the way the underlying locking is achieved.
Quoted 7 lines
Quoted 2 lines
Currently, this patch only supports x86 and PowerPc architectures, furthermore it should break the build of any others.
Thanks for working on this.
However as commented before I think the first step to make progress here is a description of everything
mmap_semprotects.
Hi Andy,
I looked for the write mmap_sem locking in x86 and ppc64 architectures, here is what I found:
mmap_sem protects
vdso mapping
VMA layout changes
VMA cache
Page protection/layout
Changes to mmu notifier chain
mmap_sem is used to serialize khugepaged's access mmap_sem is used to serialize ksm's access protection keys (pkey_alloc()...)
Calls to
get_unmap_area()
do_mmap()
do_mmap_pgoff()
do_munmap()
get_user_pages()
put_page()
set_page_dirty_lock()
find_vma()
find_vma_intersection()
alloc_empty_pages()
insert_vm_struct()
get_mm_rss()
uprobe_consumer->filter() (currently only uprobe_perf_filter()) _install_special_mapping()
pmdp_collapse_flush()
do_swap_page()
do_brk()
__split_vma()
mremap_to()
vma_to_resize()
vma_adjust()
MM fields
pinned_vm
stack_vm
total_vm
locked_vm
start_stack
start_code
end_code
start_data
start_brk
bd_addr
mm_users
core_state
context.vdso_*
def_flags
mmu_notifier_mm
VMA fields
vm_private_data
vm_flags
vm_page_prot
vm_file
vm_pgoff
vm_policy
Userfaultfd has not been looked in details yet.dup_mmap() locks the oldmm in write mode when copying it, is it necessary ?
Quoted 2 lines
Surely the init full case could be done shorter with some wrapper that combines the
init_fulland lock operation?
Yes that doable, I wrote this like that, because the range should be initialized based on the on going operation, so having an explicit init operation is making this more explicit.
Quoted 11 lines
Then it would be likely a simple search'n'replace to move the whole tree in one atomic step to the new wrappers. Initially they could be just defined to use rwsems too to not change anything at all.
It would be a good idea to merge such a patch as quickly as possible beause it will be a nightmare to maintain longer term.
Then you could add a config to use a range lock through the wrappers.
I agree, I should try a way to make that patch activated through a CONFIG_value, but there is a the additional range value that make it more complex to achieve. I'll try to figure out a way to do that.
Quoted 2 lines
Then after that you could add real ranges step by step, after doing the proper analysis.
That's the biggest part of the job.
I'm also wondering if a dedicated lock/sem should be introduced to protect the VMA cache and the VMA list, since the range itself will not protect against change while walking the VMA list.
Please advise.
Cheers,
Laurent.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>