per-VMA locks proposal
Subsystems (18)
Suren Baghdasaryan
maintainercover per-VMA locks proposal
This is a proof of concept for per-vma locks idea that was discussed during SPF [1] discussion at LSF/MM this year [2], which concluded with suggestion that “a reader/writer semaphore could be put into the VMA itself; that would have the effect of using the VMA as a sort of range lock. There would still be contention at the VMA level, but it would be an improvement.” This patchset implements this suggested approach.
When handling page faults we lookup the VMA that contains the faulting page under RCU protection and try to acquire its lock. If that fails we fall back to using mmap_lock, similar to how SPF handled this situation.
One notable way the implementation deviates from the proposal is the way VMAs are marked as locked. Because during some of mm updates multiple VMAs need to be locked until the end of the update (e.g. vma_merge, split_vma, etc). Tracking all the locked VMAs, avoiding recursive locks and other complications would make the code more complex. Therefore we provide a way to "mark" VMAs as locked and then unmark all locked VMAs all at once. This is done using two sequence numbers - one in the vm_area_struct and one in the mm_struct. VMA is considered locked when these sequence numbers are equal. To mark a VMA as locked we set the sequence number in vm_area_struct to be equal to the sequence number in mm_struct. To unlock all VMAs we increment mm_struct's seq number. This allows for an efficient way to track locked VMAs and to drop the locks on all VMAs at the end of the update.
The patchset implements per-VMA locking only for anonymous pages which are not in swap. If the initial proposal is considered acceptable, then support for swapped and file-backed page faults will be added.
Performance benchmarks show similar although slightly smaller benefits as with SPF patchset (~75% of SPF benefits). Still, with lower complexity this approach might be more desirable.
The patchset applies cleanly over 6.0-rc3
The tree for testing is posted at [3]
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lwn.net/Articles/893906/
[3] https://github.com/surenbaghdasaryan/linux/tree/per_vma_lock_rfc
Laurent Dufour (2):
powerc/mm: try VMA lock-based page fault handling first powerpc/mm: define ARCH_SUPPORTS_PER_VMA_LOCK
Michel Lespinasse (1):
mm: rcu safe VMA freeing
Suren Baghdasaryan (25):
mm: introduce CONFIG_PER_VMA_LOCK
mm: introduce __find_vma to be used without mmap_lock protection
mm: move mmap_lock assert function definitions
mm: add per-VMA lock and helper functions to control it
mm: mark VMA as locked whenever vma->vm_flags are modified kernel/fork: mark VMAs as locked before copying pages during fork mm/khugepaged: mark VMA as locked while collapsing a hugepage mm/mempolicy: mark VMA as locked when changing protection policy mm/mmap: mark VMAs as locked in vma_adjust
mm/mmap: mark VMAs as locked before merging or splitting them mm/mremap: mark VMA as locked while remapping it to a new address
range
mm: conditionally mark VMA as locked in free_pgtables and
unmap_page_range
mm: mark VMAs as locked before isolating them
mm/mmap: mark adjacent VMAs as locked if they can grow into unmapped
area
kernel/fork: assert no VMA readers during its destruction mm/mmap: prevent pagefault handler from racing with mmu_notifier
registration
mm: add FAULT_FLAG_VMA_LOCK flag
mm: disallow do_swap_page to handle page faults under VMA lock
mm: introduce per-VMA lock statistics
mm: introduce find_and_lock_anon_vma to be used from arch-specific
code
x86/mm: try VMA lock-based page fault handling first x86/mm: define ARCH_SUPPORTS_PER_VMA_LOCK
arm64/mm: try VMA lock-based page fault handling first arm64/mm: define ARCH_SUPPORTS_PER_VMA_LOCK
kernel/fork: throttle call_rcu() calls in vm_area_free
arch/arm64/Kconfig | 1 + arch/arm64/mm/fault.c | 36 +++++++++ arch/powerpc/mm/fault.c | 41 ++++++++++ arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + arch/x86/Kconfig | 1 + arch/x86/mm/fault.c | 36 +++++++++ drivers/gpu/drm/i915/i915_gpu_error.c | 4 +- fs/proc/task_mmu.c | 1 + fs/userfaultfd.c | 6 ++ include/linux/mm.h | 104 ++++++++++++++++++++++++- include/linux/mm_types.h | 33 ++++++-- include/linux/mmap_lock.h | 37 ++++++--- include/linux/vm_event_item.h | 6 ++ include/linux/vmstat.h | 6 ++ kernel/fork.c | 75 +++++++++++++++++- mm/Kconfig | 13 ++++ mm/Kconfig.debug | 8 ++ mm/init-mm.c | 6 ++ mm/internal.h | 4 +- mm/khugepaged.c | 1 + mm/madvise.c | 1 + mm/memory.c | 82 ++++++++++++++++--- mm/mempolicy.c | 6 +- mm/mlock.c | 2 + mm/mmap.c | 60 ++++++++++---- mm/mprotect.c | 1 + mm/mremap.c | 1 + mm/nommu.c | 2 + mm/oom_kill.c | 3 +- mm/vmstat.c | 6 ++ 31 files changed, 531 insertions(+), 54 deletions(-)
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 1/28 mm: introduce CONFIG_PER_VMA_LOCK
This configuration variable will be used to build the support for VMA locking during page fault handling.
This is enabled by default on supported architectures with SMP and MMU set.
The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock.
--- mm/Kconfig | 13 +++++++++++++ 1 file changed, 13 insertions(+)
mm/Kconfig +13 -0
@@ -1124,6 +1124,19 @@ config PTE_MARKER_UFFD_WP purposes. It is required to enable userfaultfd write protection on file-backed memory types like shmem and hugetlbfs. +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + bool "Per-vma locking support" + default y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 2/28 mm: rcu safe VMA freeing
In reply to Suren BaghdasaryanFrom: Michel Lespinasse <[email protected]>
This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock.
--- include/linux/mm_types.h | 16 +++++++++++----- kernel/fork.c | 13 +++++++++++++ 2 files changed, 24 insertions(+), 5 deletions(-)
include/linux/mm_types.h +11 -5
@@ -403,12 +403,18 @@ struct anon_vma_name {struct vm_area_struct {/* The first cache line has the info for VMA tree walking. */ -unsigned long vm_start;/* Our start address within vm_mm. */ -unsigned long vm_end;/* The first byte after our end address -within vm_mm. */ +union { +struct { +/* VMA covers [vm_start; vm_end) addresses within mm */ +unsigned long vm_start, vm_end; -/* linked list of VM areas per task, sorted by address */ -struct vm_area_struct *vm_next, *vm_prev; +/* linked list of VMAs per task, sorted by address */ +struct vm_area_struct *vm_next, *vm_prev; + }; +#ifdef CONFIG_PER_VMA_LOCK +struct rcu_head vm_rcu;/* Used for deferred freeing. */ +#endif + };struct rb_node vm_rb;
kernel/fork.c +13 -0
@@ -481,10 +481,23 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)return new; } +#ifdef CONFIG_PER_VMA_LOCK +static void __vm_area_free (struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); +kmem_cache_free (vm_area_cachep, vma); +} +#endif +void vm_area_free (struct vm_area_struct *vma) {free_anon_vma_name (vma); +#ifdef CONFIG_PER_VMA_LOCK +call_rcu (&vma->vm_rcu, __vm_area_free); +#else kmem_cache_free (vm_area_cachep, vma); +#endif }static void account_kernel_stack (struct task_struct *tsk,int account)
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 3/28 mm: introduce __find_vma to be used without mmap_lock protection
In reply to Suren BaghdasaryanAdd __find_vma function to be used for VMA lookup under rcu protection.
--- drivers/gpu/drm/i915/i915_gpu_error.c | 4 ++-- include/linux/mm.h | 9 ++++++++- mm/mmap.c | 6 ++---- 3 files changed, 12 insertions(+), 7 deletions(-)
drivers/gpu/drm/i915/i915_gpu_error.c +2 -2
@@ -507,7 +507,7 @@ static void error_print_context(struct drm_i915_error_state_buf *m, }static struct i915_vma_coredump * -__find_vma (struct i915_vma_coredump *vma,const char *name) +__i915_find_vma (struct i915_vma_coredump *vma,const char *name) {while (vma) {if (strcmp (vma->name, name) ==0 )
@@ -521,7 +521,7 @@ __find_vma(struct i915_vma_coredump *vma, const char *name)struct i915_vma_coredump *intel_gpu_error_find_batch (const struct intel_engine_coredump *ee) { -return __find_vma (ee->vma,"batch" ); +return __i915_find_vma (ee->vma,"batch" ); } static void error_print_engine(struct drm_i915_error_state_buf *m,
include/linux/mm.h +8 -1
@@ -2712,7 +2712,14 @@ extern int expand_upwards(struct vm_area_struct *vma, unsigned long address);#endif /* Look up the first VMA which satisfies addr < vm_end, NULL if none. */ -extern struct vm_area_struct *find_vma (struct mm_struct * mm,unsigned long addr); +extern struct vm_area_struct *__find_vma (struct mm_struct *mm,unsigned long addr); +static inline +struct vm_area_struct *find_vma (struct mm_struct *mm,unsigned long addr) +{ +mmap_assert_locked (mm); +return __find_vma (mm, addr); +} +extern struct vm_area_struct *find_vma_prev (struct mm_struct * mm,unsigned long addr,struct vm_area_struct **pprev);
mm/mmap.c +2 -4
@@ -2250,12 +2250,11 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,EXPORT_SYMBOL (get_unmapped_area);/* Look up the first VMA which satisfies addr < vm_end, NULL if none. */ -struct vm_area_struct *find_vma (struct mm_struct *mm,unsigned long addr) +struct vm_area_struct *__find_vma (struct mm_struct *mm,unsigned long addr) {struct rb_node *rb_node;struct vm_area_struct *vma; -mmap_assert_locked (mm);/* Check the cache first. */ vma =vmacache_find (mm, addr);if (likely (vma))
@@ -2281,8 +2280,7 @@ struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr)vmacache_update (addr, vma);return vma; } - -EXPORT_SYMBOL (find_vma); +EXPORT_SYMBOL (__find_vma);/* * Same as find_vma, but also return a pointer to the previous VMA in *pprev.
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 4/28 mm: move mmap_lock assert function definitions
In reply to Suren BaghdasaryanMove mmap_lock assert function definitions up so that they can be used by other mmap_lock routines.
--- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
include/linux/mmap_lock.h +12 -12
@@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write)#endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked (struct mm_struct *mm) +{ +lockdep_assert_held (&mm->mmap_lock); +VM_BUG_ON_MM (!rwsem_is_locked (&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked (struct mm_struct *mm) +{ +lockdep_assert_held_write (&mm->mmap_lock); +VM_BUG_ON_MM (!rwsem_is_locked (&mm->mmap_lock), mm); +} +static inline void mmap_init_lock (struct mm_struct *mm) {init_rwsem (&mm->mmap_lock);
@@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm)up_read_non_owner (&mm->mmap_lock); } -static inline void mmap_assert_locked (struct mm_struct *mm) -{ -lockdep_assert_held (&mm->mmap_lock); -VM_BUG_ON_MM (!rwsem_is_locked (&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked (struct mm_struct *mm) -{ -lockdep_assert_held_write (&mm->mmap_lock); -VM_BUG_ON_MM (!rwsem_is_locked (&mm->mmap_lock), mm); -} -static inline int mmap_lock_is_contended (struct mm_struct *mm) {return rwsem_is_contended (&mm->mmap_lock);
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 5/28 mm: add per-VMA lock and helper functions to control it
In reply to Suren BaghdasaryanIntroduce a per-VMA rw_semaphore to be used during page fault handling instead of mmap_lock. Because there are cases when multiple VMAs need to be exclusively locked during VMA tree modifications, instead of the usual lock/unlock patter we mark a VMA as locked by taking per-VMA lock exclusively and setting vma->lock_seq to the current mm->lock_seq. When mmap_write_lock holder is done with all modifications and drops mmap_lock, it will increment mm->lock_seq, effectively unlocking all VMAs marked as locked.
--- include/linux/mm.h | 78 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 7 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 105 insertions(+)
include/linux/mm.h +78 -0
@@ -611,6 +611,83 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock (struct vm_area_struct *vma) +{ +init_rwsem (&vma->lock); + vma->vm_lock_seq =-1 ; +} + +static inline void vma_mark_locked (struct vm_area_struct *vma) +{ +int mm_lock_seq; + +mmap_assert_write_locked (vma->vm_mm); + +/* +* current task is holding mmap_write_lock, both vma->vm_lock_seq and +* mm->mm_lock_seq can't be concurrently modified. +*/ + mm_lock_seq =READ_ONCE (vma->vm_mm->mm_lock_seq); +if (vma->vm_lock_seq == mm_lock_seq) +return ; + +down_write (&vma->lock); + vma->vm_lock_seq = mm_lock_seq; +up_write (&vma->lock); +} + +static inline bool vma_read_trylock (struct vm_area_struct *vma) +{ +if (unlikely (down_read_trylock (&vma->lock) ==0 )) +return false; + +/* +* Overflow might produce false locked result but it's not critical. +* False unlocked result is critical but is impossible because we +* modify and check vma->vm_lock_seq under vma->lock protection and +* mm->mm_lock_seq modification invalidates all existing locks. +*/ +if (vma->vm_lock_seq ==READ_ONCE (vma->vm_mm->mm_lock_seq)) { +up_read (&vma->lock); +return false; + } +return true; +} + +static inline void vma_read_unlock (struct vm_area_struct *vma) +{ +up_read (&vma->lock); +} + +static inline void vma_assert_locked (struct vm_area_struct *vma) +{ +lockdep_assert_held (&vma->lock); +VM_BUG_ON_VMA (!rwsem_is_locked (&vma->lock), vma); +} + +static inline void vma_assert_write_locked (struct vm_area_struct *vma,int pos) +{ +mmap_assert_write_locked (vma->vm_mm); +/* +* current task is holding mmap_write_lock, both vma->vm_lock_seq and +* mm->mm_lock_seq can't be concurrently modified. +*/ +VM_BUG_ON_VMA (vma->vm_lock_seq !=READ_ONCE (vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock (struct vm_area_struct *vma) {} +static inline void vma_mark_locked (struct vm_area_struct *vma) {} +static inline bool vma_read_trylock (struct vm_area_struct *vma) + {return false; } +static inline void vma_read_unlock (struct vm_area_struct *vma) {} +static inline void vma_assert_locked (struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked (struct vm_area_struct *vma,int pos) {} + +#endif /* CONFIG_PER_VMA_LOCK */ +static inline void vma_init (struct vm_area_struct *vma,struct mm_struct *mm) {static const struct vm_operations_struct dummy_vm_ops = {};
@@ -619,6 +696,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops;INIT_LIST_HEAD (&vma->anon_vma_chain); +vma_init_lock (vma); }static inline void vma_set_anonymous (struct vm_area_struct *vma)
include/linux/mm_types.h +7 -0
@@ -486,6 +486,10 @@ struct vm_area_struct {struct mempolicy *vm_policy;/* NUMA policy for the VMA */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK +struct rw_semaphore lock; +int vm_lock_seq; +#endif } __randomize_layout;struct kioctx_table ;
@@ -567,6 +571,9 @@ struct mm_struct {* init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK +int mm_lock_seq; +#endif unsigned long hiwater_rss;/* High-watermark of RSS usage */
include/linux/mmap_lock.h +13 -0
@@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm)VM_BUG_ON_MM (!rwsem_is_locked (&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_mark_unlocked_all (struct mm_struct *mm) +{ +mmap_assert_write_locked (mm); +/* No races during update due to exclusive mmap_lock being held */ +WRITE_ONCE (mm->mm_lock_seq, mm->mm_lock_seq +1 ); +} +#else +static inline void vma_mark_unlocked_all (struct mm_struct *mm) {} +#endif +static inline void mmap_init_lock (struct mm_struct *mm) {init_rwsem (&mm->mmap_lock);
@@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm)static inline void mmap_write_unlock (struct mm_struct *mm) {__mmap_lock_trace_released (mm, true); +vma_mark_unlocked_all (mm);up_write (&mm->mmap_lock); }static inline void mmap_write_downgrade (struct mm_struct *mm) {__mmap_lock_trace_acquire_returned (mm, false, true); +vma_mark_unlocked_all (mm);downgrade_write (&mm->mmap_lock); }
kernel/fork.c +4 -0
@@ -475,6 +475,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)*/ *new =data_race (*orig);INIT_LIST_HEAD (&new->anon_vma_chain); +vma_init_lock (new); new->vm_next = new->vm_prev =NULL ;dup_anon_vma_name (orig, new); }
@@ -1130,6 +1131,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,seqcount_init (&mm->write_protect_seq);mmap_init_lock (mm);INIT_LIST_HEAD (&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK +WRITE_ONCE (mm->mm_lock_seq,0 ); +#endif mm_pgtables_bytes_init (mm); mm->map_count =0 ; mm->locked_vm =0 ;
mm/init-mm.c +3 -0
@@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist =LIST_HEAD_INIT (init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 6/28 mm: mark VMA as locked whenever vma->vm_flags are modified
In reply to Suren BaghdasaryanVMA flag modifications should be done under VMA lock to prevent concurrent page fault handling in that area.
--- fs/proc/task_mmu.c | 1 + fs/userfaultfd.c | 6 ++++++ mm/madvise.c | 1 + mm/mlock.c | 2 ++ mm/mmap.c | 1 + mm/mprotect.c | 1 + 6 files changed, 12 insertions(+)
fs/proc/task_mmu.c +1 -0
@@ -1285,6 +1285,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf,for (vma = mm->mmap; vma; vma = vma->vm_next) {if (!(vma->vm_flags &VM_SOFTDIRTY ))continue ; +vma_mark_locked (vma); vma->vm_flags &= ~VM_SOFTDIRTY ;vma_set_page_prot (vma); }
fs/userfaultfd.c +6 -0
@@ -620,6 +620,7 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx,mmap_write_lock (mm);for (vma = mm->mmap; vma; vma = vma->vm_next)if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { +vma_mark_locked (vma); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ; vma->vm_flags &= ~__VM_UFFD_FLAGS; }
@@ -653,6 +654,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx;if (!octx || !(octx->features &UFFD_FEATURE_EVENT_FORK )) { +vma_mark_locked (vma); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ; vma->vm_flags &= ~__VM_UFFD_FLAGS;return 0 ;
@@ -734,6 +736,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma,atomic_inc (&ctx->mmap_changing); }else {/* Drop uffd context if remap feature not enabled */ +vma_mark_locked (vma); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ; vma->vm_flags &= ~__VM_UFFD_FLAGS; }
@@ -891,6 +894,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) vma = prev; else prev = vma; +vma_mark_locked (vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ; }
@@ -1449,6 +1453,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,* the next vma was merged into the current one and * the current one has not been updated yet. */ +vma_mark_locked (vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx.ctx = ctx;
@@ -1630,6 +1635,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,* the next vma was merged into the current one and * the current one has not been updated yet. */ +vma_mark_locked (vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ;
mm/madvise.c +1 -0
@@ -181,6 +181,7 @@ static int madvise_update_vma(struct vm_area_struct *vma,/* * vm_flags is protected by the mmap_lock held in write mode. */ +vma_mark_locked (vma); vma->vm_flags = new_flags;if (!vma->vm_file) { error =replace_anon_vma_name (vma, anon_name);
mm/mlock.c +2 -0
@@ -380,6 +380,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma,*/ if (newflags &VM_LOCKED ) newflags |=VM_IO ; +vma_mark_locked (vma);WRITE_ONCE (vma->vm_flags, newflags);lru_add_drain ();
@@ -456,6 +457,7 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev,if ((newflags &VM_LOCKED ) && (oldflags &VM_LOCKED )) {/* No work to do, and mlocking twice would be wrong */ +vma_mark_locked (vma); vma->vm_flags = newflags; }else {mlock_vma_pages_range (vma, start, end, newflags);
mm/mmap.c +1 -0
@@ -1818,6 +1818,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, out:perf_event_mmap (vma); +vma_mark_locked (vma);vm_stat_account (mm, vm_flags, len >>PAGE_SHIFT );if (vm_flags &VM_LOCKED ) { if ((vm_flags & VM_SPECIAL) || vma_is_dax(vma) ||
mm/mprotect.c +1 -0
@@ -621,6 +621,7 @@ mprotect_fixup(struct mmu_gather *tlb, struct vm_area_struct *vma,* vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ +vma_mark_locked (vma); vma->vm_flags = newflags;/* * We want to check manually if we can change individual PTEs writable
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 7/28 kernel/fork: mark VMAs as locked before copying pages during fork
In reply to Suren BaghdasaryanProtect VMAs from concurrent page fault handler while performing copy_page_range for VMAs having VM_WIPEONFORK flag set.
--- kernel/fork.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/fork.c +3 -1
@@ -709,8 +709,10 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, rb_parent = &tmp->vm_rb; mm->map_count++; -if (!(tmp->vm_flags &VM_WIPEONFORK )) +if (!(tmp->vm_flags &VM_WIPEONFORK )) { +vma_mark_locked (mpnt); retval =copy_page_range (tmp, mpnt); + }if (tmp->vm_ops && tmp->vm_ops->open) tmp->vm_ops->open (tmp);
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 8/28 mm/khugepaged: mark VMA as locked while collapsing a hugepage
In reply to Suren BaghdasaryanProtect VMA from concurrent page fault handler while modifying it in collapse_huge_page.
--- mm/khugepaged.c | 1 + 1 file changed, 1 insertion(+)
mm/khugepaged.c +1 -0
@@ -1072,6 +1072,7 @@ static void collapse_huge_page(struct mm_struct *mm,if (mm_find_pmd (mm, address) != pmd) goto out_up_write; +vma_mark_locked (vma);anon_vma_lock_write (vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm,
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 9/28 mm/mempolicy: mark VMA as locked when changing protection policy
In reply to Suren BaghdasaryanProtect VMA from concurrent page fault handler while performing VMA protection policy changes.
--- mm/mempolicy.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
mm/mempolicy.c +5 -1
@@ -383,8 +383,10 @@ void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new)struct vm_area_struct *vma;mmap_write_lock (mm); -for (vma = mm->mmap; vma; vma = vma->vm_next) +for (vma = mm->mmap; vma; vma = vma->vm_next) { +vma_mark_locked (vma);mpol_rebind_policy (vma->vm_policy, new); + }mmap_write_unlock (mm); }
@@ -632,6 +634,7 @@ unsigned long change_prot_numa(struct vm_area_struct *vma,struct mmu_gather tlb;int nr_updated; +vma_mark_locked (vma);tlb_gather_mmu (&tlb, vma->vm_mm); nr_updated = change_protection(&tlb, vma, addr, end, PAGE_NONE,
@@ -765,6 +768,7 @@ static int vma_replace_policy(struct vm_area_struct *vma,if (IS_ERR (new))return PTR_ERR (new); +vma_mark_locked (vma);if (vma->vm_ops && vma->vm_ops->set_policy) { err = vma->vm_ops->set_policy (vma, new);if (err)
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 10/28 mm/mmap: mark VMAs as locked in vma_adjust
In reply to Suren Baghdasaryanvma_adjust modifies a VMA and possibly its neighbors. Mark them as locked before making the modifications.
--- mm/mmap.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
mm/mmap.c +10 -1
@@ -710,6 +710,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,long adjust_next =0 ;int remove_next =0 ; +vma_mark_locked (vma); +if (next) +vma_mark_locked (next); +if (next && !insert) {struct vm_area_struct *exporter =NULL , *importer =NULL ;
@@ -754,8 +758,11 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,* If next doesn't have anon_vma, import from vma after * next, if the vma overlaps with it. */ -if (remove_next ==2 && !next->anon_vma) +if (remove_next ==2 && !next->anon_vma) { exporter = next->vm_next; +if (exporter) +vma_mark_locked (exporter); + } }else if (end > next->vm_start) {/*
@@ -931,6 +938,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,* "vma->vm_next" gap must be updated. */ next = vma->vm_next; +if (next) +vma_mark_locked (next); }else {/* * For the scope of the comment "next" and
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 11/28 mm/mmap: mark VMAs as locked before merging or splitting them
In reply to Suren BaghdasaryanDecisions about whether VMAs can be merged or split must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation.
Mark all VMAs which might be affected by a merge or split operation as locked before making decision how such operations should be performed.
--- mm/mmap.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
mm/mmap.c +9 -1
@@ -1147,10 +1147,17 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm,if (vm_flags &VM_SPECIAL )return NULL ; +if (prev) +vma_mark_locked (prev); next =vma_next (mm, prev); area = next; -if (area && area->vm_end == end)/* cases 6, 7, 8 */ +if (area) +vma_mark_locked (area); +if (area && area->vm_end == end) {/* cases 6, 7, 8 */ next = next->vm_next; +if (next) +vma_mark_locked (next); + }/* verify some invariant that must be enforced by the caller */ VM_WARN_ON (prev && addr <= prev->vm_start);
@@ -2687,6 +2694,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma,struct vm_area_struct *new;int err; +vma_mark_locked (vma);if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split (vma, addr);if (err)
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 12/28 mm/mremap: mark VMA as locked while remapping it to a new address range
In reply to Suren BaghdasaryanMark VMA as locked before copying it and when copy_vma produces a new VMA.
--- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+)
mm/mmap.c +1 -0
@@ -3248,6 +3248,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,get_file (new_vma->vm_file);if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open (new_vma); +vma_mark_locked (new_vma);vma_link (mm, new_vma, prev, rb_link, rb_parent); *need_rmap_locks = false; }
mm/mremap.c +1 -0
@@ -620,6 +620,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,return -ENOMEM ; } +vma_mark_locked (vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >>PAGE_SHIFT ); new_vma =copy_vma (&vma, new_addr, new_len, new_pgoff, &need_rmap_locks);
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 13/28 mm: conditionally mark VMA as locked in free_pgtables and unmap_page_range
In reply to Suren Baghdasaryanfree_pgtables and unmap_page_range functions can be called with mmap_lock held for write (e.g. in mmap_region), held for read (e.g in madvise_pageout) or not held at all (e.g in madvise_remove might drop mmap_lock before calling vfs_fallocate, which ends up calling unmap_page_range).
Provide free_pgtables and unmap_page_range with additional argument indicating whether to mark the VMA as locked or not based on the usage. The parameter is set based on whether mmap_lock is held in write mode during the call. This ensures no change in behavior between mmap_lock and per-vma locks.
--- include/linux/mm.h | 2 +- mm/internal.h | 4 ++-- mm/memory.c | 32 +++++++++++++++++++++----------- mm/mmap.c | 17 +++++++++-------- mm/oom_kill.c | 3 ++- 5 files changed, 35 insertions(+), 23 deletions(-)
include/linux/mm.h +1 -1
@@ -1874,7 +1874,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address,void zap_page_range (struct vm_area_struct *vma,unsigned long address,unsigned long size);void unmap_vmas (struct mmu_gather *tlb,struct vm_area_struct *start_vma, -unsigned long start,unsigned long end); +unsigned long start,unsigned long end,bool lock_vma);struct mmu_notifier_range ;
mm/internal.h +2 -2
@@ -85,14 +85,14 @@ bool __folio_end_writeback(struct folio *folio);void deactivate_file_folio (struct folio *folio);void free_pgtables (struct mmu_gather *tlb,struct vm_area_struct *start_vma, -unsigned long floor,unsigned long ceiling); +unsigned long floor,unsigned long ceiling,bool lock_vma);void pmd_install (struct mm_struct *mm,pmd_t *pmd,pgtable_t *pte);struct zap_details ;void unmap_page_range (struct mmu_gather *tlb,struct vm_area_struct *vma,unsigned long addr,unsigned long end, -struct zap_details *details); +struct zap_details *details,bool lock_vma);void page_cache_ra_order (struct readahead_control *,struct file_ra_state *,unsigned int order);
mm/memory.c +21 -11
@@ -403,7 +403,7 @@ void free_pgd_range(struct mmu_gather *tlb, }void free_pgtables (struct mmu_gather *tlb,struct vm_area_struct *vma, -unsigned long floor,unsigned long ceiling) +unsigned long floor,unsigned long ceiling,bool lock_vma) {while (vma) {struct vm_area_struct *next = vma->vm_next;
@@ -413,6 +413,8 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma,* Hide vma from rmap and truncate_pagecache before freeing * pgtables */ +if (lock_vma) +vma_mark_locked (vma);unlink_anon_vmas (vma);unlink_file_vma (vma);
@@ -427,6 +429,8 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, && !is_vm_hugetlb_page(next)) { vma = next; next = vma->vm_next; +if (lock_vma) +vma_mark_locked (vma);unlink_anon_vmas (vma);unlink_file_vma (vma); }
@@ -1631,12 +1635,16 @@ static inline unsigned long zap_p4d_range(struct mmu_gather *tlb,void unmap_page_range (struct mmu_gather *tlb,struct vm_area_struct *vma,unsigned long addr,unsigned long end, -struct zap_details *details) +struct zap_details *details, +bool lock_vma) {pgd_t *pgd;unsigned long next;BUG_ON (addr >= end); +if (lock_vma) +vma_mark_locked (vma); +tlb_start_vma (tlb, vma); pgd =pgd_offset (vma->vm_mm, addr); do {
@@ -1652,7 +1660,7 @@ void unmap_page_range(struct mmu_gather *tlb,static void unmap_single_vma (struct mmu_gather *tlb,struct vm_area_struct *vma,unsigned long start_addr,unsigned long end_addr, -struct zap_details *details) +struct zap_details *details,bool lock_vma) {unsigned long start =max (vma->vm_start, start_addr);unsigned long end;
@@ -1691,7 +1699,7 @@ static void unmap_single_vma(struct mmu_gather *tlb,i_mmap_unlock_write (vma->vm_file->f_mapping); } }else -unmap_page_range (tlb, vma, start, end, details); +unmap_page_range (tlb, vma, start, end, details, lock_vma); } }
@@ -1715,7 +1723,7 @@ static void unmap_single_vma(struct mmu_gather *tlb,*/ void unmap_vmas (struct mmu_gather *tlb,struct vm_area_struct *vma,unsigned long start_addr, -unsigned long end_addr) +unsigned long end_addr,bool lock_vma) {struct mmu_notifier_range range;struct zap_details details = {
@@ -1728,7 +1736,8 @@ void unmap_vmas(struct mmu_gather *tlb, start_addr, end_addr);mmu_notifier_invalidate_range_start (&range);for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) -unmap_single_vma (tlb, vma, start_addr, end_addr, &details); +unmap_single_vma (tlb, vma, start_addr, end_addr, &details, + lock_vma);mmu_notifier_invalidate_range_end (&range); }
@@ -1753,7 +1762,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start,update_hiwater_rss (vma->vm_mm);mmu_notifier_invalidate_range_start (&range);for ( ; vma && vma->vm_start < range.end; vma = vma->vm_next) -unmap_single_vma (&tlb, vma, start, range.end,NULL ); +unmap_single_vma (&tlb, vma, start, range.end,NULL , false);mmu_notifier_invalidate_range_end (&range);tlb_finish_mmu (&tlb); }
@@ -1768,7 +1777,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start,* The range must fit into one VMA. */ static void zap_page_range_single (struct vm_area_struct *vma,unsigned long address, -unsigned long size,struct zap_details *details) +unsigned long size,struct zap_details *details,bool lock_vma) {struct mmu_notifier_range range;struct mmu_gather tlb;
@@ -1779,7 +1788,7 @@ static void zap_page_range_single(struct vm_area_struct *vma, unsigned long addrtlb_gather_mmu (&tlb, vma->vm_mm);update_hiwater_rss (vma->vm_mm);mmu_notifier_invalidate_range_start (&range); -unmap_single_vma (&tlb, vma, address, range.end, details); +unmap_single_vma (&tlb, vma, address, range.end, details, lock_vma);mmu_notifier_invalidate_range_end (&range);tlb_finish_mmu (&tlb); }
@@ -1802,7 +1811,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, !(vma->vm_flags & VM_PFNMAP))return ; -zap_page_range_single (vma, address, size,NULL ); +zap_page_range_single (vma, address, size,NULL , true); }EXPORT_SYMBOL_GPL (zap_vma_ptes);
@@ -3483,7 +3492,8 @@ static void unmap_mapping_range_vma(struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, struct zap_details *details) { - zap_page_range_single(vma, start_addr, end_addr - start_addr, details); + zap_page_range_single(vma, start_addr, end_addr - start_addr, details, +false ); } static inline void unmap_mapping_range_tree(struct rb_root_cached *root,
mm/mmap.c +9 -8
@@ -79,7 +79,7 @@ core_param(ignore_rlimit_data, ignore_rlimit_data, bool, 0644);static void unmap_region (struct mm_struct *mm,struct vm_area_struct *vma,struct vm_area_struct *prev, -unsigned long start,unsigned long end); +unsigned long start,unsigned long end,bool lock_vma);static pgprot_t vm_pgprot_modify (pgprot_t oldprot,unsigned long vm_flags) {
@@ -1866,7 +1866,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma->vm_file =NULL ;/* Undo any partial mapping done by a device driver. */ -unmap_region (mm, vma, prev, vma->vm_start, vma->vm_end); +unmap_region (mm, vma, prev, vma->vm_start, vma->vm_end, true);if (vm_flags &VM_SHARED )mapping_unmap_writable (file->f_mapping); free_vma:
@@ -2626,7 +2626,7 @@ static void remove_vma_list(struct mm_struct *mm, struct vm_area_struct *vma)*/ static void unmap_region (struct mm_struct *mm,struct vm_area_struct *vma,struct vm_area_struct *prev, -unsigned long start,unsigned long end) +unsigned long start,unsigned long end,bool lock_vma) {struct vm_area_struct *next =vma_next (mm, prev);struct mmu_gather tlb;
@@ -2634,9 +2634,10 @@ static void unmap_region(struct mm_struct *mm,lru_add_drain ();tlb_gather_mmu (&tlb, mm);update_hiwater_rss (mm); -unmap_vmas (&tlb, vma, start, end); +unmap_vmas (&tlb, vma, start, end, lock_vma);free_pgtables (&tlb, vma, prev ? prev->vm_end :FIRST_USER_ADDRESS , - next ? next->vm_start :USER_PGTABLES_CEILING ); + next ? next->vm_start :USER_PGTABLES_CEILING , + lock_vma);tlb_finish_mmu (&tlb); }
@@ -2849,7 +2850,7 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len,if (downgrade)mmap_write_downgrade (mm); -unmap_region (mm, vma, prev, start, end); +unmap_region (mm, vma, prev, start, end, !downgrade);/* Fix up all other VM information */ remove_vma_list (mm, vma);
@@ -3129,8 +3130,8 @@ void exit_mmap(struct mm_struct *mm)tlb_gather_mmu_fullmm (&tlb, mm);/* update_hiwater_rss(mm) here? but nobody should be looking */ /* Use -1 here to ensure all VMAs in the mm are unmapped */ -unmap_vmas (&tlb, vma,0 ,-1 ); -free_pgtables (&tlb, vma,FIRST_USER_ADDRESS ,USER_PGTABLES_CEILING ); +unmap_vmas (&tlb, vma,0 ,-1 , true); +free_pgtables (&tlb, vma,FIRST_USER_ADDRESS ,USER_PGTABLES_CEILING , true);tlb_finish_mmu (&tlb);/* Walk the list again, actually closing and freeing it. */
mm/oom_kill.c +2 -1
@@ -549,7 +549,8 @@ bool __oom_reap_task_mm(struct mm_struct *mm) ret = false;continue ; } -unmap_page_range (&tlb, vma, range.start, range.end,NULL ); +unmap_page_range (&tlb, vma, range.start, range.end, +NULL , false);mmu_notifier_invalidate_range_end (&range);tlb_finish_mmu (&tlb); }
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 14/28 mm: mark VMAs as locked before isolating them
In reply to Suren BaghdasaryanMark VMAs as locked before isolating them and clear their tree node so that isolated VMAs are easily identifiable. In the later patches page fault handlers will try locking the found VMA and will check whether the VMA was isolated. Locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs.
--- mm/mmap.c | 2 ++ mm/nommu.c | 2 ++ 2 files changed, 4 insertions(+)
mm/mmap.c +2 -0
@@ -421,12 +421,14 @@ static inline void vma_rb_insert(struct vm_area_struct *vma,static void __vma_rb_erase (struct vm_area_struct *vma,struct rb_root *root) { +vma_mark_locked (vma);/* * Note rb_erase_augmented is a fairly large inline function, * so make sure we instantiate it only once with our desired * augmented rbtree callbacks. */ rb_erase_augmented (&vma->vm_rb, root, &vma_gap_callbacks); +RB_CLEAR_NODE (&vma->vm_rb); } static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma,
mm/nommu.c +2 -0
@@ -622,6 +622,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma)struct mm_struct *mm = vma->vm_mm;struct task_struct *curr = current; +vma_mark_locked (vma); mm->map_count--;for (i =0 ; i <VMACACHE_SIZE ; i++) {/* if the vma is cached, invalidate the entire cache */
@@ -644,6 +645,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma)/* remove from the MM's tree and list */ rb_erase (&vma->vm_rb, &mm->mm_rb); +RB_CLEAR_NODE (&vma->vm_rb);__vma_unlink_list (mm, vma); }
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 15/28 mm/mmap: mark adjacent VMAs as locked if they can grow into unmapped area
In reply to Suren BaghdasaryanWhile unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases mark adjacent VMAs as locked to prevent this growth.
--- mm/mmap.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
mm/mmap.c +6 -2
@@ -2680,10 +2680,14 @@ detach_vmas_to_be_unmapped(struct mm_struct *mm, struct vm_area_struct *vma,* VM_GROWSUP VMA. Such VMAs can change their size under * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ -if (vma && (vma->vm_flags &VM_GROWSDOWN )) +if (vma && (vma->vm_flags &VM_GROWSDOWN )) { +vma_mark_locked (vma);return false; -if (prev && (prev->vm_flags &VM_GROWSUP )) + } +if (prev && (prev->vm_flags &VM_GROWSUP )) { +vma_mark_locked (prev);return false; + }return true; }
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 16/28 kernel/fork: assert no VMA readers during its destruction
In reply to Suren BaghdasaryanAssert there are no holders of VMA lock for reading when it is about to be destroyed.
--- include/linux/mm.h | 8 ++++++++ kernel/fork.c | 2 ++ 2 files changed, 10 insertions(+)
include/linux/mm.h +8 -0
@@ -676,6 +676,13 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos)VM_BUG_ON_VMA (vma->vm_lock_seq !=READ_ONCE (vma->vm_mm->mm_lock_seq), vma); } +static inline void vma_assert_no_reader (struct vm_area_struct *vma) +{ +VM_BUG_ON_VMA (rwsem_is_locked (&vma->lock) && + vma->vm_lock_seq !=READ_ONCE (vma->vm_mm->mm_lock_seq), + vma); +} +#else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock (struct vm_area_struct *vma) {}
@@ -685,6 +692,7 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma)static inline void vma_read_unlock (struct vm_area_struct *vma) {}static inline void vma_assert_locked (struct vm_area_struct *vma) {}static inline void vma_assert_write_locked (struct vm_area_struct *vma,int pos) {} +static inline void vma_assert_no_reader (struct vm_area_struct *vma) {}#endif /* CONFIG_PER_VMA_LOCK */
kernel/fork.c +2 -0
@@ -487,6 +487,8 @@ static void __vm_area_free(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); +/* The vma should either have no lock holders or be write-locked. */ +vma_assert_no_reader (vma);kmem_cache_free (vm_area_cachep, vma); }#endif
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 17/28 mm/mmap: prevent pagefault handler from racing with mmu_notifier registration
In reply to Suren BaghdasaryanPagefault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to mark all VMAs as locked and prevent this race with fault handlers that would hold VMA locks.
--- mm/mmap.c | 3 +++ 1 file changed, 3 insertions(+)
mm/mmap.c +3 -0
@@ -3538,6 +3538,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping)* hugetlb mapping); * - all i_mmap_rwsem locks; * - all anon_vma->rwseml +* - all vmas marked locked * * We can take all locks within these types randomly because the VM code * doesn't nest them and we protected from parallel mm_take_all_locks() by
@@ -3579,6 +3580,7 @@ int mm_take_all_locks(struct mm_struct *mm)if (vma->anon_vma)list_for_each_entry (avc, &vma->anon_vma_chain, same_vma)vm_lock_anon_vma (mm, avc->anon_vma); +vma_mark_locked (vma); }return 0 ;
@@ -3636,6 +3638,7 @@ void mm_drop_all_locks(struct mm_struct *mm)mmap_assert_write_locked (mm);BUG_ON (!mutex_is_locked (&mm_all_locks_mutex)); +vma_mark_unlocked_all (mm);for (vma = mm->mmap; vma; vma = vma->vm_next) {if (vma->anon_vma)list_for_each_entry (avc, &vma->anon_vma_chain, same_vma)
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 18/28 mm: add FAULT_FLAG_VMA_LOCK flag
In reply to Suren BaghdasaryanAdd a new flag to distinguish page faults handled under protection of per-vma lock.
--- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/mm.h +2 -1
@@ -466,7 +466,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" }/* * vm_fault is filled by the pagefault handler and passed to the vma's
include/linux/mm_types.h +1 -0
@@ -886,6 +886,7 @@ enum fault_flag {FAULT_FLAG_INTERRUPTIBLE =1 <<9 ,FAULT_FLAG_UNSHARE =1 <<10 ,FAULT_FLAG_ORIG_PTE_VALID =1 <<11 , + FAULT_FLAG_VMA_LOCK = 1 << 12, };typedef unsigned int __bitwise zap_flags_t ;
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 19/28 mm: disallow do_swap_page to handle page faults under VMA lock
In reply to Suren BaghdasaryanDue to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future.
--- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+)
mm/memory.c +5 -0
@@ -3738,6 +3738,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)vm_fault_t ret =0 ;void *shadow =NULL ; +if (vmf->flags &FAULT_FLAG_VMA_LOCK ) { + ret =VM_FAULT_RETRY ; + goto out; + } +if (!pte_unmap_same (vmf)) goto out;
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 20/28 mm: introduce per-VMA lock statistics
In reply to Suren BaghdasaryanAdd a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock.
--- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 8 ++++++++ mm/vmstat.c | 6 ++++++ 4 files changed, 26 insertions(+)
include/linux/vm_event_item.h +6 -0
@@ -150,6 +150,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,#ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +VMA_LOCK_SUCCESS , +VMA_LOCK_ABORT , +VMA_LOCK_RETRY , +VMA_LOCK_MISS , #endifNR_VM_EVENT_ITEMS };
include/linux/vmstat.h +6 -0
@@ -131,6 +131,12 @@ static inline void vm_events_fold_cpu(int cpu)#define count_vm_vmacache_event (x) do {} while (0)#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event (x) count_vm_event(x) +#else +#define count_vm_vma_lock_event (x) do {} while (0) +#endif +#define __count_zid_vm_events (item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta)
mm/Kconfig.debug +8 -0
@@ -207,3 +207,11 @@ config PTDUMP_DEBUGFS kernel. If in doubt, say N. + + +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + help + Statistics for per-vma locks. + If in doubt, say N.
mm/vmstat.c +6 -0
@@ -1411,6 +1411,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 21/28 mm: introduce find_and_lock_anon_vma to be used from arch-specific code
In reply to Suren BaghdasaryanIntroduce find_and_lock_anon_vma function to lookup and lock an anonymous VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results.
--- include/linux/mm.h | 3 +++ mm/memory.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+)
include/linux/mm.h +3 -0
@@ -684,6 +684,9 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) vma); } +struct vm_area_struct *find_and_lock_anon_vma (struct mm_struct *mm, +unsigned long address); +#else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock (struct vm_area_struct *vma) {}
mm/memory.c +45 -0
51 lines not on this page · raw message
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 22/28 x86/mm: try VMA lock-based page fault handling first
In reply to Suren BaghdasaryanAttempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails.
--- arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
arch/x86/mm/fault.c +36 -0
@@ -19,6 +19,7 @@#include <linux/uaccess.h> /* faulthandler_disabled() */ #include <linux/efi.h> /* efi_crash_gracefully_on_page_fault()*/ #include <linux/mm_types.h> +#include <linux/mm.h> /* find_and_lock_vma() */ #include <asm/cpufeature.h> /* boot_cpu_has, ... */ #include <asm/traps.h> /* dotraplinkage, ... */
@@ -1323,6 +1324,38 @@ void do_user_addr_fault(struct pt_regs *regs, }#endif +#ifdef CONFIG_PER_VMA_LOCK +if (!(flags &FAULT_FLAG_USER ) ||atomic_read (&mm->mm_users) ==1 ) + goto lock_mmap; + + vma =find_and_lock_anon_vma (mm, address); +if (!vma) + goto lock_mmap; + +if (unlikely (access_error (error_code, vma))) { +vma_read_unlock (vma); + goto lock_mmap; + } + fault =handle_mm_fault (vma, address, flags |FAULT_FLAG_VMA_LOCK , regs); +vma_read_unlock (vma); + +if (!(fault &VM_FAULT_RETRY )) { +count_vm_vma_lock_event (VMA_LOCK_SUCCESS ); + goto done; + } +count_vm_vma_lock_event (VMA_LOCK_RETRY ); + +/* Quick path to respond to signals */ +if (fault_signal_pending (fault, regs)) { +if (!user_mode (regs)) +kernelmode_fixup_or_oops (regs, error_code, address, +SIGBUS ,BUS_ADRERR , +ARCH_DEFAULT_PKEY ); +return ; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ +/* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception
@@ -1423,6 +1456,9 @@ void do_user_addr_fault(struct pt_regs *regs, }mmap_read_unlock (mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely (!(fault &VM_FAULT_ERROR )))return ;
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 23/28 x86/mm: define ARCH_SUPPORTS_PER_VMA_LOCK
In reply to Suren BaghdasaryanSet ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on this architecture.
--- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
@@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 24/28 arm64/mm: try VMA lock-based page fault handling first
In reply to Suren BaghdasaryanAttempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails.
--- arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
arch/arm64/mm/fault.c +36 -0
@@ -525,6 +525,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,unsigned long vm_flags;unsigned int mm_flags =FAULT_FLAG_DEFAULT ;unsigned long addr =untagged_addr (far); +#ifdef CONFIG_PER_VMA_LOCK +struct vm_area_struct *vma; +#endif if (kprobe_page_fault (regs, esr))return 0 ;
@@ -575,6 +578,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, addr); +#ifdef CONFIG_PER_VMA_LOCK +if (!(mm_flags &FAULT_FLAG_USER ) ||atomic_read (&mm->mm_users) ==1 ) + goto lock_mmap; + + vma =find_and_lock_anon_vma (mm, addr); +if (!vma) + goto lock_mmap; + +if (!(vma->vm_flags & vm_flags)) { +vma_read_unlock (vma); + goto lock_mmap; + } + fault =handle_mm_fault (vma, addr &PAGE_MASK , + mm_flags |FAULT_FLAG_VMA_LOCK , regs); +vma_read_unlock (vma); + +if (!(fault &VM_FAULT_RETRY )) { +count_vm_vma_lock_event (VMA_LOCK_SUCCESS ); + goto done; + } +count_vm_vma_lock_event (VMA_LOCK_RETRY ); + +/* Quick path to respond to signals */ +if (fault_signal_pending (fault, regs)) { +if (!user_mode (regs)) + goto no_context; +return 0 ; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code,
@@ -618,6 +651,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, }mmap_read_unlock (mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 25/28 arm64/mm: define ARCH_SUPPORTS_PER_VMA_LOCK
In reply to Suren BaghdasaryanSet ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on this architecture.
--- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/arm64/Kconfig +1 -0
@@ -93,6 +93,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 26/28 powerc/mm: try VMA lock-based page fault handling first
In reply to Suren BaghdasaryanFrom: Laurent Dufour <[email protected]>
Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails.
Copied from "x86/mm: try VMA lock-based page fault handling first"
--- arch/powerpc/mm/fault.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
arch/powerpc/mm/fault.c +41 -0
@@ -450,6 +450,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,if (is_exec) flags |=FAULT_FLAG_INSTRUCTION ; +#ifdef CONFIG_PER_VMA_LOCK +if (!(flags &FAULT_FLAG_USER ) ||atomic_read (&mm->mm_users) ==1 ) + goto lock_mmap; + + vma =find_and_lock_anon_vma (mm, address); +if (!vma) + goto lock_mmap; + +if (unlikely (access_pkey_error (is_write, is_exec, + (error_code &DSISR_KEYFAULT ), vma))) { +int rc =bad_access_pkey (regs, address, vma); + +vma_read_unlock (vma); +return rc; + } + +if (unlikely (access_error (is_write, is_exec, vma))) { +int rc =bad_access (regs, address); + +vma_read_unlock (vma); +return rc; + } + + fault =handle_mm_fault (vma, address, flags |FAULT_FLAG_VMA_LOCK , regs); +vma_read_unlock (vma); + +if (!(fault &VM_FAULT_RETRY )) { +count_vm_vma_lock_event (VMA_LOCK_SUCCESS ); + goto done; + } +count_vm_vma_lock_event (VMA_LOCK_RETRY ); + +if (fault_signal_pending (fault, regs)) +return user_mode (regs) ?0 :SIGBUS ; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ +/* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an
@@ -526,6 +564,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,mmap_read_unlock (current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely (fault &VM_FAULT_ERROR ))return mm_fault_error (regs, address, fault);
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 27/28 powerpc/mm: define ARCH_SUPPORTS_PER_VMA_LOCK
In reply to Suren BaghdasaryanFrom: Laurent Dufour <[email protected]>
Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on powernv and pseries.
It may be use on the other platforms but I can't test that seriously.
--- arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 2 files changed, 2 insertions(+)
arch/powerpc/platforms/powernv/Kconfig +1 -0
@@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD
arch/powerpc/platforms/pseries/Kconfig +1 -0
@@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT_SPINLOCKS
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Suren Baghdasaryan
maintainerRFC PATCH 28/28 kernel/fork: throttle call_rcu() calls in vm_area_free
In reply to Suren Baghdasaryancall_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. To minimize that impact, place VMAs into a list and free them in groups using one call_rcu() call per group.
--- include/linux/mm.h | 1 + include/linux/mm_types.h | 11 ++++++- kernel/fork.c | 68 +++++++++++++++++++++++++++++++++++----- mm/init-mm.c | 3 ++ mm/mmap.c | 1 + 5 files changed, 75 insertions(+), 9 deletions(-)
include/linux/mm.h +1 -0
@@ -249,6 +249,7 @@ void setup_initial_init_mm(void *start_code, void *end_code,struct vm_area_struct *vm_area_alloc (struct mm_struct *);struct vm_area_struct *vm_area_dup (struct vm_area_struct *);void vm_area_free (struct vm_area_struct *); +void drain_free_vmas (struct mm_struct *mm);#ifndef CONFIG_MMU extern struct rb_root nommu_region_tree;
include/linux/mm_types.h +10 -1
@@ -412,7 +412,11 @@ struct vm_area_struct {struct vm_area_struct *vm_next, *vm_prev; };#ifdef CONFIG_PER_VMA_LOCK -struct rcu_head vm_rcu;/* Used for deferred freeing. */ +struct { +struct list_head vm_free_list; +/* Used for deferred freeing. */ +struct rcu_head vm_rcu; + };#endif };
@@ -573,6 +577,11 @@ struct mm_struct {*/ #ifdef CONFIG_PER_VMA_LOCK int mm_lock_seq; +struct { +struct list_head head; +spinlock_t lock; +int size; + } vma_free_list;#endif
kernel/fork.c +60 -8
92 lines not on this page · raw message
mm/init-mm.c +3 -0
@@ -39,6 +39,9 @@ struct mm_struct init_mm = { .mmlist = LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK .mm_lock_seq =0 , + .vma_free_list.head = LIST_HEAD_INIT(init_mm.vma_free_list.head), + .vma_free_list.lock = __SPIN_LOCK_UNLOCKED(init_mm.vma_free_list.lock), + .vma_free_list.size = 0, #endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE,
mm/mmap.c +1 -0
@@ -3149,6 +3149,7 @@ void exit_mmap(struct mm_struct *mm) } mm->mmap =NULL ;mmap_write_unlock (mm); +drain_free_vmas (mm);vm_unacct_memory (nr_accounted); }
Signature · 1 line
2.37.2.672.g94769d06f0-goog