make vma locking more obvious
Subsystems (25)
Suren Baghdasaryan
reviewerv2 cover make vma locking more obvious
During recent vma locking patch reviews Linus and Jann Horn noted a number of issues with vma locking and suggested improvements:
1. walk_page_range() does not have ability to write-lock a vma during the walk when it's done under mmap_write_lock. For example s390_reset_cmma().
2. Vma locking is hidden inside vm_flags modifiers and is hard to follow. Suggestion is to change vm_flags_reset{_once} to assert that vma is write-locked and require an explicit locking.
3. Same issue with vma_prepare() hiding vma locking.
4. In userfaultfd vm_flags are modified after vma->vm_userfaultfd_ctx and page faults can operate on a context while it's changed.
5. do_brk_flags() and __install_special_mapping() not locking a newly created vma before adding it into the mm. While not strictly a problem, this is fragile if vma is modified after insertion, as in the mmap_region() case which was recently fixed. Suggestion is to always lock a new vma before inserting it and making it visible to page faults.
6. vma_assert_write_locked() for CONFIG_PER_VMA_LOCK=n would benefit from being mmap_assert_write_locked() instead of no-op and then any place which operates on a vma and calls mmap_assert_write_locked() can be converted into vma_assert_write_locked().
I CC'ed stable only on the first patch because others are cleanups and the bug in userfaultfd does not affect stable (lock_vma_under_rcu prevents uffds from being handled under vma lock protection). However I would be happy if the whole series is merged into stable 6.4 since it makes vma locking more maintainable.
The patches apply cleanly over Linus' ToT and will conflict when applied over mm-unstable due to missing [1]. The conflict can be easily resolved by ignoring conflicting deletions but probably simpler to take [1] into mm-unstable and avoid later conflict.
[1] commit 6c21e066f925 ("mm/mempolicy: Take VMA lock before replacing policy")
Changes since v1:
- replace walk_page_range() parameter with mm_walk_ops.walk_lock, per Linus
- introduced page_walk_lock enum to allow different locking modes during a walk, per Linus
- added Liam's Reviewed-by
Suren Baghdasaryan (6):
mm: enable page walking API to lock vmas during the walk
mm: for !CONFIG_PER_VMA_LOCK equate write lock assertion for vma and
mmap
mm: replace mmap with vma write lock assertions when operating on a
vma
mm: lock vma explicitly before doing vm_flags_reset and
vm_flags_reset_once
mm: always lock new vma before inserting into vma tree
mm: move vma locking out of vma_prepare
arch/powerpc/kvm/book3s_hv_uvmem.c | 1 + arch/powerpc/mm/book3s64/subpage_prot.c | 1 + arch/riscv/mm/pageattr.c | 1 + arch/s390/mm/gmap.c | 5 ++++ drivers/infiniband/hw/hfi1/file_ops.c | 1 + fs/proc/task_mmu.c | 5 ++++ fs/userfaultfd.c | 6 +++++ include/linux/mm.h | 13 ++++++--- include/linux/pagewalk.h | 11 ++++++++ mm/damon/vaddr.c | 2 ++ mm/hmm.c | 1 + mm/hugetlb.c | 2 +- mm/khugepaged.c | 5 ++-- mm/ksm.c | 25 ++++++++++------- mm/madvise.c | 8 +++--- mm/memcontrol.c | 2 ++ mm/memory-failure.c | 1 + mm/memory.c | 2 +- mm/mempolicy.c | 22 +++++++++------ mm/migrate_device.c | 1 + mm/mincore.c | 1 + mm/mlock.c | 4 ++- mm/mmap.c | 29 +++++++++++++------- mm/mprotect.c | 2 ++ mm/pagewalk.c | 36 ++++++++++++++++++++++--- mm/vmscan.c | 1 + 26 files changed, 146 insertions(+), 42 deletions(-)
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
Suren Baghdasaryan
reviewerPATCH v2 1/6 mm: enable page walking API to lock vmas during the walk
walk_page_range() and friends often operate under write-locked mmap_lock. With introduction of vma locks, the vmas have to be locked as well during such walks to prevent concurrent page faults in these areas. Add an additional member to mm_walk_ops to indicate locking requirements for the walk.
--- arch/powerpc/mm/book3s64/subpage_prot.c | 1 + arch/riscv/mm/pageattr.c | 1 + arch/s390/mm/gmap.c | 5 ++++ fs/proc/task_mmu.c | 5 ++++ include/linux/pagewalk.h | 11 ++++++++ mm/damon/vaddr.c | 2 ++ mm/hmm.c | 1 + mm/ksm.c | 25 ++++++++++------- mm/madvise.c | 3 +++ mm/memcontrol.c | 2 ++ mm/memory-failure.c | 1 + mm/mempolicy.c | 22 +++++++++------ mm/migrate_device.c | 1 + mm/mincore.c | 1 + mm/mlock.c | 1 + mm/mprotect.c | 1 + mm/pagewalk.c | 36 ++++++++++++++++++++++--- mm/vmscan.c | 1 + 18 files changed, 100 insertions(+), 20 deletions(-)
arch/powerpc/mm/book3s64/subpage_prot.c +1 -0
@@ -145,6 +145,7 @@ static int subpage_walk_pmd_entry(pmd_t *pmd, unsigned long addr,static const struct mm_walk_ops subpage_walk_ops = { .pmd_entry = subpage_walk_pmd_entry, + .walk_lock =PGWALK_WRLOCK_VERIFY , }; static void subpage_mark_vma_nohuge(struct mm_struct *mm, unsigned long addr,
arch/riscv/mm/pageattr.c +1 -0
@@ -102,6 +102,7 @@ static const struct mm_walk_ops pageattr_ops = { .pmd_entry = pageattr_pmd_entry, .pte_entry = pageattr_pte_entry, .pte_hole = pageattr_pte_hole, + .walk_lock = PGWALK_RDLOCK, }; static int __set_memory(unsigned long addr, int numpages, pgprot_t set_mask,
arch/s390/mm/gmap.c +5 -0
@@ -2514,6 +2514,7 @@ static int thp_split_walk_pmd_entry(pmd_t *pmd, unsigned long addr,static const struct mm_walk_ops thp_split_walk_ops = { .pmd_entry = thp_split_walk_pmd_entry, + .walk_lock =PGWALK_WRLOCK_VERIFY , };static inline void thp_split_mm (struct mm_struct *mm)
@@ -2565,6 +2566,7 @@ static int __zap_zero_pages(pmd_t *pmd, unsigned long start,static const struct mm_walk_ops zap_zero_walk_ops = { .pmd_entry = __zap_zero_pages, + .walk_lock =PGWALK_WRLOCK , };/*
@@ -2655,6 +2657,7 @@ static const struct mm_walk_ops enable_skey_walk_ops = { .hugetlb_entry = __s390_enable_skey_hugetlb, .pte_entry = __s390_enable_skey_pte, .pmd_entry = __s390_enable_skey_pmd, + .walk_lock = PGWALK_WRLOCK, };int s390_enable_skey (void )
@@ -2692,6 +2695,7 @@ static int __s390_reset_cmma(pte_t *pte, unsigned long addr,static const struct mm_walk_ops reset_cmma_walk_ops = { .pte_entry = __s390_reset_cmma, + .walk_lock =PGWALK_WRLOCK , };void s390_reset_cmma (struct mm_struct *mm)
@@ -2728,6 +2732,7 @@ static int s390_gather_pages(pte_t *ptep, unsigned long addr,static const struct mm_walk_ops gather_pages_ops = { .pte_entry = s390_gather_pages, + .walk_lock =PGWALK_RDLOCK , };/*
fs/proc/task_mmu.c +5 -0
@@ -758,12 +758,14 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,static const struct mm_walk_ops smaps_walk_ops = { .pmd_entry = smaps_pte_range, .hugetlb_entry = smaps_hugetlb_range, + .walk_lock =PGWALK_RDLOCK , };static const struct mm_walk_ops smaps_shmem_walk_ops = { .pmd_entry = smaps_pte_range, .hugetlb_entry = smaps_hugetlb_range, .pte_hole = smaps_pte_hole, + .walk_lock =PGWALK_RDLOCK , };/*
@@ -1245,6 +1247,7 @@ static int clear_refs_test_walk(unsigned long start, unsigned long end,static const struct mm_walk_ops clear_refs_walk_ops = { .pmd_entry = clear_refs_pte_range, .test_walk = clear_refs_test_walk, + .walk_lock =PGWALK_WRLOCK , }; static ssize_t clear_refs_write(struct file *file, const char __user *buf,
@@ -1622,6 +1625,7 @@ static const struct mm_walk_ops pagemap_ops = { .pmd_entry = pagemap_pmd_range, .pte_hole = pagemap_pte_hole, .hugetlb_entry = pagemap_hugetlb_range, + .walk_lock = PGWALK_RDLOCK, };/*
@@ -1935,6 +1939,7 @@ static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask,static const struct mm_walk_ops show_numa_ops = { .hugetlb_entry = gather_hugetlb_stats, .pmd_entry = gather_pte_stats, + .walk_lock =PGWALK_RDLOCK , };/*
include/linux/pagewalk.h +11 -0
@@ -6,6 +6,16 @@struct mm_walk ; +/* Locking requirement during a page walk. */ +enum page_walk_lock { +/* mmap_lock should be locked for read to stabilize the vma tree */ +PGWALK_RDLOCK =0 , +/* vma will be write-locked during the walk */ +PGWALK_WRLOCK =1 , +/* vma is expected to be already write-locked during the walk */ +PGWALK_WRLOCK_VERIFY =2 , +}; +/** * struct mm_walk_ops - callbacks for walk_page_range * @pgd_entry: if set, called for each non-empty PGD (top-level) entry
@@ -66,6 +76,7 @@ struct mm_walk_ops {int (*pre_vma)(unsigned long start,unsigned long end,struct mm_walk *walk);void (*post_vma)(struct mm_walk *walk); +enum page_walk_lock walk_lock; };/*
mm/damon/vaddr.c +2 -0
@@ -386,6 +386,7 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask,static const struct mm_walk_ops damon_mkold_ops = { .pmd_entry = damon_mkold_pmd_entry, .hugetlb_entry = damon_mkold_hugetlb_entry, + .walk_lock =PGWALK_RDLOCK , };static void damon_va_mkold (struct mm_struct *mm,unsigned long addr)
@@ -525,6 +526,7 @@ static int damon_young_hugetlb_entry(pte_t *pte, unsigned long hmask,static const struct mm_walk_ops damon_young_ops = { .pmd_entry = damon_young_pmd_entry, .hugetlb_entry = damon_young_hugetlb_entry, + .walk_lock =PGWALK_RDLOCK , }; static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
mm/hmm.c +1 -0
@@ -562,6 +562,7 @@ static const struct mm_walk_ops hmm_walk_ops = { .pte_hole = hmm_vma_walk_hole, .hugetlb_entry = hmm_vma_walk_hugetlb_entry, .test_walk = hmm_vma_walk_test, + .walk_lock = PGWALK_RDLOCK, };/**
mm/ksm.c +16 -9
@@ -455,6 +455,12 @@ static int break_ksm_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long nexstatic const struct mm_walk_ops break_ksm_ops = { .pmd_entry = break_ksm_pmd_entry, + .walk_lock =PGWALK_RDLOCK , +}; + +static const struct mm_walk_ops break_ksm_lock_vma_ops = { + .pmd_entry = break_ksm_pmd_entry, + .walk_lock =PGWALK_WRLOCK , };/*
@@ -470,16 +476,17 @@ static const struct mm_walk_ops break_ksm_ops = {* of the process that owns 'vma'. We also do not want to enforce * protection keys here anyway. */ -static int break_ksm (struct vm_area_struct *vma,unsigned long addr) +static int break_ksm (struct vm_area_struct *vma,unsigned long addr,bool lock_vma) {vm_fault_t ret =0 ; +const struct mm_walk_ops *ops = lock_vma ? + &break_ksm_lock_vma_ops : &break_ksm_ops; do { int ksm_page; cond_resched(); - ksm_page = walk_page_range_vma(vma, addr, addr + 1, - &break_ksm_ops, NULL); + ksm_page = walk_page_range_vma(vma, addr, addr + 1, ops, NULL); if (WARN_ON_ONCE(ksm_page < 0)) return ksm_page; if (!ksm_page)
@@ -565,7 +572,7 @@ static void break_cow(struct ksm_rmap_item *rmap_item)mmap_read_lock (mm); vma =find_mergeable_vma (mm, addr);if (vma) -break_ksm (vma, addr); +break_ksm (vma, addr, false);mmap_read_unlock (mm); }
@@ -871,7 +878,7 @@ static void remove_trailing_rmap_items(struct ksm_rmap_item **rmap_list)* in cmp_and_merge_page on one of the rmap_items we would be removing. */ static int unmerge_ksm_pages (struct vm_area_struct *vma, -unsigned long start,unsigned long end) +unsigned long start,unsigned long end,bool lock_vma) {unsigned long addr;int err =0 ;
@@ -882,7 +889,7 @@ static int unmerge_ksm_pages(struct vm_area_struct *vma,if (signal_pending (current)) err = -ERESTARTSYS ;else - err =break_ksm (vma, addr); + err =break_ksm (vma, addr, lock_vma); }return err; }
@@ -1029,7 +1036,7 @@ static int unmerge_and_remove_all_rmap_items(void)if (!(vma->vm_flags &VM_MERGEABLE ) || !vma->anon_vma)continue ; err =unmerge_ksm_pages (vma, - vma->vm_start, vma->vm_end); + vma->vm_start, vma->vm_end, false);if (err) goto error; }
@@ -2530,7 +2537,7 @@ static int __ksm_del_vma(struct vm_area_struct *vma)return 0 ;if (vma->anon_vma) { - err =unmerge_ksm_pages (vma, vma->vm_start, vma->vm_end); + err =unmerge_ksm_pages (vma, vma->vm_start, vma->vm_end, true);if (err)return err; }
@@ -2668,7 +2675,7 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start,return 0 ;/* just ignore the advice */ if (vma->anon_vma) { - err =unmerge_ksm_pages (vma, start, end); + err =unmerge_ksm_pages (vma, start, end, true);if (err)return err; }
mm/madvise.c +3 -0
@@ -233,6 +233,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,static const struct mm_walk_ops swapin_walk_ops = { .pmd_entry = swapin_walk_pmd_entry, + .walk_lock =PGWALK_RDLOCK , }; static void shmem_swapin_range(struct vm_area_struct *vma,
@@ -534,6 +535,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,static const struct mm_walk_ops cold_walk_ops = { .pmd_entry = madvise_cold_or_pageout_pte_range, + .walk_lock =PGWALK_RDLOCK , }; static void madvise_cold_page_range(struct mmu_gather *tlb,
@@ -757,6 +759,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,static const struct mm_walk_ops madvise_free_walk_ops = { .pmd_entry = madvise_free_pte_range, + .walk_lock =PGWALK_RDLOCK , }; static int madvise_free_single_vma(struct vm_area_struct *vma,
mm/memcontrol.c +2 -0
@@ -6024,6 +6024,7 @@ static int mem_cgroup_count_precharge_pte_range(pmd_t *pmd,static const struct mm_walk_ops precharge_walk_ops = { .pmd_entry = mem_cgroup_count_precharge_pte_range, + .walk_lock =PGWALK_RDLOCK , };static unsigned long mem_cgroup_count_precharge (struct mm_struct *mm)
@@ -6303,6 +6304,7 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd,static const struct mm_walk_ops charge_walk_ops = { .pmd_entry = mem_cgroup_move_charge_pte_range, + .walk_lock =PGWALK_RDLOCK , };static void mem_cgroup_move_charge (void )
mm/memory-failure.c +1 -0
@@ -831,6 +831,7 @@ static int hwpoison_hugetlb_range(pte_t *ptep, unsigned long hmask,static const struct mm_walk_ops hwp_walk_ops = { .pmd_entry = hwpoison_pte_range, .hugetlb_entry = hwpoison_hugetlb_range, + .walk_lock =PGWALK_RDLOCK , };/*
mm/mempolicy.c +14 -8
@@ -718,6 +718,14 @@ static const struct mm_walk_ops queue_pages_walk_ops = { .hugetlb_entry = queue_folios_hugetlb, .pmd_entry = queue_folios_pte_range, .test_walk = queue_pages_test_walk, + .walk_lock = PGWALK_RDLOCK, +}; + +static const struct mm_walk_ops queue_pages_lock_vma_walk_ops = { + .hugetlb_entry = queue_folios_hugetlb, + .pmd_entry = queue_folios_pte_range, + .test_walk = queue_pages_test_walk, + .walk_lock =PGWALK_WRLOCK , };/*
@@ -738,7 +746,7 @@ static const struct mm_walk_ops queue_pages_walk_ops = {static int queue_pages_range (struct mm_struct *mm,unsigned long start,unsigned long end,nodemask_t *nodes,unsigned long flags, -struct list_head *pagelist) +struct list_head *pagelist,bool lock_vma) {int err;struct queue_pages qp = {
@@ -749,8 +757,10 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, .end = end, .first = NULL, }; +const struct mm_walk_ops *ops = lock_vma ? + &queue_pages_lock_vma_walk_ops : &queue_pages_walk_ops; - err =walk_page_range (mm, start, end, &queue_pages_walk_ops, &qp); + err =walk_page_range (mm, start, end, ops, &qp);if (!qp.first)/* whole range in hole */
@@ -1078,7 +1088,7 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, vma =find_vma (mm,0 );VM_BUG_ON (!(flags & (MPOL_MF_MOVE |MPOL_MF_MOVE_ALL )));queue_pages_range (mm, vma->vm_start, mm->task_size, &nmask, - flags |MPOL_MF_DISCONTIG_OK , &pagelist); + flags |MPOL_MF_DISCONTIG_OK , &pagelist, false);if (!list_empty (&pagelist)) { err = migrate_pages(&pagelist, alloc_migration_target, NULL,
@@ -1321,12 +1331,8 @@ static long do_mbind(unsigned long start, unsigned long len,* Lock the VMAs before scanning for pages to migrate, to ensure we don't * miss a concurrently inserted page. */ -vma_iter_init (&vmi, mm, start); -for_each_vma_range (vmi, vma, end) -vma_start_write (vma); - ret =queue_pages_range (mm, start, end, nmask, - flags |MPOL_MF_INVERT , &pagelist); + flags |MPOL_MF_INVERT , &pagelist, true);if (ret <0 ) { err = ret;
mm/migrate_device.c +1 -0
@@ -279,6 +279,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,static const struct mm_walk_ops migrate_vma_walk_ops = { .pmd_entry = migrate_vma_collect_pmd, .pte_hole = migrate_vma_collect_hole, + .walk_lock =PGWALK_RDLOCK , };/*
mm/mincore.c +1 -0
@@ -176,6 +176,7 @@ static const struct mm_walk_ops mincore_walk_ops = { .pmd_entry = mincore_pte_range, .pte_hole = mincore_unmapped_range, .hugetlb_entry = mincore_hugetlb, + .walk_lock = PGWALK_RDLOCK, };/*
mm/mlock.c +1 -0
@@ -371,6 +371,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, {static const struct mm_walk_ops mlock_walk_ops = { .pmd_entry = mlock_pte_range, + .walk_lock =PGWALK_WRLOCK_VERIFY , };/*
mm/mprotect.c +1 -0
@@ -568,6 +568,7 @@ static const struct mm_walk_ops prot_none_walk_ops = { .pte_entry = prot_none_pte_entry, .hugetlb_entry = prot_none_hugetlb_entry, .test_walk = prot_none_test, + .walk_lock = PGWALK_WRLOCK, };int
mm/pagewalk.c +33 -3
@@ -400,6 +400,33 @@ static int __walk_page_range(unsigned long start, unsigned long end,return err; } +static inline void process_mm_walk_lock (struct mm_struct *mm, +enum page_walk_lock walk_lock) +{ +if (walk_lock ==PGWALK_RDLOCK ) +mmap_assert_locked (mm); +else +mmap_assert_write_locked (mm); +} + +static inline void process_vma_walk_lock (struct vm_area_struct *vma, +enum page_walk_lock walk_lock) +{ +#ifdef CONFIG_PER_VMA_LOCK +switch (walk_lock) { +case PGWALK_WRLOCK : +vma_start_write (vma); +break ; +case PGWALK_WRLOCK_VERIFY : +vma_assert_write_locked (vma); +break ; +case PGWALK_RDLOCK : +/* PGWALK_RDLOCK is handled by process_mm_walk_lock */ +break ; + } +#endif +} +/** * walk_page_range - walk page table with caller specific callbacks * @mm: mm_struct representing the target process of page table walk
@@ -459,7 +486,7 @@ int walk_page_range(struct mm_struct *mm, unsigned long start,if (!walk.mm)return -EINVAL ; -mmap_assert_locked (walk.mm); +process_mm_walk_lock (walk.mm, ops->walk_lock); vma =find_vma (walk.mm, start); do {
@@ -474,6 +501,7 @@ int walk_page_range(struct mm_struct *mm, unsigned long start,if (ops->pte_hole) err = ops->pte_hole (start, next,-1 , &walk); }else {/* inside vma */ +process_vma_walk_lock (vma, ops->walk_lock); walk.vma = vma; next =min (end, vma->vm_end); vma =find_vma (mm, vma->vm_end);
@@ -549,7 +577,8 @@ int walk_page_range_vma(struct vm_area_struct *vma, unsigned long start,if (start < vma->vm_start || end > vma->vm_end)return -EINVAL ; -mmap_assert_locked (walk.mm); +process_mm_walk_lock (walk.mm, ops->walk_lock); +process_vma_walk_lock (vma, ops->walk_lock);return __walk_page_range (start, end, &walk); }
@@ -566,7 +595,8 @@ int walk_page_vma(struct vm_area_struct *vma, const struct mm_walk_ops *ops,if (!walk.mm)return -EINVAL ; -mmap_assert_locked (walk.mm); +process_mm_walk_lock (walk.mm, ops->walk_lock); +process_vma_walk_lock (vma, ops->walk_lock);return __walk_page_range (vma->vm_start, vma->vm_end, &walk); }
mm/vmscan.c +1 -0
@@ -4284,6 +4284,7 @@ static void walk_mm(struct lruvec *lruvec, struct mm_struct *mm, struct lru_gen_static const struct mm_walk_ops mm_walk_ops = { .test_walk = should_skip_vma, .p4d_entry = walk_pud_range, + .walk_lock =PGWALK_RDLOCK , };int err;
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
Suren Baghdasaryan
reviewerPATCH v2 2/6 mm: for !CONFIG_PER_VMA_LOCK equate write lock assertion for vma and mmap
In reply to Suren BaghdasaryanWhen CONFIG_PER_VMA_LOCK=n, vma_assert_write_locked() should be equivalent to mmap_assert_write_locked().
--- include/linux/mm.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/mm.h +2 -1
@@ -750,7 +750,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {}static inline void vma_start_write (struct vm_area_struct *vma) {}static inline bool vma_try_start_write (struct vm_area_struct *vma) {return true; } -static inline void vma_assert_write_locked (struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked (struct vm_area_struct *vma) + {mmap_assert_write_locked (vma->vm_mm); }static inline void vma_mark_detached (struct vm_area_struct *vma,bool detached) {}
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
kernel test robot
build botRe: [PATCH v2 2/6] mm: for !CONFIG_PER_VMA_LOCK equate write lock assertion for vma and mmap
FYI: kernel test robot notices the stable kernel rule is not satisfied.
Hi,
Thanks for your patch.
FYI: kernel test robot notices the stable kernel rule is not satisfied.
Rule: 'Cc: [email protected]' or 'commit <sha1> upstream.'
Subject: [PATCH v2 2/6] mm: for !CONFIG_PER_VMA_LOCK equate write lock assertion for vma and mmap
The check is based on https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v2 2/6] mm: for !CONFIG_PER_VMA_LOCK equate write lock assertion for vma and mmap
In reply to Suren BaghdasaryanQuoted 5 lines · Suren Baghdasaryan
When
CONFIG_PER_VMA_LOCK=n,vma_assert_write_locked()should be equivalent tommap_assert_write_locked().Suggested-by Jann Horn <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>
Quoted 21 lines
--- include/linux/mm.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/mm.h +2
-1@@ -750,7 +750,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {}static inline void vma_start_write (struct vm_area_struct *vma) {}static inline bool vma_try_start_write (struct vm_area_struct *vma) {return true; } -static inline void vma_assert_write_locked (struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked (struct vm_area_struct *vma) + {mmap_assert_write_locked (vma->vm_mm); }static inline void vma_mark_detached (struct vm_area_struct *vma,bool detached) {}Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
Suren Baghdasaryan
reviewerPATCH v2 3/6 mm: replace mmap with vma write lock assertions when operating on a vma
In reply to Suren BaghdasaryanVma write lock assertion always includes mmap write lock assertion and additional vma lock checks when per-VMA locks are enabled. Replace weaker mmap_assert_write_locked() assertions with stronger vma_assert_write_locked() ones when we are operating on a vma which is expected to be locked.
--- mm/hugetlb.c | 2 +- mm/khugepaged.c | 5 +++-- mm/memory.c | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-)
mm/hugetlb.c +1 -1
@@ -5028,7 +5028,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, src_vma->vm_start, src_vma->vm_end);mmu_notifier_invalidate_range_start (&range); -mmap_assert_write_locked (src); +vma_assert_write_locked (src_vma);raw_write_seqcount_begin (&src->write_protect_seq); }else {/*
mm/khugepaged.c +3 -2
@@ -1495,7 +1495,7 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, };VM_BUG_ON (!PageTransHuge (hpage)); -mmap_assert_write_locked (vma->vm_mm); +vma_assert_write_locked (vma);if (do_set_pmd (&vmf, hpage))return SCAN_FAIL ;
@@ -1525,7 +1525,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *vpmd_t pmd;struct mmu_notifier_range range; -mmap_assert_write_locked (mm); +vma_assert_write_locked (vma);if (vma->vm_file)lockdep_assert_held_write (&vma->vm_file->f_mapping->i_mmap_rwsem);/*
@@ -1570,6 +1570,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,int count =0 , result =SCAN_FAIL ;int i; +/* Ensure vma can't change, it will be locked below after checks */ mmap_assert_write_locked (mm);/* Fast check before locking page if already PMD-mapped */
mm/memory.c +1 -1
@@ -1312,7 +1312,7 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)* Use the raw variant of the seqcount_t write API to avoid * lockdep complaining about preemptibility. */ -mmap_assert_write_locked (src_mm); +vma_assert_write_locked (src_vma);raw_write_seqcount_begin (&src_mm->write_protect_seq); }
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
Re: [PATCH v2 3/6] mm: replace mmap with vma write lock assertions when operating on a vma
Quoted 8 lines · Suren Baghdasaryan
Vma write lock assertion always includes mmap write lock assertion and additional vma lock checks when per-VMA locks are enabled. Replace weaker
mmap_assert_write_locked()assertions with strongervma_assert_write_locked()ones when we are operating on a vma which is expected to be locked.Suggested-by Jann Horn <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>
Quoted 65 lines
--- mm/hugetlb.c | 2 +- mm/khugepaged.c | 5 +++-- mm/memory.c | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-)mm/hugetlb.c +1
-1@@ -5028,7 +5028,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, src_vma->vm_start, src_vma->vm_end);mmu_notifier_invalidate_range_start (&range); -mmap_assert_write_locked (src); +vma_assert_write_locked (src_vma);raw_write_seqcount_begin (&src->write_protect_seq); }else {/* mm/khugepaged.c +3
-2@@ -1495,7 +1495,7 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, };VM_BUG_ON (!PageTransHuge (hpage)); -mmap_assert_write_locked (vma->vm_mm); +vma_assert_write_locked (vma);if (do_set_pmd (&vmf, hpage))return SCAN_FAIL ;@@ -1525,7 +1525,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *vpmd_t pmd;struct mmu_notifier_range range; -mmap_assert_write_locked (mm); +vma_assert_write_locked (vma);if (vma->vm_file)lockdep_assert_held_write (&vma->vm_file->f_mapping->i_mmap_rwsem);/* @@ -1570,6 +1570,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,int count =0 , result =SCAN_FAIL ;int i; +/* Ensure vma can't change, it will be locked below after checks */ mmap_assert_write_locked (mm);/* Fast check before locking page if already PMD-mapped */ mm/memory.c +1
-1@@ -1312,7 +1312,7 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)* Use the raw variant of the seqcount_t write API to avoid * lockdep complaining about preemptibility. */ -mmap_assert_write_locked (src_mm); +vma_assert_write_locked (src_vma);raw_write_seqcount_begin (&src_mm->write_protect_seq); }Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
Suren Baghdasaryan
reviewerPATCH v2 4/6 mm: lock vma explicitly before doing vm_flags_reset and vm_flags_reset_once
In reply to Suren BaghdasaryanImplicit vma locking inside vm_flags_reset() and vm_flags_reset_once() is not obvious and makes it hard to understand where vma locking is happening. Also in some cases (like in dup_userfaultfd()) vma should be locked earlier than vma_flags modification. To make locking more visible, change these functions to assert that the vma write lock is taken and explicitly lock the vma beforehand. Fix userfaultfd functions which should lock the vma earlier.
--- arch/powerpc/kvm/book3s_hv_uvmem.c | 1 + drivers/infiniband/hw/hfi1/file_ops.c | 1 + fs/userfaultfd.c | 6 ++++++ include/linux/mm.h | 10 +++++++--- mm/madvise.c | 5 ++--- mm/mlock.c | 3 ++- mm/mprotect.c | 1 + 7 files changed, 20 insertions(+), 7 deletions(-)
arch/powerpc/kvm/book3s_hv_uvmem.c +1 -0
@@ -410,6 +410,7 @@ static int kvmppc_memslot_page_merge(struct kvm *kvm, ret =H_STATE ;break ; } +vma_start_write (vma);/* Copy vm_flags to avoid partial modifications in ksm_madvise */ vm_flags = vma->vm_flags; ret = ksm_madvise(vma, vma->vm_start, vma->vm_end,
drivers/infiniband/hw/hfi1/file_ops.c +1 -0
@@ -344,6 +344,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) goto done; } +vma_start_write (vma);/* * vm_pgoff is used as a buffer selector cookie. Always mmap from * the beginning.
fs/userfaultfd.c +6 -0
@@ -667,6 +667,7 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx,mmap_write_lock (mm);for_each_vma (vmi, vma) {if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { +vma_start_write (vma); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ;userfaultfd_set_vm_flags (vma, vma->vm_flags & ~__VM_UFFD_FLAGS);
@@ -702,6 +703,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx;if (!octx || !(octx->features &UFFD_FEATURE_EVENT_FORK )) { +vma_start_write (vma); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ;userfaultfd_set_vm_flags (vma, vma->vm_flags & ~__VM_UFFD_FLAGS);return 0 ;
@@ -783,6 +785,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma,atomic_inc (&ctx->mmap_changing); }else {/* Drop uffd context if remap feature not enabled */ +vma_start_write (vma); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ;userfaultfd_set_vm_flags (vma, vma->vm_flags & ~__VM_UFFD_FLAGS); }
@@ -940,6 +943,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) prev = vma; } +vma_start_write (vma);userfaultfd_set_vm_flags (vma, new_flags); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ; }
@@ -1502,6 +1506,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,* the next vma was merged into the current one and * the current one has not been updated yet. */ +vma_start_write (vma);userfaultfd_set_vm_flags (vma, new_flags); vma->vm_userfaultfd_ctx.ctx = ctx;
@@ -1685,6 +1690,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,* the next vma was merged into the current one and * the current one has not been updated yet. */ +vma_start_write (vma);userfaultfd_set_vm_flags (vma, new_flags); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ;
include/linux/mm.h +7 -3
@@ -780,18 +780,22 @@ static inline void vm_flags_init(struct vm_area_struct *vma,ACCESS_PRIVATE (vma, __vm_flags) = flags; } -/* Use when VMA is part of the VMA tree and modifications need coordination */ +/* +* Use when VMA is part of the VMA tree and modifications need coordination +* Note: vm_flags_reset and vm_flags_reset_once do not lock the vma and +* it should be locked explicitly beforehand. +*/ static inline void vm_flags_reset (struct vm_area_struct *vma,vm_flags_t flags) { -vma_start_write (vma); +vma_assert_write_locked (vma);vm_flags_init (vma, flags); }static inline void vm_flags_reset_once (struct vm_area_struct *vma,vm_flags_t flags) { -vma_start_write (vma); +vma_assert_write_locked (vma);WRITE_ONCE (ACCESS_PRIVATE (vma, __vm_flags), flags); }
mm/madvise.c +2 -3
@@ -173,9 +173,8 @@ static int madvise_update_vma(struct vm_area_struct *vma, } success: -/* -* vm_flags is protected by the mmap_lock held in write mode. -*/ +/* vm_flags is protected by the mmap_lock held in write mode. */ +vma_start_write (vma);vm_flags_reset (vma, new_flags);if (!vma->vm_file ||vma_is_anon_shmem (vma)) { error =replace_anon_vma_name (vma, anon_name);
mm/mlock.c +2 -1
@@ -387,6 +387,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma,*/ if (newflags &VM_LOCKED ) newflags |=VM_IO ; +vma_start_write (vma);vm_flags_reset_once (vma, newflags);lru_add_drain ();
@@ -461,9 +462,9 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma,* It's okay if try_to_unmap_one unmaps a page just after we * set VM_LOCKED, populate_vma_page_range will bring it back. */ -if ((newflags &VM_LOCKED ) && (oldflags &VM_LOCKED )) {/* No work to do, and mlocking twice would be wrong */ +vma_start_write (vma);vm_flags_reset (vma, newflags); }else {mlock_vma_pages_range (vma, start, end, newflags);
mm/mprotect.c +1 -0
@@ -657,6 +657,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,* vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ +vma_start_write (vma);vm_flags_reset (vma, newflags);if (vma_wants_manual_pte_write_upgrade (vma)) mm_cp_flags |=MM_CP_TRY_CHANGE_WRITABLE ;
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
Re: [PATCH v2 4/6] mm: lock vma explicitly before doing vm_flags_reset and vm_flags_reset_once
Quoted 10 lines · Suren Baghdasaryan
Implicit vma locking inside
vm_flags_reset()andvm_flags_reset_once()is not obvious and makes it hard to understand where vma locking is happening. Also in some cases (like indup_userfaultfd()) vma should be locked earlier thanvma_flagsmodification. To make locking more visible, change these functions to assert that the vma write lock is taken and explicitly lock the vma beforehand. Fix userfaultfd functions which should lock the vma earlier.Suggested-by Linus Torvalds <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>
Quoted 170 lines
--- arch/powerpc/kvm/book3s_hv_uvmem.c | 1 + drivers/infiniband/hw/hfi1/file_ops.c | 1 + fs/userfaultfd.c | 6 ++++++ include/linux/mm.h | 10 +++++++--- mm/madvise.c | 5 ++--- mm/mlock.c | 3 ++- mm/mprotect.c | 1 + 7 files changed, 20 insertions(+), 7 deletions(-)arch/powerpc/kvm/book3s_hv_uvmem.c +1
-0@@ -410,6 +410,7 @@ static int kvmppc_memslot_page_merge(struct kvm *kvm, ret =H_STATE ;break ; } +vma_start_write (vma);/* Copy vm_flags to avoid partial modifications in ksm_madvise */ vm_flags = vma->vm_flags; ret = ksm_madvise(vma, vma->vm_start, vma->vm_end,drivers/infiniband/hw/hfi1/file_ops.c +1
-0@@ -344,6 +344,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) goto done; } +vma_start_write (vma);/* * vm_pgoff is used as a buffer selector cookie. Always mmap from * the beginning. fs/userfaultfd.c +6
-0@@ -667,6 +667,7 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx,mmap_write_lock (mm);for_each_vma (vmi, vma) {if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { +vma_start_write (vma); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ;userfaultfd_set_vm_flags (vma, vma->vm_flags & ~__VM_UFFD_FLAGS);@@ -702,6 +703,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx;if (!octx || !(octx->features &UFFD_FEATURE_EVENT_FORK )) { +vma_start_write (vma); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ;userfaultfd_set_vm_flags (vma, vma->vm_flags & ~__VM_UFFD_FLAGS);return 0 ;@@ -783,6 +785,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma,atomic_inc (&ctx->mmap_changing); }else {/* Drop uffd context if remap feature not enabled */ +vma_start_write (vma); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ;userfaultfd_set_vm_flags (vma, vma->vm_flags & ~__VM_UFFD_FLAGS); }@@ -940,6 +943,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) prev = vma; } +vma_start_write (vma);userfaultfd_set_vm_flags (vma, new_flags); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ; }@@ -1502,6 +1506,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,* the next vma was merged into the current one and * the current one has not been updated yet. */ +vma_start_write (vma);userfaultfd_set_vm_flags (vma, new_flags); vma->vm_userfaultfd_ctx.ctx = ctx;@@ -1685,6 +1690,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,* the next vma was merged into the current one and * the current one has not been updated yet. */ +vma_start_write (vma);userfaultfd_set_vm_flags (vma, new_flags); vma->vm_userfaultfd_ctx =NULL_VM_UFFD_CTX ;include/linux/mm.h +7
-3@@ -780,18 +780,22 @@ static inline void vm_flags_init(struct vm_area_struct *vma,ACCESS_PRIVATE (vma, __vm_flags) = flags; } -/* Use when VMA is part of the VMA tree and modifications need coordination */ +/* +* Use when VMA is part of the VMA tree and modifications need coordination +* Note: vm_flags_reset and vm_flags_reset_once do not lock the vma and +* it should be locked explicitly beforehand. +*/ static inline void vm_flags_reset (struct vm_area_struct *vma,vm_flags_t flags) { -vma_start_write (vma); +vma_assert_write_locked (vma);vm_flags_init (vma, flags); }static inline void vm_flags_reset_once (struct vm_area_struct *vma,vm_flags_t flags) { -vma_start_write (vma); +vma_assert_write_locked (vma);WRITE_ONCE (ACCESS_PRIVATE (vma, __vm_flags), flags); }mm/madvise.c +2
-3@@ -173,9 +173,8 @@ static int madvise_update_vma(struct vm_area_struct *vma, } success: -/* -* vm_flags is protected by the mmap_lock held in write mode. -*/ +/* vm_flags is protected by the mmap_lock held in write mode. */ +vma_start_write (vma);vm_flags_reset (vma, new_flags);if (!vma->vm_file ||vma_is_anon_shmem (vma)) { error =replace_anon_vma_name (vma, anon_name);mm/mlock.c +2
-1@@ -387,6 +387,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma,*/ if (newflags &VM_LOCKED ) newflags |=VM_IO ; +vma_start_write (vma);vm_flags_reset_once (vma, newflags);lru_add_drain ();@@ -461,9 +462,9 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma,* It's okay if try_to_unmap_one unmaps a page just after we * set VM_LOCKED, populate_vma_page_range will bring it back. */ -if ((newflags &VM_LOCKED ) && (oldflags &VM_LOCKED )) {/* No work to do, and mlocking twice would be wrong */ +vma_start_write (vma);vm_flags_reset (vma, newflags); }else {mlock_vma_pages_range (vma, start, end, newflags);mm/mprotect.c +1
-0@@ -657,6 +657,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,* vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ +vma_start_write (vma);vm_flags_reset (vma, newflags);if (vma_wants_manual_pte_write_upgrade (vma)) mm_cp_flags |=MM_CP_TRY_CHANGE_WRITABLE ;Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
Re: [PATCH v2 4/6] mm: lock vma explicitly before doing vm_flags_reset and vm_flags_reset_once
In reply to Suren BaghdasaryanQuoted 4 lines · Suren Baghdasaryan
To make locking more visible, change these
functions to assert that the vma write lock is taken and explicitly lock the vma beforehand.
So I obviously think this is a good change, but the fact that it touched driver files makes me go "we're still doing something wrong".
I'm not super-happy with hfi1_file_mmap() doing something like vma_start_write(), in that I *really* don't think drivers should ever have to think about issues like this.
And I think it's unnecessary. This is the mmap op in the hfi1_file_ops, and I think that any actual mmap() code had _better_ had locked the new vma before asking any driver to set things up (and the assert would catch it if somebody didn't).
I realize that it doesn't hurt in a technical sense, but I think having drivers call these VM-internal subtle locking functions does hurt in a maintenance sense, so we should make sure to not have it.
Linus
Suren Baghdasaryan
reviewerRe: [PATCH v2 4/6] mm: lock vma explicitly before doing vm_flags_reset and vm_flags_reset_once
Quoted 14 lines · Linus Torvalds
Quoted 4 lines · Suren Baghdasaryan
To make locking more visible, change these
functions to assert that the vma write lock is taken and explicitly lock the vma beforehand.So I obviously think this is a good change, but the fact that it touched driver files makes me go "we're still doing something wrong".
I'm not super-happy with
hfi1_file_mmap()doing something likevma_start_write(), in that I *really* don't think drivers should ever have to think about issues like this.
Quoted 8 lines
And I think it's unnecessary. This is the mmap op in the
hfi1_file_ops, and I think that any actualmmap()code had _better_ had locked the new vma before asking any driver to set things up (and the assert would catch it if somebody didn't).I realize that it doesn't hurt in a technical sense, but I think having drivers call these VM-internal subtle locking functions does hurt in a maintenance sense, so we should make sure to not have it.
Ok, IOW the vma would be already locked before mmap() is called... Just to confirm, you are suggesting to remove vma_start_write() call from hfi1_file_mmap() and let vm_flags_reset() generate an assertion if it's ever called with an unlocked vma, correct?
Quoted 2 lines
Linus
Re: [PATCH v2 4/6] mm: lock vma explicitly before doing vm_flags_reset and vm_flags_reset_once
Quoted 2 lines · Suren Baghdasaryan
Ok, IOW the vma would be already locked before
mmap()is called...
Yup.
Quoted 3 lines
Just to confirm, you are suggesting to remove
vma_start_write()call fromhfi1_file_mmap()and letvm_flags_reset()generate an assertion if it's ever called with an unlocked vma, correct?
Correct.
Linus
Suren Baghdasaryan
reviewerRe: [PATCH v2 4/6] mm: lock vma explicitly before doing vm_flags_reset and vm_flags_reset_once
Quoted 5 lines · Linus Torvalds
Quoted 2 lines · Suren Baghdasaryan
Ok, IOW the vma would be already locked before
mmap()is called...
Quoted 7 lines
Yup.
Quoted 3 lines
Just to confirm, you are suggesting to remove
vma_start_write()call fromhfi1_file_mmap()and letvm_flags_reset()generate an assertion if it's ever called with an unlocked vma, correct?Correct.
Got it. Will update in the next version. Thanks!
Quoted 3 lines
Linus
Suren Baghdasaryan
reviewerPATCH v2 5/6 mm: always lock new vma before inserting into vma tree
In reply to Suren BaghdasaryanWhile it's not strictly necessary to lock a newly created vma before adding it into the vma tree (as long as no further changes are performed to it), it seems like a good policy to lock it and prevent accidental changes after it becomes visible to the page faults. Lock the vma before adding it into the vma tree.
--- mm/mmap.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
mm/mmap.c +5 -2
@@ -412,6 +412,8 @@ static int vma_link(struct mm_struct *mm, struct vm_area_struct *vma)if (vma_iter_prealloc (&vmi))return -ENOMEM ; +vma_start_write (vma); +if (vma->vm_file) { mapping = vma->vm_file->f_mapping;i_mmap_lock_write (mapping);
@@ -477,7 +479,8 @@ static inline void vma_prepare(struct vma_prepare *vp)vma_start_write (vp->vma);if (vp->adj_next)vma_start_write (vp->adj_next); -/* vp->insert is always a newly created VMA, no need for locking */ +if (vp->insert) +vma_start_write (vp->insert);if (vp->remove)vma_start_write (vp->remove);if (vp->remove2)
@@ -3098,6 +3101,7 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, vma->vm_pgoff = addr >>PAGE_SHIFT ;vm_flags_init (vma, flags); vma->vm_page_prot =vm_get_page_prot (flags); +vma_start_write (vma);if (vma_iter_store_gfp (vmi, vma,GFP_KERNEL )) goto mas_store_fail;
@@ -3345,7 +3349,6 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,get_file (new_vma->vm_file);if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open (new_vma); -vma_start_write (new_vma);if (vma_link (mm, new_vma)) goto out_vma_link; *need_rmap_locks = false;
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
Re: [PATCH v2 5/6] mm: always lock new vma before inserting into vma tree
Quoted 8 lines · Suren Baghdasaryan
While it's not strictly necessary to lock a newly created vma before adding it into the vma tree (as long as no further changes are performed to it), it seems like a good policy to lock it and prevent accidental changes after it becomes visible to the page faults. Lock the vma before adding it into the vma tree.
Suggested-by Jann Horn <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>
Quoted 46 lines
--- mm/mmap.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)mm/mmap.c +5
-2@@ -412,6 +412,8 @@ static int vma_link(struct mm_struct *mm, struct vm_area_struct *vma)if (vma_iter_prealloc (&vmi))return -ENOMEM ; +vma_start_write (vma); +if (vma->vm_file) { mapping = vma->vm_file->f_mapping;i_mmap_lock_write (mapping);@@ -477,7 +479,8 @@ static inline void vma_prepare(struct vma_prepare *vp)vma_start_write (vp->vma);if (vp->adj_next)vma_start_write (vp->adj_next); -/* vp->insert is always a newly created VMA, no need for locking */ +if (vp->insert) +vma_start_write (vp->insert);if (vp->remove)vma_start_write (vp->remove);if (vp->remove2)@@ -3098,6 +3101,7 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, vma->vm_pgoff = addr >>PAGE_SHIFT ;vm_flags_init (vma, flags); vma->vm_page_prot =vm_get_page_prot (flags); +vma_start_write (vma);if (vma_iter_store_gfp (vmi, vma,GFP_KERNEL )) goto mas_store_fail;@@ -3345,7 +3349,6 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,get_file (new_vma->vm_file);if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open (new_vma); -vma_start_write (new_vma);if (vma_link (mm, new_vma)) goto out_vma_link; *need_rmap_locks = false;Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
Suren Baghdasaryan
reviewerPATCH v2 6/6 mm: move vma locking out of vma_prepare
In reply to Suren Baghdasaryanvma_prepare() is currently the central place where vmas are being locked before vma_complete() applies changes to them. While this is convenient, it also obscures vma locking and makes it hard to follow the locking rules. Move vma locking out of vma_prepare() and take vma locks explicitly at the locations where vmas are being modified.
--- mm/mmap.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)
mm/mmap.c +16 -10
@@ -476,16 +476,6 @@ static inline void init_vma_prep(struct vma_prepare *vp,*/ static inline void vma_prepare (struct vma_prepare *vp) { -vma_start_write (vp->vma); -if (vp->adj_next) -vma_start_write (vp->adj_next); -if (vp->insert) -vma_start_write (vp->insert); -if (vp->remove) -vma_start_write (vp->remove); -if (vp->remove2) -vma_start_write (vp->remove2); -if (vp->file) {uprobe_munmap (vp->vma, vp->vma->vm_start, vp->vma->vm_end);
@@ -650,6 +640,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma,bool remove_next = false;struct vma_prepare vp; +vma_start_write (vma);if (next && (vma != next) && (end == next->vm_end)) {int ret;
@@ -657,6 +648,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, ret =dup_anon_vma (vma, next);if (ret)return ret; +vma_start_write (next); }init_multi_vma_prep (&vp, vma,NULL , remove_next ? next :NULL ,NULL );
@@ -708,6 +700,8 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma,if (vma_iter_prealloc (vmi))return -ENOMEM ; +vma_start_write (vma); +init_vma_prep (&vp, vma);vma_prepare (&vp);vma_adjust_trans_huge (vma, start, end,0 );
@@ -946,10 +940,12 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm,/* Can we merge both the predecessor and the successor? */ if (merge_prev && merge_next &&is_mergeable_anon_vma (prev->anon_vma, next->anon_vma,NULL )) { +vma_start_write (next); remove = next;/* case 1 */ vma_end = next->vm_end; err =dup_anon_vma (prev, next);if (curr) {/* case 6 */ +vma_start_write (curr); remove = curr; remove2 = next;if (!next->anon_vma)
@@ -958,6 +954,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, }else if (merge_prev) {/* case 2 */ if (curr) { err =dup_anon_vma (prev, curr); +vma_start_write (curr);if (end == curr->vm_end) {/* case 7 */ remove = curr; }else {/* case 5 */
@@ -969,6 +966,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, res = next;if (prev && addr < prev->vm_end) {/* case 4 */ vma_end = addr; +vma_start_write (next); adjust = next; adj_start = -(prev->vm_end - addr); err =dup_anon_vma (next, prev);
@@ -983,6 +981,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, vma_pgoff = next->vm_pgoff - pglen;if (curr) {/* case 8 */ vma_pgoff = curr->vm_pgoff; +vma_start_write (curr); remove = curr; err =dup_anon_vma (next, curr); }
@@ -996,6 +995,8 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm,if (vma_iter_prealloc (vmi))return NULL ; +vma_start_write (vma); +init_multi_vma_prep (&vp, vma, adjust, remove, remove2);VM_WARN_ON (vp.anon_vma && adjust && adjust->anon_vma && vp.anon_vma != adjust->anon_vma);
@@ -2373,6 +2374,9 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,if (new->vm_ops && new->vm_ops->open) new->vm_ops->open (new); +vma_start_write (vma); +vma_start_write (new); +init_vma_prep (&vp, vma); vp.insert = new;vma_prepare (&vp);
@@ -3078,6 +3082,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma,if (vma_iter_prealloc (vmi)) goto unacct_fail; +vma_start_write (vma); +init_vma_prep (&vp, vma);vma_prepare (&vp);vma_adjust_trans_huge (vma, vma->vm_start, addr + len,0 );
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
Re: [PATCH v2 6/6] mm: move vma locking out of vma_prepare
Quoted 35 lines · Suren Baghdasaryan
vma_prepare()is currently the central place where vmas are being locked beforevma_complete()applies changes to them. While this is convenient, it also obscures vma locking and makes it hard to follow the locking rules. Move vma locking out ofvma_prepare()and take vma locks explicitly at the locations where vmas are being modified.Suggested-by Linus Torvalds <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Liam R. Howlett <[email protected]>--- mm/mmap.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)mm/mmap.c +0
-10@@ -476,16 +476,6 @@ static inline void init_vma_prep(struct vma_prepare *vp,*/ static inline void vma_prepare (struct vma_prepare *vp) { -vma_start_write (vp->vma); -if (vp->adj_next) -vma_start_write (vp->adj_next); -if (vp->insert) -vma_start_write (vp->insert); -if (vp->remove) -vma_start_write (vp->remove); -if (vp->remove2) -vma_start_write (vp->remove2); -if (vp->file) {uprobe_munmap (vp->vma, vp->vma->vm_start, vp->vma->vm_end);
mm/mmap.c int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 643
Quoted 4 lines
mm/mmap.c +1
-0@@ -650,6 +640,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma,bool remove_next = false;struct vma_prepare vp; +vma_start_write (vma);
This lock made me think about the lock in dup_anon_vma().. the only reason we need to dup the anon vma is because the VMA _will_ be modified.. So if we are to remove next the dup_anon_vma() call may lock vma again. This happens multiple times through this patch set.
If we lock both dst and src VMAs before calling dup_anon_vma, then I think it will become more clear in the code below which VMAs are locked. We could remove the lock in the dup_anon_vma() (and replace with an assert?), and drop the vma locking of "vma".
I think this would address the confusion of the locking below that I experienced and avoid calling the vma_start_write() multiple times for the same vma. I mean, having the vma_start_write(vma) obscures which VMA is locked as well.
Quoted 88 lines
mm/mmap.c +15
-0if (next && (vma != next) && (end == next->vm_end)) {int ret;@@ -657,6 +648,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, ret =dup_anon_vma (vma, next);if (ret)return ret; +vma_start_write (next); }init_multi_vma_prep (&vp, vma,NULL , remove_next ? next :NULL ,NULL );@@ -708,6 +700,8 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma,if (vma_iter_prealloc (vmi))return -ENOMEM ; +vma_start_write (vma); +init_vma_prep (&vp, vma);vma_prepare (&vp);vma_adjust_trans_huge (vma, start, end,0 );@@ -946,10 +940,12 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm,/* Can we merge both the predecessor and the successor? */ if (merge_prev && merge_next &&is_mergeable_anon_vma (prev->anon_vma, next->anon_vma,NULL )) { +vma_start_write (next); remove = next;/* case 1 */ vma_end = next->vm_end; err =dup_anon_vma (prev, next);if (curr) {/* case 6 */ +vma_start_write (curr); remove = curr; remove2 = next;if (!next->anon_vma)@@ -958,6 +954,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, }else if (merge_prev) {/* case 2 */ if (curr) { err =dup_anon_vma (prev, curr); +vma_start_write (curr);if (end == curr->vm_end) {/* case 7 */ remove = curr; }else {/* case 5 */ @@ -969,6 +966,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, res = next;if (prev && addr < prev->vm_end) {/* case 4 */ vma_end = addr; +vma_start_write (next); adjust = next; adj_start = -(prev->vm_end - addr); err =dup_anon_vma (next, prev);@@ -983,6 +981,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, vma_pgoff = next->vm_pgoff - pglen;if (curr) {/* case 8 */ vma_pgoff = curr->vm_pgoff; +vma_start_write (curr); remove = curr; err =dup_anon_vma (next, curr); }@@ -996,6 +995,8 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm,if (vma_iter_prealloc (vmi))return NULL ; +vma_start_write (vma); +init_multi_vma_prep (&vp, vma, adjust, remove, remove2);VM_WARN_ON (vp.anon_vma && adjust && adjust->anon_vma && vp.anon_vma != adjust->anon_vma);@@ -2373,6 +2374,9 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,if (new->vm_ops && new->vm_ops->open) new->vm_ops->open (new); +vma_start_write (vma); +vma_start_write (new); +init_vma_prep (&vp, vma); vp.insert = new;vma_prepare (&vp);@@ -3078,6 +3082,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma,if (vma_iter_prealloc (vmi)) goto unacct_fail; +vma_start_write (vma); +init_vma_prep (&vp, vma);vma_prepare (&vp);vma_adjust_trans_huge (vma, vma->vm_start, addr + len,0 );Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
Suren Baghdasaryan
reviewerRe: [PATCH v2 6/6] mm: move vma locking out of vma_prepare
Quoted 52 lines · Liam R. Howlett
Quoted 35 lines · Suren Baghdasaryan
vma_prepare()is currently the central place where vmas are being locked beforevma_complete()applies changes to them. While this is convenient, it also obscures vma locking and makes it hard to follow the locking rules. Move vma locking out ofvma_prepare()and take vma locks explicitly at the locations where vmas are being modified.Suggested-by Linus Torvalds <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Reviewed-by Liam R. Howlett <[email protected]>--- mm/mmap.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)mm/mmap.c +0
-10@@ -476,16 +476,6 @@ static inline void init_vma_prep(struct vma_prepare *vp,*/ static inline void vma_prepare (struct vma_prepare *vp) { -vma_start_write (vp->vma); -if (vp->adj_next) -vma_start_write (vp->adj_next); -if (vp->insert) -vma_start_write (vp->insert); -if (vp->remove) -vma_start_write (vp->remove); -if (vp->remove2) -vma_start_write (vp->remove2); -if (vp->file) {uprobe_munmap (vp->vma, vp->vma->vm_start, vp->vma->vm_end);mm/mmap.c int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 643
Quoted 4 lines
mm/mmap.c +1
-0@@ -650,6 +640,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma,bool remove_next = false;struct vma_prepare vp; +vma_start_write (vma);This lock made me think about the lock in
dup_anon_vma().. the only reason we need to dup the anon vma is because the VMA _will_ be modified.. So if we are to remove next thedup_anon_vma()call may lock vma again. This happens multiple times through this patch set.If we lock both dst and src VMAs before calling
dup_anon_vma, then I think it will become more clear in the code below which VMAs are locked. We could remove the lock in thedup_anon_vma()(and replace with an assert?), and drop the vma locking of "vma".
Quoted 4 lines
I think this would address the confusion of the locking below that I experienced and avoid calling the
vma_start_write()multiple times for the same vma. I mean, having thevma_start_write(vma)obscures which VMA is locked as well.
Agree, it would result in an easier to understand locking code. I'll move the lock outside of dup_anon_vma() and will replace it with an assert. Thanks!
Quoted 90 lines
Quoted 90 lines
mm/mmap.c +15
-1if (next && (vma != next) && (end == next->vm_end)) {int ret;@@ -657,6 +648,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, ret =dup_anon_vma (vma, next);if (ret)return ret; +vma_start_write (next); }init_multi_vma_prep (&vp, vma,NULL , remove_next ? next :NULL ,NULL );@@ -708,6 +700,8 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma,if (vma_iter_prealloc (vmi))return -ENOMEM ; +vma_start_write (vma); +init_vma_prep (&vp, vma);vma_prepare (&vp);vma_adjust_trans_huge (vma, start, end,0 );@@ -946,10 +940,12 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm,/* Can we merge both the predecessor and the successor? */ if (merge_prev && merge_next &&is_mergeable_anon_vma (prev->anon_vma, next->anon_vma,NULL )) { +vma_start_write (next); remove = next;/* case 1 */ vma_end = next->vm_end; err =dup_anon_vma (prev, next);if (curr) {/* case 6 */ +vma_start_write (curr); remove = curr; remove2 = next;if (!next->anon_vma)@@ -958,6 +954,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, }else if (merge_prev) {/* case 2 */ if (curr) { err =dup_anon_vma (prev, curr); +vma_start_write (curr);if (end == curr->vm_end) {/* case 7 */ remove = curr; }else {/* case 5 */ @@ -969,6 +966,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, res = next;if (prev && addr < prev->vm_end) {/* case 4 */ vma_end = addr; +vma_start_write (next); adjust = next; adj_start = -(prev->vm_end - addr); err =dup_anon_vma (next, prev);@@ -983,6 +981,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, vma_pgoff = next->vm_pgoff - pglen;if (curr) {/* case 8 */ vma_pgoff = curr->vm_pgoff; +vma_start_write (curr); remove = curr; err =dup_anon_vma (next, curr); }@@ -996,6 +995,8 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm,if (vma_iter_prealloc (vmi))return NULL ; +vma_start_write (vma); +init_multi_vma_prep (&vp, vma, adjust, remove, remove2);VM_WARN_ON (vp.anon_vma && adjust && adjust->anon_vma && vp.anon_vma != adjust->anon_vma);@@ -2373,6 +2374,9 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,if (new->vm_ops && new->vm_ops->open) new->vm_ops->open (new); +vma_start_write (vma); +vma_start_write (new); +init_vma_prep (&vp, vma); vp.insert = new;vma_prepare (&vp);@@ -3078,6 +3082,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma,if (vma_iter_prealloc (vmi)) goto unacct_fail; +vma_start_write (vma); +init_vma_prep (&vp, vma);vma_prepare (&vp);vma_adjust_trans_huge (vma, vma->vm_start, addr + len,0 ); --2.41.0.585.gd2178a4bd4-goog
Suren Baghdasaryan
reviewerQuoted 38 lines · Suren Baghdasaryan
During recent vma locking patch reviews Linus and Jann Horn noted a number of issues with vma locking and suggested improvements:
1.
walk_page_range()does not have ability to write-lock a vma during the walk when it's done undermmap_write_lock. For examples390_reset_cmma().2. Vma locking is hidden inside
vm_flagsmodifiers and is hard to follow. Suggestion is to change vm_flags_reset{_once} to assert that vma is write-locked and require an explicit locking.3. Same issue with
vma_prepare()hiding vma locking.4. In userfaultfd
vm_flagsare modified aftervma->vm_userfaultfd_ctxand page faults can operate on a context while it's changed.5.
do_brk_flags()and__install_special_mapping()not locking a newly created vma before adding it into the mm. While not strictly a problem, this is fragile if vma is modified after insertion, as in themmap_region()case which was recently fixed. Suggestion is to always lock a new vma before inserting it and making it visible to page faults.6.
vma_assert_write_locked()forCONFIG_PER_VMA_LOCK=nwould benefit from beingmmap_assert_write_locked()instead of no-op and then any place which operates on a vma and callsmmap_assert_write_locked()can be converted intovma_assert_write_locked().I CC'ed stable only on the first patch because others are cleanups and the bug in userfaultfd does not affect stable (
lock_vma_under_rcuprevents uffds from being handled under vma lock protection). However I would be happy if the whole series is merged into stable 6.4 since it makes vma locking more maintainable.The patches apply cleanly over Linus' ToT and will conflict when applied over mm-unstable due to missing [1]. The conflict can be easily resolved by ignoring conflicting deletions but probably simpler to take [1] into mm-unstable and avoid later conflict.
Quoted 8 lines
[1] commit
6c21e066f925("mm/mempolicy: Take VMA lock before replacing policy")Changes since v1:
- replacewalk_page_range()parameter with mm_walk_ops.walk_lock, per Linus
- introducedpage_walk_lockenum to allow different locking modes during a walk, per Linus
- added Liam's Reviewed-by
v3 is posted at
https://lore.kernel.org/all/[email protected]/
Quoted 43 lines
Suren Baghdasaryan (6):
mm: enable page walking API to lock vmas during the walk
mm: for !CONFIG_PER_VMA_LOCK equate write lock assertion for vma and
mmap
mm: replace mmap with vma write lock assertions when operating on a
vma
mm: lock vma explicitly before doingvm_flags_resetand
vm_flags_reset_once
mm: always lock new vma before inserting into vma tree
mm: move vma locking out ofvma_preparearch/powerpc/kvm/book3s_hv_uvmem.c | 1 + arch/powerpc/mm/book3s64/subpage_prot.c | 1 + arch/riscv/mm/pageattr.c | 1 + arch/s390/mm/gmap.c | 5 ++++ drivers/infiniband/hw/hfi1/file_ops.c | 1 + fs/proc/task_mmu.c | 5 ++++ fs/userfaultfd.c | 6 +++++ include/linux/mm.h | 13 ++++++--- include/linux/pagewalk.h | 11 ++++++++ mm/damon/vaddr.c | 2 ++ mm/hmm.c | 1 + mm/hugetlb.c | 2 +- mm/khugepaged.c | 5 ++-- mm/ksm.c | 25 ++++++++++------- mm/madvise.c | 8 +++--- mm/memcontrol.c | 2 ++ mm/memory-failure.c | 1 + mm/memory.c | 2 +- mm/mempolicy.c | 22 +++++++++------ mm/migrate_device.c | 1 + mm/mincore.c | 1 + mm/mlock.c | 4 ++- mm/mmap.c | 29 +++++++++++++------- mm/mprotect.c | 2 ++ mm/pagewalk.c | 36 ++++++++++++++++++++++--- mm/vmscan.c | 1 + 26 files changed, 146 insertions(+), 42 deletions(-)--
2.41.0.585.gd2178a4bd4-goog