mm-handle-swap-page-faults-under-per-vma-lock.patch removed from -mm tree
The quilt patch titled
Subject: mm: handle swap page faults under per-VMA lock
has been removed from the -mm tree. Its filename was
mm-handle-swap-page-faults-under-per-vma-lock.patch
This patch was dropped because it was merged into the mm-stable branch of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
------------------------------------------------------
From: Suren Baghdasaryan <[email protected]>
Subject: mm: handle swap page faults under per-VMA lock
Date: Fri, 30 Jun 2023 14:19:56 -0700
When page fault is handled under per-VMA lock protection, all swap page faults are retried with mmap_lock because folio_lock_or_retry has to drop and reacquire mmap_lock if folio could not be immediately locked. Follow the same pattern as mmap_lock to drop per-VMA lock when waiting for folio and retrying once folio is available.
With this obstacle removed, enable do_swap_page to operate under per-VMA lock protection. Drivers implementing ops->migrate_to_ram might still rely on mmap_lock, therefore we have to fall back to mmap_lock in that particular case.
Note that the only time do_swap_page calls synchronous swap_readpage is when SWP_SYNCHRONOUS_IO is set, which is only set for QUEUE_FLAG_SYNCHRONOUS devices: brd, zram and nvdimms (both btt and pmem). Therefore we don't sleep in this path, and there's no need to drop the mmap or per-VMA lock.
---
include/linux/mm.h | 13 +++++++++++++ mm/filemap.c | 17 ++++++++--------- mm/memory.c | 16 ++++++++++------ 3 files changed, 31 insertions(+), 15 deletions(-)
include/linux/mm.h renamed from include/linux/mm.h~mm-handle-swap-page-faults-under-per-vma-lock +13 -0
@@ -729,6 +729,14 @@ static inline void vma_mark_detached(str vma->detached = detached; } +static inline void release_fault_lock (struct vm_fault *vmf) +{ +if (vmf->flags &FAULT_FLAG_VMA_LOCK ) +vma_end_read (vmf->vma); +else +mmap_read_unlock (vmf->vma->vm_mm); +} +struct vm_area_struct *lock_vma_under_rcu (struct mm_struct *mm,unsigned long address);
@@ -749,6 +757,11 @@ static inline struct vm_area_struct *loc return NULL; } +static inline void release_fault_lock (struct vm_fault *vmf) +{ +mmap_read_unlock (vmf->vma->vm_mm); +} +#endif /* CONFIG_PER_VMA_LOCK */ extern const struct vm_operations_struct vma_dummy_vm_ops;
mm/filemap.c renamed from mm/filemap.c~mm-handle-swap-page-faults-under-per-vma-lock +8 -9
@@ -1671,27 +1671,26 @@ static int __folio_lock_async(struct fol* Return values: * 0 - folio is locked. * non-zero - folio is not locked. -* mmap_lock has been released (mmap_read_unlock(), unless flags had both -* FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_RETRY_NOWAIT set, in -* which case mmap_lock is still held. +* mmap_lock or per-VMA lock has been released (mmap_read_unlock() or +* vma_end_read()), unless flags had both FAULT_FLAG_ALLOW_RETRY and +* FAULT_FLAG_RETRY_NOWAIT set, in which case the lock is still held. * * If neither ALLOW_RETRY nor KILLABLE are set, will always return 0 -* with the folio locked and the mmap_lock unperturbed. +* with the folio locked and the mmap_lock/per-VMA lock is left unperturbed. */ vm_fault_t __folio_lock_or_retry (struct folio *folio,struct vm_fault *vmf) { -struct mm_struct *mm = vmf->vma->vm_mm;unsigned int flags = vmf->flags;if (fault_flag_allow_retry_first (flags)) {/* -* CAUTION! In this case, mmap_lock is not released -* even though return VM_FAULT_RETRY. +* CAUTION! In this case, mmap_lock/per-VMA lock is not +* released even though returning VM_FAULT_RETRY. */ if (flags &FAULT_FLAG_RETRY_NOWAIT )return VM_FAULT_RETRY ; -mmap_read_unlock (mm); +release_fault_lock (vmf);if (flags &FAULT_FLAG_KILLABLE )folio_wait_locked_killable (folio);else
@@ -1703,7 +1702,7 @@ vm_fault_t __folio_lock_or_retry(struct ret =__folio_lock_killable (folio);if (ret) { -mmap_read_unlock (mm); +release_fault_lock (vmf);return VM_FAULT_RETRY ; } }else {
mm/memory.c renamed from mm/memory.c~mm-handle-swap-page-faults-under-per-vma-lock +10 -6
@@ -3746,12 +3746,6 @@ vm_fault_t do_swap_page(struct vm_faultif (!pte_unmap_same (vmf)) goto out; -if (vmf->flags &FAULT_FLAG_VMA_LOCK ) { - ret =VM_FAULT_RETRY ; -vma_end_read (vma); - goto out; - } - entry =pte_to_swp_entry (vmf->orig_pte);if (unlikely (non_swap_entry (entry))) {if (is_migration_entry (entry)) {
@@ -3761,6 +3755,16 @@ vm_fault_t do_swap_page(struct vm_fault vmf->page =pfn_swap_entry_to_page (entry); ret =remove_device_exclusive_entry (vmf); }else if (is_device_private_entry (entry)) { +if (vmf->flags &FAULT_FLAG_VMA_LOCK ) { +/* +* migrate_to_ram is not yet ready to operate +* under VMA lock. +*/ +vma_end_read (vma); + ret =VM_FAULT_RETRY ; + goto out; + } + vmf->page =pfn_swap_entry_to_page (entry); vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl);
_
Patches currently in -mm which might be from [email protected] are