mm: Avoid unnecessary page fault retires on shared memory types
Subsystems (25)
Peter Xu
reviewerPATCH 1/1 mm: Avoid unnecessary page fault retires on shared memory types
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
Then after that throttling we return VM_FAULT_RETRY.
We did that probably because VM_FAULT_RETRY is the only way we can return to the fault handler at that time telling it we've released the mmap lock.
However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in /proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.
To make it more efficient, add a new VM_FAULT_COMPLETED return code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.
This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed:
Before: 650980.20 (+-1.94%)
After: 569396.40 (+-1.38%)
I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Another thing to mention is that mm_account_fault() does take this new fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping them as-is.
---
Note: I didn't copy any arch maintainers yet because I want to get some feedback from the mm people first.. I'll start doing so when there's a new version. Please have a look, thanks.
--- arch/alpha/mm/fault.c | 4 ++++ arch/arc/mm/fault.c | 4 ++++ arch/arm/mm/fault.c | 4 ++++ arch/arm64/mm/fault.c | 4 ++++ arch/csky/mm/fault.c | 4 ++++ arch/hexagon/mm/vm_fault.c | 4 ++++ arch/ia64/mm/fault.c | 4 ++++ arch/m68k/mm/fault.c | 4 ++++ arch/microblaze/mm/fault.c | 4 ++++ arch/mips/mm/fault.c | 4 ++++ arch/nios2/mm/fault.c | 4 ++++ arch/openrisc/mm/fault.c | 4 ++++ arch/parisc/mm/fault.c | 4 ++++ arch/powerpc/mm/copro_fault.c | 5 +++++ arch/powerpc/mm/fault.c | 5 +++++ arch/riscv/mm/fault.c | 4 ++++ arch/s390/mm/fault.c | 12 +++++++++++- arch/sh/mm/fault.c | 4 ++++ arch/sparc/mm/fault_32.c | 7 +++++++ arch/sparc/mm/fault_64.c | 5 +++++ arch/um/kernel/trap.c | 4 ++++ arch/x86/mm/fault.c | 4 ++++ arch/xtensa/mm/fault.c | 4 ++++ include/linux/mm_types.h | 2 ++ mm/gup.c | 34 +++++++++++++++++++++++++++++++++- mm/memory.c | 2 +- 26 files changed, 141 insertions(+), 3 deletions(-)
arch/alpha/mm/fault.c +4 -0
@@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/arc/mm/fault.c +4 -0
@@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +/* * Fault retry nuances, mmap_lock already relinquished by core mm */
arch/arm/mm/fault.c +4 -0
@@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)return 0 ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (!(fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
arch/arm64/mm/fault.c +4 -0
@@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,return 0 ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (fault &VM_FAULT_RETRY ) { mm_flags |=FAULT_FLAG_TRIED ; goto retry;
arch/csky/mm/fault.c +4 -0
@@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely ((fault &VM_FAULT_RETRY ) && (flags &FAULT_FLAG_ALLOW_RETRY ))) { flags |=FAULT_FLAG_TRIED ;
arch/hexagon/mm/vm_fault.c +4 -0
@@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +/* The most common case -- we are done. */ if (likely (!(fault &VM_FAULT_ERROR ))) {if (fault &VM_FAULT_RETRY ) {
arch/ia64/mm/fault.c +4 -0
@@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *reif (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {/* * We ran out of memory, or some other thing happened
arch/m68k/mm/fault.c +4 -0
@@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return 0 ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/microblaze/mm/fault.c +4 -0
@@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/mips/mm/fault.c +4 -0
@@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/nios2/mm/fault.c +4 -0
@@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/openrisc/mm/fault.c +4 -0
@@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/parisc/mm/fault.c +4 -0
@@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {/* * We hit a shared mapping outside of the file, or some
arch/powerpc/mm/copro_fault.c +5 -0
@@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea, ret =0 ; *flt =handle_mm_fault (vma, ea, is_write ?FAULT_FLAG_WRITE :0 ,NULL ); + +/* The fault is fully completed (including releasing mmap lock) */ +if (*flt &VM_FAULT_COMPLETED ) +return 0 ; +if (unlikely (*flt &VM_FAULT_ERROR )) {if (*flt &VM_FAULT_OOM ) { ret = -ENOMEM ;
arch/powerpc/mm/fault.c +5 -0
@@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return user_mode (regs) ?0 :SIGBUS ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) + goto out; +/* * Handle the retry right now, the mmap_lock has been released in that * case.
@@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,if (unlikely (fault &VM_FAULT_ERROR ))return mm_fault_error (regs, address, fault); +out:/* * Major/minor page fault accounting. */
arch/riscv/mm/fault.c +4 -0
@@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_RETRY )) { flags |=FAULT_FLAG_TRIED ;
arch/s390/mm/fault.c +11 -1
@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)unsigned long address;unsigned int flags;vm_fault_t fault; +bool need_unlock = true;bool is_write; tsk = current;
@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) goto out_up; goto out; } + +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) { + need_unlock = false; + goto out_gmap; + } +if (unlikely (fault &VM_FAULT_ERROR )) goto out_up;
@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)mmap_read_lock (mm); goto retry; } +out_gmap:if (IS_ENABLED (CONFIG_PGSTE ) && gmap) { address =__gmap_link (gmap, current->thread.gmap_addr, address);
@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) } fault =0 ; out_up: -mmap_read_unlock (mm); +if (need_unlock) +mmap_read_unlock (mm); out:return fault; }
arch/sh/mm/fault.c +4 -0
@@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,if (mm_fault_error (regs, error_code, address, fault))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
arch/sparc/mm/fault_32.c +7 -0
@@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
@@ -339,6 +343,9 @@ static void force_user_fault(unsigned long address, int write)case VM_FAULT_OOM : goto do_sigbus; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ;mmap_read_unlock (mm);return ; bad_area:
arch/sparc/mm/fault_64.c +5 -0
@@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)if (fault_signal_pending (fault, regs)) goto exit_exception; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) + goto lock_released; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
@@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs) }mmap_read_unlock (mm); +lock_released: mm_rss =get_mm_rss (mm);#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE /PAGE_SIZE ));
arch/um/kernel/trap.c +4 -0
@@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,if ((fault &VM_FAULT_RETRY ) &&fatal_signal_pending (current)) goto out_nosemaphore; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) { goto out_of_memory;
arch/x86/mm/fault.c +4 -0
@@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +/* * If we need to retry the mmap_lock has already been released, * and if there is a fatal signal pending there is no guarantee
arch/xtensa/mm/fault.c +4 -0
@@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
include/linux/mm_types.h +2 -0
@@ -717,6 +717,7 @@ typedef __bitwise unsigned int vm_fault_t;* @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs * fsync() to complete (for synchronous page faults * in DAX) +* @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released * @VM_FAULT_HINDEX_MASK: mask HINDEX value * */
@@ -734,6 +735,7 @@ enum vm_fault_reason {VM_FAULT_FALLBACK = (__force vm_fault_t )0x000800 ,VM_FAULT_DONE_COW = (__force vm_fault_t )0x001000 ,VM_FAULT_NEEDDSYNC = (__force vm_fault_t )0x002000 , +VM_FAULT_COMPLETED = (__force vm_fault_t )0x004000 , VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000, };
mm/gup.c +33 -1
@@ -876,6 +876,25 @@ static int faultin_page(struct vm_area_struct *vma, } ret =handle_mm_fault (vma, address, fault_flags,NULL ); + +if (ret &VM_FAULT_COMPLETED ) { +/* +* With FAULT_FLAG_RETRY_NOWAIT we'll never release the +* mmap lock in the page fault handler. Sanity check this. +*/ +WARN_ON_ONCE (fault_flags &FAULT_FLAG_RETRY_NOWAIT ); +if (locked) + *locked =0 ; +/* +* We should do the same as VM_FAULT_RETRY, but let's not +* return -EBUSY since that's not reflecting the reality on +* what has happened - we've just fully completed a page +* fault, with the mmap lock released. Use -EAGAIN to show +* that we want to take the mmap lock _again_. +*/ +return -EAGAIN ; + } +if (ret &VM_FAULT_ERROR ) {int err =vm_fault_to_errno (ret, *flags);
@@ -1101,6 +1120,7 @@ static long __get_user_pages(struct mm_struct *mm,case 0 : goto retry;case -EBUSY : +case -EAGAIN : ret =0 ; fallthrough;case -EFAULT :
@@ -1227,6 +1247,18 @@ int fixup_user_fault(struct mm_struct *mm,return -EINTR ; ret =handle_mm_fault (vma, address, fault_flags,NULL ); + +if (ret &VM_FAULT_COMPLETED ) { +/* +* NOTE: it's a pity that we need to retake the lock here +* to pair with the unlock() in the callers. Ideally we +* could tell the callers so they do not need to unlock. +*/ +mmap_read_lock (mm); + *unlocked = true; +return 0 ; + } +if (ret &VM_FAULT_ERROR ) {int err =vm_fault_to_errno (ret,0 );
@@ -1292,7 +1324,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,/* VM_FAULT_RETRY couldn't trigger, bypass */ return ret; -/* VM_FAULT_RETRY cannot return errors */ +/* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */ if (!*locked) {BUG_ON (ret <0 );BUG_ON (ret >= nr_pages);
mm/memory.c +1 -1
@@ -2942,7 +2942,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)balance_dirty_pages_ratelimited (mapping);if (fpin) {fput (fpin); -return VM_FAULT_RETRY ; +return VM_FAULT_COMPLETED ; } }
Signature · 1 line
2.32.0
kernel test robot
build botThank you for the patch! Yet something to improve:
Hi Peter,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on hnaz-mm/master]
url: https://github.com/intel-lab-lkp/linux/commits/Peter-Xu/mm-Avoid-unnecessary-page-fault-retires-on-shared-memory-types/20220506-051845 base: https://github.com/hnaz/linux-mm master config: sparc-defconfig (https://download.01.org/0day-ci/archive/20220506/[email protected]/config) compiler: sparc-linux-gcc (GCC) 11.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/e5356147afd3824144b605c428f1b892addd11d0
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Peter-Xu/mm-Avoid-unnecessary-page-fault-retires-on-shared-memory-types/20220506-051845 git checkout e5356147afd3824144b605c428f1b892addd11d0
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.3.0 make.cross W=1 O=build_dir ARCH=sparc SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
arch/sparc/mm/fault_32.c: In function 'force_user_fault':
>> arch/sparc/mm/fault_32.c:347:13: error: 'fault' undeclared (first use in this function)
347 | if (fault & VM_FAULT_COMPLETED)
| ^~~~~
arch/sparc/mm/fault_32.c:347:13: note: each undeclared identifier is reported only once for each function it appears in
vim +/fault +347 arch/sparc/mm/fault_32.c
309
310 /* This always deals with user addresses. */
311 static void force_user_fault(unsigned long address, int write)
312 {
313 struct vm_area_struct *vma;
314 struct task_struct *tsk = current;
315 struct mm_struct *mm = tsk->mm;
316 unsigned int flags = FAULT_FLAG_USER;
317 int code;
318
319 code = SEGV_MAPERR;
320
321 mmap_read_lock(mm);
322 vma = find_vma(mm, address);
323 if (!vma)
324 goto bad_area;
325 if (vma->vm_start <= address)
326 goto good_area;
327 if (!(vma->vm_flags & VM_GROWSDOWN))
328 goto bad_area;
329 if (expand_stack(vma, address))
330 goto bad_area;
331 good_area:
332 code = SEGV_ACCERR;
333 if (write) {
334 if (!(vma->vm_flags & VM_WRITE))
335 goto bad_area;
336 flags |= FAULT_FLAG_WRITE;
337 } else {
338 if (!(vma->vm_flags & (VM_READ | VM_EXEC)))
339 goto bad_area;
340 }
341 switch (handle_mm_fault(vma, address, flags, NULL)) {
342 case VM_FAULT_SIGBUS:
343 case VM_FAULT_OOM:
344 goto do_sigbus;
345 }
346 /* The fault is fully completed (including releasing mmap lock) */
> 347 if (fault & VM_FAULT_COMPLETED)
348 return;
349 mmap_read_unlock(mm);
350 return;
351 bad_area:
352 mmap_read_unlock(mm);
353 __do_fault_siginfo(code, SIGSEGV, tsk->thread.kregs, address);
354 return;
355
356 do_sigbus:
357 mmap_read_unlock(mm);
358 __do_fault_siginfo(BUS_ADRERR, SIGBUS, tsk->thread.kregs, address);
359 }
360
Signature · 2 lines
0-DAY CI Kernel Test Service https://01.org/lkp
Quoted 40 lines · Peter Xu
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with
balance_dirty_pages_ratelimited()(infault_dirty_shared_page()).Then after that throttling we return
VM_FAULT_RETRY.We did that probably because
VM_FAULT_RETRYis the only way we can return to the fault handler at that time telling it we've released the mmap lock.However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in
/proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.To make it more efficient, add a new
VM_FAULT_COMPLETEDreturn code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed:
Before: 650980.20 (+-1.94%)
After: 569396.40 (+-1.38%)I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Quoted 8 lines
Another thing to mention is that
mm_account_fault()does take this new fault as a generic fault to be accounted, unlikeVM_FAULT_RETRY.I explicitly didn't touch
hmm_vma_fault()andbreak_ksm()because they do not handleVM_FAULT_RETRYeven with existing code, so I'm literally keeping them as-is.Signed-off-by Peter Xu <[email protected]>
The change makes sense to me, but the unlock/retry signaling is tricky...
Quoted 5 lines
mm/gup.c +1
-0@@ -1227,6 +1247,18 @@ int fixup_user_fault(struct mm_struct *mm,return -EINTR ; ret =handle_mm_fault (vma, address, fault_flags,NULL ); +
mm/gup.c int fixup_user_fault(struct mm_struct *mm, · line 1260
Quoted 10 lines
mm/gup.c +10
-0+if (ret &VM_FAULT_COMPLETED ) { +/* +* NOTE: it's a pity that we need to retake the lock here +* to pair with the unlock() in the callers. Ideally we +* could tell the callers so they do not need to unlock. +*/ +mmap_read_lock (mm); + *unlocked = true; +return 0 ; + }
unlocked can be NULL inside the function, yet you assume it's non-NULL here. This is okay because COMPLETED can only be returned if RETRY is set, and when RETRY is set unlocked must be non-NULL. It's correct but not very obvious.
It might be cleaner to have separate flags for ALLOW_RETRY and ALLOW_UNLOCK, with corresponding VM_FAULT_RETRY and VM_FAULT_UNLOCKED? Even if not all combinations are used.
mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 2945
Quoted 8 lines
mm/memory.c +1
-1@@ -2942,7 +2942,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)balance_dirty_pages_ratelimited (mapping);if (fpin) {fput (fpin); -return VM_FAULT_RETRY ; +return VM_FAULT_COMPLETED ;
There is one oddity in this now.
It completes the fault and no longer triggers a retry. Yet it's still using maybe_unlock_mmap_for_io() and subject to retry limiting. This means that if the fault already retried once, this code won't drop the mmap_sem to call balance_dirty_pages() - even though it safely could and should do so, without risking endless retries.
Here too IMO the distinction between ALLOW_RETRY|TRIED and ALLOW_UNLOCK would make things cleaner and more obvious.
Peter Xu
reviewerHi, Johannes,
Quoted 74 lines · Johannes Weiner
Quoted 40 lines · Peter Xu
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with
balance_dirty_pages_ratelimited()(infault_dirty_shared_page()).Then after that throttling we return
VM_FAULT_RETRY.We did that probably because
VM_FAULT_RETRYis the only way we can return to the fault handler at that time telling it we've released the mmap lock.However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in
/proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.To make it more efficient, add a new
VM_FAULT_COMPLETEDreturn code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed:
Before: 650980.20 (+-1.94%)
After: 569396.40 (+-1.38%)I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Quoted 8 lines
Another thing to mention is that
mm_account_fault()does take this new fault as a generic fault to be accounted, unlikeVM_FAULT_RETRY.I explicitly didn't touch
hmm_vma_fault()andbreak_ksm()because they do not handleVM_FAULT_RETRYeven with existing code, so I'm literally keeping them as-is.Signed-off-by Peter Xu <[email protected]>The change makes sense to me, but the unlock/retry signaling is tricky...
Quoted 5 lines
mm/gup.c +1
-0@@ -1227,6 +1247,18 @@ int fixup_user_fault(struct mm_struct *mm,return -EINTR ; ret =handle_mm_fault (vma, address, fault_flags,NULL ); +mm/gup.c int fixup_user_fault(struct mm_struct *mm, · line 1260
Quoted 10 lines
mm/gup.c +10
-0+if (ret &VM_FAULT_COMPLETED ) { +/* +* NOTE: it's a pity that we need to retake the lock here +* to pair with the unlock() in the callers. Ideally we +* could tell the callers so they do not need to unlock. +*/ +mmap_read_lock (mm); + *unlocked = true; +return 0 ; + }unlocked can be NULL inside the function, yet you assume it's non-NULL here. This is okay because COMPLETED can only be returned if RETRY is set, and when RETRY is set unlocked must be non-NULL. It's correct but not very obvious.
Quoted 3 lines
It might be cleaner to have separate flags for
ALLOW_RETRYandALLOW_UNLOCK, with correspondingVM_FAULT_RETRYandVM_FAULT_UNLOCKED? Even if not all combinations are used.
I can do that, but note that I don't see a major difference even if we're going to introduce ALLOW_UNLOCK. IOW, afaict when we have ALLOW_RETRY then we probably will also have ALLOW_UNLOCK, and vice versa. If that's the case then it kind of loses the benefit of introducing a new flag.
What I want to achieve with this patch is to provide a variance of VM_FAULT_RETRY. I did it because VM_FAULT_RETRY actually contains more than one meaning:
(1) It means we have released the mmap_sem read lock, AND,
(2) It needs the caller to retry one more time on the fault.
Here the tricky thing is (2) depends on (1) so when we have (2) we must have (1).
However it's not the same the other way round: we could have (1) only but without (2), just like what we're doing with fault_dirty_shared_page() right now, where we may only want (1) but we're using (1+2) because we don't have a retval that only contains (1).
So I see the new VM_FAULT_COMPLETED as a new variance of VM_FAULT_RETRY just with condition (2) removed.
Quoted 12 lines
mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 2945
Quoted 9 lines
mm/memory.c +1
-1@@ -2942,7 +2942,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)balance_dirty_pages_ratelimited (mapping);if (fpin) {fput (fpin); -return VM_FAULT_RETRY ; +return VM_FAULT_COMPLETED ;There is one oddity in this now.
Quoted 5 lines
It completes the fault and no longer triggers a retry. Yet it's still using
maybe_unlock_mmap_for_io()and subject to retry limiting. This means that if the fault already retried once, this code won't drop themmap_semto callbalance_dirty_pages()- even though it safely could and should do so, without risking endless retries.
This sounds like another change to me. It's about whether we should consider using maybe_unlock_mmap_for_io() even for after-first page faults if it has retried already.
It looks good to me, but I don't know enough on the rate limiting code path so I can't really provide anything more useful. I do see quite a few other callers of maybe_unlock_mmap_for_io() outside of this call site so we'll want to analyze all of them to make a conclusion, IMHO.
What I can say is if one day we choose to do so then as long as fpin is set then I think we can still safely return the VM_FAULT_COMPLETED here as long as all the assumptions keep the same - firstly we released mmap_sem, then we have finished the current page fault processing as a hint to be delivered upward to the callers.
Thanks,
Signature · 1 line
Peter Xu