mm: Avoid unnecessary page fault retires on shared memory types
Subsystems (25)
Peter Xu
reviewerPATCH v3 1/1 mm: Avoid unnecessary page fault retires on shared memory types
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
Then after that throttling we return VM_FAULT_RETRY.
We did that probably because VM_FAULT_RETRY is the only way we can return to the fault handler at that time telling it we've released the mmap lock.
However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in /proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.
To make it more efficient, add a new VM_FAULT_COMPLETED return code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.
This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed and these are the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)
I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Another thing to mention is that mm_account_fault() does take this new fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping them as-is.
---
v3:
- Rebase to akpm/mm-unstable
- Copy arch maintainers
--- arch/alpha/mm/fault.c | 4 ++++ arch/arc/mm/fault.c | 4 ++++ arch/arm/mm/fault.c | 4 ++++ arch/arm64/mm/fault.c | 4 ++++ arch/csky/mm/fault.c | 4 ++++ arch/hexagon/mm/vm_fault.c | 4 ++++ arch/ia64/mm/fault.c | 4 ++++ arch/m68k/mm/fault.c | 4 ++++ arch/microblaze/mm/fault.c | 4 ++++ arch/mips/mm/fault.c | 4 ++++ arch/nios2/mm/fault.c | 4 ++++ arch/openrisc/mm/fault.c | 4 ++++ arch/parisc/mm/fault.c | 4 ++++ arch/powerpc/mm/copro_fault.c | 5 +++++ arch/powerpc/mm/fault.c | 5 +++++ arch/riscv/mm/fault.c | 4 ++++ arch/s390/mm/fault.c | 12 +++++++++++- arch/sh/mm/fault.c | 4 ++++ arch/sparc/mm/fault_32.c | 4 ++++ arch/sparc/mm/fault_64.c | 5 +++++ arch/um/kernel/trap.c | 4 ++++ arch/x86/mm/fault.c | 4 ++++ arch/xtensa/mm/fault.c | 4 ++++ include/linux/mm_types.h | 2 ++ mm/gup.c | 34 +++++++++++++++++++++++++++++++++- mm/memory.c | 2 +- 26 files changed, 138 insertions(+), 3 deletions(-)
arch/alpha/mm/fault.c +4 -0
@@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/arc/mm/fault.c +4 -0
@@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +/* * Fault retry nuances, mmap_lock already relinquished by core mm */
arch/arm/mm/fault.c +4 -0
@@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)return 0 ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (!(fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
arch/arm64/mm/fault.c +4 -0
@@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,return 0 ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (fault &VM_FAULT_RETRY ) { mm_flags |=FAULT_FLAG_TRIED ; goto retry;
arch/csky/mm/fault.c +4 -0
@@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely ((fault &VM_FAULT_RETRY ) && (flags &FAULT_FLAG_ALLOW_RETRY ))) { flags |=FAULT_FLAG_TRIED ;
arch/hexagon/mm/vm_fault.c +4 -0
@@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +/* The most common case -- we are done. */ if (likely (!(fault &VM_FAULT_ERROR ))) {if (fault &VM_FAULT_RETRY ) {
arch/ia64/mm/fault.c +4 -0
@@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *reif (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {/* * We ran out of memory, or some other thing happened
arch/m68k/mm/fault.c +4 -0
@@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return 0 ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/microblaze/mm/fault.c +4 -0
@@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/mips/mm/fault.c +4 -0
@@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/nios2/mm/fault.c +4 -0
@@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/openrisc/mm/fault.c +4 -0
@@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/parisc/mm/fault.c +4 -0
@@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {/* * We hit a shared mapping outside of the file, or some
arch/powerpc/mm/copro_fault.c +5 -0
@@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea, ret =0 ; *flt =handle_mm_fault (vma, ea, is_write ?FAULT_FLAG_WRITE :0 ,NULL ); + +/* The fault is fully completed (including releasing mmap lock) */ +if (*flt &VM_FAULT_COMPLETED ) +return 0 ; +if (unlikely (*flt &VM_FAULT_ERROR )) {if (*flt &VM_FAULT_OOM ) { ret = -ENOMEM ;
arch/powerpc/mm/fault.c +5 -0
@@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return user_mode (regs) ?0 :SIGBUS ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) + goto out; +/* * Handle the retry right now, the mmap_lock has been released in that * case.
@@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,if (unlikely (fault &VM_FAULT_ERROR ))return mm_fault_error (regs, address, fault); +out:/* * Major/minor page fault accounting. */
arch/riscv/mm/fault.c +4 -0
@@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_RETRY )) { flags |=FAULT_FLAG_TRIED ;
arch/s390/mm/fault.c +11 -1
@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)unsigned long address;unsigned int flags;vm_fault_t fault; +bool need_unlock = true;bool is_write; tsk = current;
@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) goto out_up; goto out; } + +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) { + need_unlock = false; + goto out_gmap; + } +if (unlikely (fault &VM_FAULT_ERROR )) goto out_up;
@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)mmap_read_lock (mm); goto retry; } +out_gmap:if (IS_ENABLED (CONFIG_PGSTE ) && gmap) { address =__gmap_link (gmap, current->thread.gmap_addr, address);
@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) } fault =0 ; out_up: -mmap_read_unlock (mm); +if (need_unlock) +mmap_read_unlock (mm); out:return fault; }
arch/sh/mm/fault.c +4 -0
@@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,if (mm_fault_error (regs, error_code, address, fault))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
arch/sparc/mm/fault_32.c +4 -0
@@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/sparc/mm/fault_64.c +5 -0
@@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)if (fault_signal_pending (fault, regs)) goto exit_exception; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) + goto lock_released; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
@@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs) }mmap_read_unlock (mm); +lock_released: mm_rss =get_mm_rss (mm);#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE /PAGE_SIZE ));
arch/um/kernel/trap.c +4 -0
@@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,if ((fault &VM_FAULT_RETRY ) &&fatal_signal_pending (current)) goto out_nosemaphore; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) { goto out_of_memory;
arch/x86/mm/fault.c +4 -0
@@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +/* * If we need to retry the mmap_lock has already been released, * and if there is a fatal signal pending there is no guarantee
arch/xtensa/mm/fault.c +4 -0
@@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
include/linux/mm_types.h +2 -0
@@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;* @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs * fsync() to complete (for synchronous page faults * in DAX) +* @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released * @VM_FAULT_HINDEX_MASK: mask HINDEX value * */
@@ -746,6 +747,7 @@ enum vm_fault_reason {VM_FAULT_FALLBACK = (__force vm_fault_t )0x000800 ,VM_FAULT_DONE_COW = (__force vm_fault_t )0x001000 ,VM_FAULT_NEEDDSYNC = (__force vm_fault_t )0x002000 , +VM_FAULT_COMPLETED = (__force vm_fault_t )0x004000 , VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000, };
mm/gup.c +33 -1
@@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma, } ret =handle_mm_fault (vma, address, fault_flags,NULL ); + +if (ret &VM_FAULT_COMPLETED ) { +/* +* With FAULT_FLAG_RETRY_NOWAIT we'll never release the +* mmap lock in the page fault handler. Sanity check this. +*/ +WARN_ON_ONCE (fault_flags &FAULT_FLAG_RETRY_NOWAIT ); +if (locked) + *locked =0 ; +/* +* We should do the same as VM_FAULT_RETRY, but let's not +* return -EBUSY since that's not reflecting the reality on +* what has happened - we've just fully completed a page +* fault, with the mmap lock released. Use -EAGAIN to show +* that we want to take the mmap lock _again_. +*/ +return -EAGAIN ; + } +if (ret &VM_FAULT_ERROR ) {int err =vm_fault_to_errno (ret, *flags);
@@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,case 0 : goto retry;case -EBUSY : +case -EAGAIN : ret =0 ; fallthrough;case -EFAULT :
@@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,return -EINTR ; ret =handle_mm_fault (vma, address, fault_flags,NULL ); + +if (ret &VM_FAULT_COMPLETED ) { +/* +* NOTE: it's a pity that we need to retake the lock here +* to pair with the unlock() in the callers. Ideally we +* could tell the callers so they do not need to unlock. +*/ +mmap_read_lock (mm); + *unlocked = true; +return 0 ; + } +if (ret &VM_FAULT_ERROR ) {int err =vm_fault_to_errno (ret,0 );
@@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,/* VM_FAULT_RETRY couldn't trigger, bypass */ return ret; -/* VM_FAULT_RETRY cannot return errors */ +/* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */ if (!*locked) {BUG_ON (ret <0 );BUG_ON (ret >= nr_pages);
mm/memory.c +1 -1
@@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)balance_dirty_pages_ratelimited (mapping);if (fpin) {fput (fpin); -return VM_FAULT_RETRY ; +return VM_FAULT_COMPLETED ; } }
Signature · 1 line
2.32.0
Quoted 44 lines · Peter Xu
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with
balance_dirty_pages_ratelimited()(infault_dirty_shared_page()).Then after that throttling we return
VM_FAULT_RETRY.We did that probably because
VM_FAULT_RETRYis the only way we can return to the fault handler at that time telling it we've released the mmap lock.However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in
/proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.To make it more efficient, add a new
VM_FAULT_COMPLETEDreturn code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed and these are the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Another thing to mention is that
mm_account_fault()does take this new fault as a generic fault to be accounted, unlikeVM_FAULT_RETRY.
Quoted 7 lines
I explicitly didn't touch
hmm_vma_fault()andbreak_ksm()because they do not handleVM_FAULT_RETRYeven with existing code, so I'm literally keeping them as-is.Signed-off-by Peter Xu <[email protected]>arch/m68k/mm/fault.c | 4 ++++
Gr{oetje,eeting}s,
Geert
Quoted 41 lines · Peter Xu
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with
balance_dirty_pages_ratelimited()(infault_dirty_shared_page()).Then after that throttling we return
VM_FAULT_RETRY.We did that probably because
VM_FAULT_RETRYis the only way we can return to the fault handler at that time telling it we've released the mmap lock.However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in
/proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.To make it more efficient, add a new
VM_FAULT_COMPLETEDreturn code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed and these are the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Quoted 8 lines
Another thing to mention is that
mm_account_fault()does take this new fault as a generic fault to be accounted, unlikeVM_FAULT_RETRY.I explicitly didn't touch
hmm_vma_fault()andbreak_ksm()because they do not handleVM_FAULT_RETRYeven with existing code, so I'm literally keeping them as-is.Signed-off-by Peter Xu <[email protected]>
Quoted 41 lines · Peter Xu
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with
balance_dirty_pages_ratelimited()(infault_dirty_shared_page()).Then after that throttling we return
VM_FAULT_RETRY.We did that probably because
VM_FAULT_RETRYis the only way we can return to the fault handler at that time telling it we've released the mmap lock.However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in
/proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.To make it more efficient, add a new
VM_FAULT_COMPLETEDreturn code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed and these are the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Quoted 8 lines
Another thing to mention is that
mm_account_fault()does take this new fault as a generic fault to be accounted, unlikeVM_FAULT_RETRY.I explicitly didn't touch
hmm_vma_fault()andbreak_ksm()because they do not handleVM_FAULT_RETRYeven with existing code, so I'm literally keeping them as-is.Signed-off-by Peter Xu <[email protected]>
Quoted 48 lines · Peter Xu
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with
balance_dirty_pages_ratelimited()(infault_dirty_shared_page()).Then after that throttling we return
VM_FAULT_RETRY.We did that probably because
VM_FAULT_RETRYis the only way we can return to the fault handler at that time telling it we've released the mmap lock.However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in
/proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.To make it more efficient, add a new
VM_FAULT_COMPLETEDreturn code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed and these are the time it needs:
Before: 650.980 ms (+-1.94%) After: 569.396 ms (+-1.38%)I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Another thing to mention is that
mm_account_fault()does take this new fault as a generic fault to be accounted, unlikeVM_FAULT_RETRY.I explicitly didn't touch
hmm_vma_fault()andbreak_ksm()because they do not handleVM_FAULT_RETRYeven with existing code, so I'm literally keeping them as-is.
Quoted 8 lines
Signed-off-by Peter Xu<[email protected]>---
v3:
- Rebase to akpm/mm-unstable
- Copy arch maintainers--- arch/arc/mm/fault.c | 4 ++++
Thx,
-Vineet
For csky part.
Quoted 577 lines · Peter Xu
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with
balance_dirty_pages_ratelimited()(infault_dirty_shared_page()).Then after that throttling we return
VM_FAULT_RETRY.We did that probably because
VM_FAULT_RETRYis the only way we can return to the fault handler at that time telling it we've released the mmap lock.However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in
/proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.To make it more efficient, add a new
VM_FAULT_COMPLETEDreturn code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed and these are the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Another thing to mention is that
mm_account_fault()does take this new fault as a generic fault to be accounted, unlikeVM_FAULT_RETRY.I explicitly didn't touch
hmm_vma_fault()andbreak_ksm()because they do not handleVM_FAULT_RETRYeven with existing code, so I'm literally keeping them as-is.Signed-off-by Peter Xu <[email protected]>---
v3:
- Rebase to akpm/mm-unstable
- Copy arch maintainers--- arch/alpha/mm/fault.c | 4 ++++ arch/arc/mm/fault.c | 4 ++++ arch/arm/mm/fault.c | 4 ++++ arch/arm64/mm/fault.c | 4 ++++ arch/csky/mm/fault.c | 4 ++++ arch/hexagon/mm/vm_fault.c | 4 ++++ arch/ia64/mm/fault.c | 4 ++++ arch/m68k/mm/fault.c | 4 ++++ arch/microblaze/mm/fault.c | 4 ++++ arch/mips/mm/fault.c | 4 ++++ arch/nios2/mm/fault.c | 4 ++++ arch/openrisc/mm/fault.c | 4 ++++ arch/parisc/mm/fault.c | 4 ++++ arch/powerpc/mm/copro_fault.c | 5 +++++ arch/powerpc/mm/fault.c | 5 +++++ arch/riscv/mm/fault.c | 4 ++++ arch/s390/mm/fault.c | 12 +++++++++++- arch/sh/mm/fault.c | 4 ++++ arch/sparc/mm/fault_32.c | 4 ++++ arch/sparc/mm/fault_64.c | 5 +++++ arch/um/kernel/trap.c | 4 ++++ arch/x86/mm/fault.c | 4 ++++ arch/xtensa/mm/fault.c | 4 ++++ include/linux/mm_types.h | 2 ++ mm/gup.c | 34 +++++++++++++++++++++++++++++++++- mm/memory.c | 2 +- 26 files changed, 138 insertions(+), 3 deletions(-)arch/alpha/mm/fault.c +4
-0@@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/arc/mm/fault.c +4
-0@@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +/* * Fault retry nuances, mmap_lock already relinquished by core mm */ arch/arm/mm/fault.c +4
-0@@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)return 0 ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (!(fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;arch/arm64/mm/fault.c +4
-0@@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,return 0 ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (fault &VM_FAULT_RETRY ) { mm_flags |=FAULT_FLAG_TRIED ; goto retry;arch/csky/mm/fault.c +4
-0@@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely ((fault &VM_FAULT_RETRY ) && (flags &FAULT_FLAG_ALLOW_RETRY ))) { flags |=FAULT_FLAG_TRIED ;arch/hexagon/mm/vm_fault.c +4
-0@@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +/* The most common case -- we are done. */ if (likely (!(fault &VM_FAULT_ERROR ))) {if (fault &VM_FAULT_RETRY ) {arch/ia64/mm/fault.c +4
-0@@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *reif (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {/* * We ran out of memory, or some other thing happened arch/m68k/mm/fault.c +4
-0@@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return 0 ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/microblaze/mm/fault.c +4
-0@@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/mips/mm/fault.c +4
-0@@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/nios2/mm/fault.c +4
-0@@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/openrisc/mm/fault.c +4
-0@@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/parisc/mm/fault.c +4
-0@@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {/* * We hit a shared mapping outside of the file, or some arch/powerpc/mm/copro_fault.c +5
-0@@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea, ret =0 ; *flt =handle_mm_fault (vma, ea, is_write ?FAULT_FLAG_WRITE :0 ,NULL ); + +/* The fault is fully completed (including releasing mmap lock) */ +if (*flt &VM_FAULT_COMPLETED ) +return 0 ; +if (unlikely (*flt &VM_FAULT_ERROR )) {if (*flt &VM_FAULT_OOM ) { ret = -ENOMEM ;arch/powerpc/mm/fault.c +5
-0@@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return user_mode (regs) ?0 :SIGBUS ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) + goto out; +/* * Handle the retry right now, the mmap_lock has been released in that * case. @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,if (unlikely (fault &VM_FAULT_ERROR ))return mm_fault_error (regs, address, fault); +out:/* * Major/minor page fault accounting. */ arch/riscv/mm/fault.c +4
-0@@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_RETRY )) { flags |=FAULT_FLAG_TRIED ;arch/s390/mm/fault.c +11
-1@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)unsigned long address;unsigned int flags;vm_fault_t fault; +bool need_unlock = true;bool is_write; tsk = current;@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) goto out_up; goto out; } + +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) { + need_unlock = false; + goto out_gmap; + } +if (unlikely (fault &VM_FAULT_ERROR )) goto out_up;@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)mmap_read_lock (mm); goto retry; } +out_gmap:if (IS_ENABLED (CONFIG_PGSTE ) && gmap) { address =__gmap_link (gmap, current->thread.gmap_addr, address);@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) } fault =0 ; out_up: -mmap_read_unlock (mm); +if (need_unlock) +mmap_read_unlock (mm); out:return fault; }arch/sh/mm/fault.c +4
-0@@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,if (mm_fault_error (regs, error_code, address, fault))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;arch/sparc/mm/fault_32.c +4
-0@@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/sparc/mm/fault_64.c +5
-0@@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)if (fault_signal_pending (fault, regs)) goto exit_exception; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) + goto lock_released; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;@@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs) }mmap_read_unlock (mm); +lock_released: mm_rss =get_mm_rss (mm);#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE /PAGE_SIZE ));arch/um/kernel/trap.c +4
-0@@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,if ((fault &VM_FAULT_RETRY ) &&fatal_signal_pending (current)) goto out_nosemaphore; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) { goto out_of_memory;arch/x86/mm/fault.c +4
-0@@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +/* * If we need to retry the mmap_lock has already been released, * and if there is a fatal signal pending there is no guarantee arch/xtensa/mm/fault.c +4
-0@@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;include/linux/mm_types.h +2
-0@@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;* @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs * fsync() to complete (for synchronous page faults * in DAX) +* @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released * @VM_FAULT_HINDEX_MASK: mask HINDEX value * */ @@ -746,6 +747,7 @@ enum vm_fault_reason {VM_FAULT_FALLBACK = (__force vm_fault_t )0x000800 ,VM_FAULT_DONE_COW = (__force vm_fault_t )0x001000 ,VM_FAULT_NEEDDSYNC = (__force vm_fault_t )0x002000 , +VM_FAULT_COMPLETED = (__force vm_fault_t )0x004000 , VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000, };mm/gup.c +33
-1@@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma, } ret =handle_mm_fault (vma, address, fault_flags,NULL ); + +if (ret &VM_FAULT_COMPLETED ) { +/* +* With FAULT_FLAG_RETRY_NOWAIT we'll never release the +* mmap lock in the page fault handler. Sanity check this. +*/ +WARN_ON_ONCE (fault_flags &FAULT_FLAG_RETRY_NOWAIT ); +if (locked) + *locked =0 ; +/* +* We should do the same as VM_FAULT_RETRY, but let's not +* return -EBUSY since that's not reflecting the reality on +* what has happened - we've just fully completed a page +* fault, with the mmap lock released. Use -EAGAIN to show +* that we want to take the mmap lock _again_. +*/ +return -EAGAIN ; + } +if (ret &VM_FAULT_ERROR ) {int err =vm_fault_to_errno (ret, *flags);@@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,case 0 : goto retry;case -EBUSY : +case -EAGAIN : ret =0 ; fallthrough;case -EFAULT :@@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,return -EINTR ; ret =handle_mm_fault (vma, address, fault_flags,NULL ); + +if (ret &VM_FAULT_COMPLETED ) { +/* +* NOTE: it's a pity that we need to retake the lock here +* to pair with the unlock() in the callers. Ideally we +* could tell the callers so they do not need to unlock. +*/ +mmap_read_lock (mm); + *unlocked = true; +return 0 ; + } +if (ret &VM_FAULT_ERROR ) {int err =vm_fault_to_errno (ret,0 );@@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,/* VM_FAULT_RETRY couldn't trigger, bypass */ return ret; -/* VM_FAULT_RETRY cannot return errors */ +/* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */ if (!*locked) {BUG_ON (ret <0 );BUG_ON (ret >= nr_pages);mm/memory.c +1
-2@@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)balance_dirty_pages_ratelimited (mapping);if (fpin) {fput (fpin); -return VM_FAULT_RETRY ; +return VM_FAULT_COMPLETED ; } } --2.32.0
Signature · 4 lines
Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/
Quoted 49 lines · Peter Xu
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with
balance_dirty_pages_ratelimited()(infault_dirty_shared_page()).Then after that throttling we return
VM_FAULT_RETRY.We did that probably because
VM_FAULT_RETRYis the only way we can return to the fault handler at that time telling it we've released the mmap lock.However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in
/proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.To make it more efficient, add a new
VM_FAULT_COMPLETEDreturn code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed and these are the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Another thing to mention is that
mm_account_fault()does take this new fault as a generic fault to be accounted, unlikeVM_FAULT_RETRY.I explicitly didn't touch
hmm_vma_fault()andbreak_ksm()because they do not handleVM_FAULT_RETRYeven with existing code, so I'm literally keeping them as-is.
Quoted 7 lines
Signed-off-by Peter Xu <[email protected]>---
v3:
- Rebase to akpm/mm-unstable
- Copy arch maintainers
---
For xtensa:
Signature · 2 lines
Thanks. -- Max
Quoted 41 lines · Peter Xu
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with
balance_dirty_pages_ratelimited()(infault_dirty_shared_page()).Then after that throttling we return
VM_FAULT_RETRY.We did that probably because
VM_FAULT_RETRYis the only way we can return to the fault handler at that time telling it we've released the mmap lock.However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in
/proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.To make it more efficient, add a new
VM_FAULT_COMPLETEDreturn code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed and these are the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Quoted 6 lines
Another thing to mention is that
mm_account_fault()does take this new fault as a generic fault to be accounted, unlikeVM_FAULT_RETRY.I explicitly didn't touch
hmm_vma_fault()andbreak_ksm()because they do not handleVM_FAULT_RETRYeven with existing code, so I'm literally keeping them as-is.
I looked at the change generally and in particular for hmm_vma_fault() and didn't see any issues there so feel free to add:
Quoted 524 lines
Signed-off-by Peter Xu <[email protected]>---
v3:
- Rebase to akpm/mm-unstable
- Copy arch maintainers--- arch/alpha/mm/fault.c | 4 ++++ arch/arc/mm/fault.c | 4 ++++ arch/arm/mm/fault.c | 4 ++++ arch/arm64/mm/fault.c | 4 ++++ arch/csky/mm/fault.c | 4 ++++ arch/hexagon/mm/vm_fault.c | 4 ++++ arch/ia64/mm/fault.c | 4 ++++ arch/m68k/mm/fault.c | 4 ++++ arch/microblaze/mm/fault.c | 4 ++++ arch/mips/mm/fault.c | 4 ++++ arch/nios2/mm/fault.c | 4 ++++ arch/openrisc/mm/fault.c | 4 ++++ arch/parisc/mm/fault.c | 4 ++++ arch/powerpc/mm/copro_fault.c | 5 +++++ arch/powerpc/mm/fault.c | 5 +++++ arch/riscv/mm/fault.c | 4 ++++ arch/s390/mm/fault.c | 12 +++++++++++- arch/sh/mm/fault.c | 4 ++++ arch/sparc/mm/fault_32.c | 4 ++++ arch/sparc/mm/fault_64.c | 5 +++++ arch/um/kernel/trap.c | 4 ++++ arch/x86/mm/fault.c | 4 ++++ arch/xtensa/mm/fault.c | 4 ++++ include/linux/mm_types.h | 2 ++ mm/gup.c | 34 +++++++++++++++++++++++++++++++++- mm/memory.c | 2 +- 26 files changed, 138 insertions(+), 3 deletions(-)arch/alpha/mm/fault.c +4
-0@@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/arc/mm/fault.c +4
-0@@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +/* * Fault retry nuances, mmap_lock already relinquished by core mm */ arch/arm/mm/fault.c +4
-0@@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)return 0 ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (!(fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;arch/arm64/mm/fault.c +4
-0@@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,return 0 ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (fault &VM_FAULT_RETRY ) { mm_flags |=FAULT_FLAG_TRIED ; goto retry;arch/csky/mm/fault.c +4
-0@@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely ((fault &VM_FAULT_RETRY ) && (flags &FAULT_FLAG_ALLOW_RETRY ))) { flags |=FAULT_FLAG_TRIED ;arch/hexagon/mm/vm_fault.c +4
-0@@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +/* The most common case -- we are done. */ if (likely (!(fault &VM_FAULT_ERROR ))) {if (fault &VM_FAULT_RETRY ) {arch/ia64/mm/fault.c +4
-0@@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *reif (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {/* * We ran out of memory, or some other thing happened arch/m68k/mm/fault.c +4
-0@@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return 0 ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/microblaze/mm/fault.c +4
-0@@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/mips/mm/fault.c +4
-0@@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/nios2/mm/fault.c +4
-0@@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/openrisc/mm/fault.c +4
-0@@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/parisc/mm/fault.c +4
-0@@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {/* * We hit a shared mapping outside of the file, or some arch/powerpc/mm/copro_fault.c +5
-0@@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea, ret =0 ; *flt =handle_mm_fault (vma, ea, is_write ?FAULT_FLAG_WRITE :0 ,NULL ); + +/* The fault is fully completed (including releasing mmap lock) */ +if (*flt &VM_FAULT_COMPLETED ) +return 0 ; +if (unlikely (*flt &VM_FAULT_ERROR )) {if (*flt &VM_FAULT_OOM ) { ret = -ENOMEM ;arch/powerpc/mm/fault.c +5
-0@@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,if (fault_signal_pending (fault, regs))return user_mode (regs) ?0 :SIGBUS ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) + goto out; +/* * Handle the retry right now, the mmap_lock has been released in that * case. @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,if (unlikely (fault &VM_FAULT_ERROR ))return mm_fault_error (regs, address, fault); +out:/* * Major/minor page fault accounting. */ arch/riscv/mm/fault.c +4
-0@@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_RETRY )) { flags |=FAULT_FLAG_TRIED ;arch/s390/mm/fault.c +11
-1@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)unsigned long address;unsigned int flags;vm_fault_t fault; +bool need_unlock = true;bool is_write; tsk = current;@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) goto out_up; goto out; } + +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) { + need_unlock = false; + goto out_gmap; + } +if (unlikely (fault &VM_FAULT_ERROR )) goto out_up;@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)mmap_read_lock (mm); goto retry; } +out_gmap:if (IS_ENABLED (CONFIG_PGSTE ) && gmap) { address =__gmap_link (gmap, current->thread.gmap_addr, address);@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) } fault =0 ; out_up: -mmap_read_unlock (mm); +if (need_unlock) +mmap_read_unlock (mm); out:return fault; }arch/sh/mm/fault.c +4
-0@@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,if (mm_fault_error (regs, error_code, address, fault))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;arch/sparc/mm/fault_32.c +4
-0@@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,if (fault_signal_pending (fault, regs))return ; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;arch/sparc/mm/fault_64.c +5
-0@@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)if (fault_signal_pending (fault, regs)) goto exit_exception; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) + goto lock_released; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;@@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs) }mmap_read_unlock (mm); +lock_released: mm_rss =get_mm_rss (mm);#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE /PAGE_SIZE ));arch/um/kernel/trap.c +4
-0@@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,if ((fault &VM_FAULT_RETRY ) &&fatal_signal_pending (current)) goto out_nosemaphore; +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return 0 ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) { goto out_of_memory;arch/x86/mm/fault.c +4
-0@@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +/* * If we need to retry the mmap_lock has already been released, * and if there is a fatal signal pending there is no guarantee arch/xtensa/mm/fault.c +4
-0@@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)return ; } +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) +return ; +if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;include/linux/mm_types.h +2
-0@@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;* @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs * fsync() to complete (for synchronous page faults * in DAX) +* @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released * @VM_FAULT_HINDEX_MASK: mask HINDEX value * */ @@ -746,6 +747,7 @@ enum vm_fault_reason {VM_FAULT_FALLBACK = (__force vm_fault_t )0x000800 ,VM_FAULT_DONE_COW = (__force vm_fault_t )0x001000 ,VM_FAULT_NEEDDSYNC = (__force vm_fault_t )0x002000 , +VM_FAULT_COMPLETED = (__force vm_fault_t )0x004000 , VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000, };mm/gup.c +33
-1@@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma, } ret =handle_mm_fault (vma, address, fault_flags,NULL ); + +if (ret &VM_FAULT_COMPLETED ) { +/* +* With FAULT_FLAG_RETRY_NOWAIT we'll never release the +* mmap lock in the page fault handler. Sanity check this. +*/ +WARN_ON_ONCE (fault_flags &FAULT_FLAG_RETRY_NOWAIT ); +if (locked) + *locked =0 ; +/* +* We should do the same as VM_FAULT_RETRY, but let's not +* return -EBUSY since that's not reflecting the reality on +* what has happened - we've just fully completed a page +* fault, with the mmap lock released. Use -EAGAIN to show +* that we want to take the mmap lock _again_. +*/ +return -EAGAIN ; + } +if (ret &VM_FAULT_ERROR ) {int err =vm_fault_to_errno (ret, *flags);@@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,case 0 : goto retry;case -EBUSY : +case -EAGAIN : ret =0 ; fallthrough;case -EFAULT :@@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,return -EINTR ; ret =handle_mm_fault (vma, address, fault_flags,NULL ); + +if (ret &VM_FAULT_COMPLETED ) { +/* +* NOTE: it's a pity that we need to retake the lock here +* to pair with the unlock() in the callers. Ideally we +* could tell the callers so they do not need to unlock. +*/ +mmap_read_lock (mm); + *unlocked = true; +return 0 ; + } +if (ret &VM_FAULT_ERROR ) {int err =vm_fault_to_errno (ret,0 );@@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,/* VM_FAULT_RETRY couldn't trigger, bypass */ return ret; -/* VM_FAULT_RETRY cannot return errors */ +/* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */ if (!*locked) {BUG_ON (ret <0 );BUG_ON (ret >= nr_pages);mm/memory.c +1
-1@@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)balance_dirty_pages_ratelimited (mapping);if (fpin) {fput (fpin); -return VM_FAULT_RETRY ; +return VM_FAULT_COMPLETED ; } }
Quoted 6 lines · Peter Xu
This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed and these are the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)
Nice!
Quoted 1 line
arch/x86/mm/fault.c | 4 ++++
Minor comment typo:
mm/gup.c static int faultin_page(struct vm_area_struct *vma, · line 969
Quoted 7 lines
mm/gup.c +7
-0+/* +* We should do the same as VM_FAULT_RETRY, but let's not +* return -EBUSY since that's not reflecting the reality on +* what has happened - we've just fully completed a page +* fault, with the mmap lock released. Use -EAGAIN to show +* that we want to take the mmap lock _again_. +*/
s/reflecting the reality on what has happened
/reflecting the reality of what has happened
mm/gup.c int fixup_user_fault(struct mm_struct *mm, · line 1335
Quoted 11 lines
mm/gup.c +10
-0ret =handle_mm_fault (vma, address, fault_flags,NULL ); + +if (ret &VM_FAULT_COMPLETED ) { +/* +* NOTE: it's a pity that we need to retake the lock here +* to pair with the unlock() in the callers. Ideally we +* could tell the callers so they do not need to unlock. +*/ +mmap_read_lock (mm); + *unlocked = true; +return 0 ;
Indeed that's a pity - I guess more performance could be gained here, especially in highly parallel threaded workloads?
Thanks,
Ingo
Peter Xu
reviewerQuoted 18 lines · Ingo Molnar
Quoted 6 lines · Peter Xu
This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed and these are the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)Nice!
Quoted 1 line
arch/x86/mm/fault.c | 4 ++++Reviewed-by Ingo Molnar <[email protected]>Minor comment typo:
Quoted 10 lines
mm/gup.c static int faultin_page(struct vm_area_struct *vma, · line 969
Quoted 7 lines
mm/gup.c +7
-0+/* +* We should do the same as VM_FAULT_RETRY, but let's not +* return -EBUSY since that's not reflecting the reality on +* what has happened - we've just fully completed a page +* fault, with the mmap lock released. Use -EAGAIN to show +* that we want to take the mmap lock _again_. +*/ s/reflecting the reality on what has happened
/reflecting the reality of what has happened
Will fix.
Quoted 15 lines
mm/gup.c int fixup_user_fault(struct mm_struct *mm, · line 1335
Quoted 12 lines
mm/gup.c +10
-0ret =handle_mm_fault (vma, address, fault_flags,NULL ); + +if (ret &VM_FAULT_COMPLETED ) { +/* +* NOTE: it's a pity that we need to retake the lock here +* to pair with the unlock() in the callers. Ideally we +* could tell the callers so they do not need to unlock. +*/ +mmap_read_lock (mm); + *unlocked = true; +return 0 ;Indeed that's a pity - I guess more performance could be gained here, especially in highly parallel threaded workloads?
Yes I think so.
The patch avoids the page fault retry, including the mmap lock/unlock side. Now if we retake the lock for fixup_user_fault() we still safe time for pgtable walks but the lock overhead will be somehow kept, just with smaller critical sections.
Some fixup_user_fault() callers won't be affected as long as unlocked==NULL is passed - e.g. the futex code path (fault_in_user_writeable). After all they never needed to retake the lock before/after this patch.
It's about the other callers, and they may need some more touch-ups case by case. Examples are follow_fault_pfn() in vfio and hva_to_pfn_remapped() in
KVM: both of them returns -EAGAIN when *unlocked==true. We need to teach them to know "*unlocked==true" does not necessarily mean a retry attempt.
I think I can look into them if this patch can be accepted as a follow up.
Thanks for taking a look!
Signature · 1 line
Peter Xu
Heiko Carstens
maintainerQuoted 49 lines · Peter Xu
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with
balance_dirty_pages_ratelimited()(infault_dirty_shared_page()).Then after that throttling we return
VM_FAULT_RETRY.We did that probably because
VM_FAULT_RETRYis the only way we can return to the fault handler at that time telling it we've released the mmap lock.However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in
/proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.To make it more efficient, add a new
VM_FAULT_COMPLETEDreturn code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed and these are the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Another thing to mention is that
mm_account_fault()does take this new fault as a generic fault to be accounted, unlikeVM_FAULT_RETRY.I explicitly didn't touch
hmm_vma_fault()andbreak_ksm()because they do not handleVM_FAULT_RETRYeven with existing code, so I'm literally keeping them as-is.Signed-off-by Peter Xu <[email protected]>
...
Quoted 35 lines
arch/s390/mm/fault.c +9
-0@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)unsigned long address;unsigned int flags;vm_fault_t fault; +bool need_unlock = true;bool is_write; tsk = current;@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) goto out_up; goto out; } + +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) { + need_unlock = false; + goto out_gmap; + } +if (unlikely (fault &VM_FAULT_ERROR )) goto out_up;@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)mmap_read_lock (mm); goto retry; } +out_gmap:if (IS_ENABLED (CONFIG_PGSTE ) && gmap) { address =__gmap_link (gmap, current->thread.gmap_addr, address);
arch/s390/mm/fault.c static inline vm_fault_t do_exception(struct pt_regs *regs, int access) · line 480
Quoted 7 lines
arch/s390/mm/fault.c +2
-1@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) } fault =0 ; out_up: -mmap_read_unlock (mm); +if (need_unlock) +mmap_read_unlock (mm); out:
This seems to be incorrect. __gmap_link() requires the mmap_lock to be held. Christian, Janosch, or David, could you please check?
Peter Xu
reviewerHi, Heiko,
Quoted 88 lines · Heiko Carstens
Quoted 49 lines · Peter Xu
I observed that for each of the shared file-backed page faults, we're very likely to retry one more time for the 1st write fault upon no page. It's because we'll need to release the mmap lock for dirty rate limit purpose with
balance_dirty_pages_ratelimited()(infault_dirty_shared_page()).Then after that throttling we return
VM_FAULT_RETRY.We did that probably because
VM_FAULT_RETRYis the only way we can return to the fault handler at that time telling it we've released the mmap lock.However that's not ideal because it's very likely the fault does not need to be retried at all since the pgtable was well installed before the throttling, so the next continuous fault (including taking mmap read lock, walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at "pgfault" value in
/proc/vmstat, then we should expect 2 counts for each shmem write simply because we retried, and vm event "pgfault" will capture that.To make it more efficient, add a new
VM_FAULT_COMPLETEDreturn code just to show that we've completed the whole fault and released the lock. It's also a hint that we should very possibly not need another fault immediately on this page because we've just completed it.This patch provides a ~12% perf boost on my aarch64 test VM with a simple program sequentially dirtying 400MB shmem file being mmap()ed and these are the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap code before returning from pgfault), the rest changes in the page fault handlers should be relatively straightforward.
Another thing to mention is that
mm_account_fault()does take this new fault as a generic fault to be accounted, unlikeVM_FAULT_RETRY.I explicitly didn't touch
hmm_vma_fault()andbreak_ksm()because they do not handleVM_FAULT_RETRYeven with existing code, so I'm literally keeping them as-is.Signed-off-by Peter Xu <[email protected]>...
Quoted 37 lines
arch/s390/mm/fault.c +9
-0@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)unsigned long address;unsigned int flags;vm_fault_t fault; +bool need_unlock = true;bool is_write; tsk = current;@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) goto out_up; goto out; } + +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) { + need_unlock = false; + goto out_gmap; + } +if (unlikely (fault &VM_FAULT_ERROR )) goto out_up;@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)mmap_read_lock (mm); goto retry; } +out_gmap:if (IS_ENABLED (CONFIG_PGSTE ) && gmap) { address =__gmap_link (gmap, current->thread.gmap_addr, address);@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) } fault =0 ;
Quoted 8 lines
arch/s390/mm/fault.c static inline vm_fault_t do_exception(struct pt_regs *regs, int access) · line 480
Quoted 5 lines
arch/s390/mm/fault.c +2
-1out_up: -mmap_read_unlock (mm); +if (need_unlock) +mmap_read_unlock (mm); out:This seems to be incorrect.
__gmap_link()requires themmap_lockto be held. Christian, Janosch, or David, could you please check?
Thanks for pointing that out. Indeed I see the clue right above the comment of __gmap_link():
/*
* ...
* The mmap_lock of the mm that belongs to the address space must be held
* when this function gets called.
*/
int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)
A further fact is it'll walk the pgtable right afterwards, assuming gmap->mm will definitely be the current mm or it'll definitely go wrong.
I'll change s390 to retake the lock with the new COMPLETE retcode, so at least it'll avoid one pgtable work procedure even if the lock overhead was kept.
With that, one more possible further optimization for s390 only will be conditionally not taking that lock when !CONFIG_PGSTE, but I have totally no idea whether that'll be a common use case, so I plan to leave that for later in all cases.
Actually after doing that the whole changeset of s390 is even more straightforward:
---8<---
arch/s390/mm/fault.c +12 -0
@@ -433,6 +433,17 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) goto out_up; goto out; } + +/* The fault is fully completed (including releasing mmap lock) */ +if (fault &VM_FAULT_COMPLETED ) { +/* +* Gmap will need the mmap lock again, so retake it. TODO: +* only conditionally take the lock when CONFIG_PGSTE set. +*/ +mmap_read_lock (mm); + goto out_gmap; + } +if (unlikely (fault &VM_FAULT_ERROR )) goto out_up;
@@ -452,6 +463,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)mmap_read_lock (mm); goto retry; } +out_gmap:if (IS_ENABLED (CONFIG_PGSTE ) && gmap) { address =__gmap_link (gmap, current->thread.gmap_addr, address);
---8<---
Thanks,
Signature · 1 line
Peter Xu