mm: Page fault accounting cleanups
Subsystems (43)
Peter Xu
reviewerv5 cover mm: Page fault accounting cleanups
v5:
- rebase to linux-next/akpm, add r-bs
- patch 1: pass vm_fault_t into mm_account_fault() [DavidR]
- patch 25: fix comment in get_user_pages() to remove the ref to task [John]
- dropped the unicore32 patch because the arch was thrown away
This is v5 of the pf accounting cleanup series. It originates from Gerald Schaefer's report on an issue a week ago regarding to incorrect page fault accountings for retried page fault after commit 4064b9827063 ("mm: allow VM_FAULT_RETRY for multiple times"):
https://lore.kernel.org/lkml/20200610174811.44b94525@thinkpad/
What this series did:
- Correct page fault accounting: we do accounting for a page fault (no matter whether it's from #PF handling, or gup, or anything else) only with the one that completed the fault. For example, page fault retries should not be counted in page fault counters. Same to the perf events.
- Unify definition of PERF_COUNT_SW_PAGE_FAULTS: currently this perf event is used in an adhoc way across different archs.
Case (1): for many archs it's done at the entry of a page fault handler, so that it will also cover e.g. errornous faults.
Case (2): for some other archs, it is only accounted when the page fault is resolved successfully.
Case (3): there're still quite some archs that have not enabled this perf event.
Since this series will touch merely all the archs, we unify this perf event to always follow case (1), which is the one that makes most sense. And since we moved the accounting into handle_mm_fault, the other two MAJ/MIN perf events are well taken care of naturally.
- Unify definition of "major faults": the definition of "major fault" is slightly changed when used in accounting (not VM_FAULT_MAJOR). More information in patch 1.
- Always account the page fault onto the one that triggered the page fault. This does not matter much for #PF handlings, but mostly for gup. More information on this in patch 25.
Patchset layout:
Patch 1: Introduced the accounting in handle_mm_fault(), not enabled. Patch 2-23: Enable the new accounting for arch #PF handlers one by one. Patch 24: Enable the new accounting for the rest outliers (gup, iommu, etc.) Patch 25: Cleanup GUP task_struct pointer since it's not needed any more
For each of the patch that fixes a specific arch, I'm CCing the maintainers and the arch list if there is. Besides, I only lightly tested this series on x86.
Please have a look, thanks.
Peter Xu (25):
mm: Do page fault accounting in handle_mm_fault mm/alpha: Use general page fault accounting mm/arc: Use general page fault accounting mm/arm: Use general page fault accounting mm/arm64: Use general page fault accounting mm/csky: Use general page fault accounting mm/hexagon: Use general page fault accounting mm/ia64: Use general page fault accounting mm/m68k: Use general page fault accounting mm/microblaze: Use general page fault accounting mm/mips: Use general page fault accounting mm/nds32: Use general page fault accounting mm/nios2: Use general page fault accounting mm/openrisc: Use general page fault accounting mm/parisc: Use general page fault accounting mm/powerpc: Use general page fault accounting mm/riscv: Use general page fault accounting mm/s390: Use general page fault accounting mm/sh: Use general page fault accounting mm/sparc32: Use general page fault accounting mm/sparc64: Use general page fault accounting mm/x86: Use general page fault accounting mm/xtensa: Use general page fault accounting
mm: Clean up the last pieces of page fault accountings mm/gup: Remove task_struct pointer for all gup code
arch/alpha/mm/fault.c | 8 +- arch/arc/kernel/process.c | 2 +- arch/arc/mm/fault.c | 18 +-- arch/arm/mm/fault.c | 25 +---- arch/arm64/mm/fault.c | 29 +---- arch/csky/mm/fault.c | 13 +-- arch/hexagon/mm/vm_fault.c | 9 +- arch/ia64/mm/fault.c | 9 +- arch/m68k/mm/fault.c | 14 +-- arch/microblaze/mm/fault.c | 9 +- arch/mips/mm/fault.c | 14 +-- arch/nds32/mm/fault.c | 19 +--- arch/nios2/mm/fault.c | 14 +-- arch/openrisc/mm/fault.c | 9 +- arch/parisc/mm/fault.c | 8 +- arch/powerpc/mm/copro_fault.c | 7 +- arch/powerpc/mm/fault.c | 11 +- arch/riscv/mm/fault.c | 16 +-- arch/s390/kvm/interrupt.c | 2 +- arch/s390/kvm/kvm-s390.c | 2 +- arch/s390/kvm/priv.c | 8 +- arch/s390/mm/fault.c | 16 +-- arch/s390/mm/gmap.c | 4 +- arch/sh/mm/fault.c | 11 +- arch/sparc/mm/fault_32.c | 13 +-- arch/sparc/mm/fault_64.c | 11 +- arch/um/kernel/trap.c | 6 +- arch/x86/mm/fault.c | 17 +-- arch/xtensa/mm/fault.c | 15 +-- drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 2 +- drivers/infiniband/core/umem_odp.c | 2 +- drivers/iommu/amd/iommu_v2.c | 2 +- drivers/iommu/intel/svm.c | 3 +- drivers/vfio/vfio_iommu_type1.c | 4 +- fs/exec.c | 2 +- include/linux/mm.h | 16 +-- kernel/events/uprobes.c | 6 +- kernel/futex.c | 2 +- mm/gup.c | 118 ++++++++------------ mm/hmm.c | 3 +- mm/ksm.c | 3 +- mm/memory.c | 69 +++++++++++- mm/process_vm_access.c | 2 +- security/tomoyo/domain.c | 2 +- virt/kvm/async_pf.c | 2 +- virt/kvm/kvm_main.c | 2 +- 46 files changed, 219 insertions(+), 360 deletions(-)
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 1/25 mm: Do page fault accounting in handle_mm_fault
This is a preparation patch to move page fault accountings into the general code in handle_mm_fault(). This includes both the per task flt_maj/flt_min counters, and the major/minor page fault perf events. To do this, the pt_regs pointer is passed into handle_mm_fault().
PERF_COUNT_SW_PAGE_FAULTS should still be kept in per-arch page fault handlers.
So far, all the pt_regs pointer that passed into handle_mm_fault() is NULL, which means this patch should have no intented functional change.
--- arch/alpha/mm/fault.c | 2 +- arch/arc/mm/fault.c | 2 +- arch/arm/mm/fault.c | 2 +- arch/arm64/mm/fault.c | 2 +- arch/csky/mm/fault.c | 3 +- arch/hexagon/mm/vm_fault.c | 2 +- arch/ia64/mm/fault.c | 2 +- arch/m68k/mm/fault.c | 2 +- arch/microblaze/mm/fault.c | 2 +- arch/mips/mm/fault.c | 2 +- arch/nds32/mm/fault.c | 2 +- arch/nios2/mm/fault.c | 2 +- arch/openrisc/mm/fault.c | 2 +- arch/parisc/mm/fault.c | 2 +- arch/powerpc/mm/copro_fault.c | 2 +- arch/powerpc/mm/fault.c | 2 +- arch/riscv/mm/fault.c | 2 +- arch/s390/mm/fault.c | 2 +- arch/sh/mm/fault.c | 2 +- arch/sparc/mm/fault_32.c | 4 +-- arch/sparc/mm/fault_64.c | 2 +- arch/um/kernel/trap.c | 2 +- arch/x86/mm/fault.c | 2 +- arch/xtensa/mm/fault.c | 2 +- drivers/iommu/amd/iommu_v2.c | 2 +- drivers/iommu/intel/svm.c | 3 +- include/linux/mm.h | 7 ++-- mm/gup.c | 4 +-- mm/hmm.c | 3 +- mm/ksm.c | 3 +- mm/memory.c | 64 ++++++++++++++++++++++++++++++++++- 31 files changed, 103 insertions(+), 34 deletions(-)
arch/alpha/mm/fault.c +1 -1
@@ -148,7 +148,7 @@ do_page_fault(unsigned long address, unsigned long mmcsr,/* If for any reason at all we couldn't handle the fault, make sure we exit gracefully rather than endlessly redo the fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (fault_signal_pending (fault, regs))return ;
arch/arc/mm/fault.c +1 -1
@@ -130,7 +130,7 @@ void do_page_fault(unsigned long address, struct pt_regs *regs) goto bad_area; } - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );/* Quick path to respond to signals */ if (fault_signal_pending (fault, regs)) {
arch/arm/mm/fault.c +1 -1
@@ -224,7 +224,7 @@ __do_page_fault(struct mm_struct *mm, unsigned long addr, unsigned int fsr, goto out; } -return handle_mm_fault (vma, addr &PAGE_MASK , flags); +return handle_mm_fault (vma, addr &PAGE_MASK , flags,NULL ); check_stack:/* Don't allow expansion below FIRST_USER_ADDRESS */
arch/arm64/mm/fault.c +1 -1
@@ -428,7 +428,7 @@ static vm_fault_t __do_page_fault(struct mm_struct *mm, unsigned long addr,*/ if (!(vma->vm_flags & vm_flags))return VM_FAULT_BADACCESS ; -return handle_mm_fault (vma, addr &PAGE_MASK , mm_flags); +return handle_mm_fault (vma, addr &PAGE_MASK , mm_flags,NULL ); }static bool is_el0_instruction_abort (unsigned int esr)
arch/csky/mm/fault.c +2 -1
@@ -150,7 +150,8 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long write,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, write ?FAULT_FLAG_WRITE :0 ); + fault =handle_mm_fault (vma, address, write ?FAULT_FLAG_WRITE :0 , +NULL );if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
arch/hexagon/mm/vm_fault.c +1 -1
@@ -88,7 +88,7 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)break ; } - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (fault_signal_pending (fault, regs))return ;
arch/ia64/mm/fault.c +1 -1
@@ -143,7 +143,7 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re* sure we exit gracefully rather than endlessly redo the * fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (fault_signal_pending (fault, regs))return ;
arch/m68k/mm/fault.c +1 -1
@@ -134,7 +134,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,* the fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );pr_debug ("handle_mm_fault returns %x\n" , fault);if (fault_signal_pending (fault, regs))
arch/microblaze/mm/fault.c +1 -1
@@ -214,7 +214,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (fault_signal_pending (fault, regs))return ;
arch/mips/mm/fault.c +1 -1
@@ -152,7 +152,7 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (fault_signal_pending (fault, regs))return ;
arch/nds32/mm/fault.c +1 -1
@@ -206,7 +206,7 @@ void do_page_fault(unsigned long entry, unsigned long addr,* the fault. */ - fault =handle_mm_fault (vma, addr, flags); + fault =handle_mm_fault (vma, addr, flags,NULL );/* * If we need to retry but a fatal signal is pending, handle the
arch/nios2/mm/fault.c +1 -1
@@ -131,7 +131,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (fault_signal_pending (fault, regs))return ;
arch/openrisc/mm/fault.c +1 -1
@@ -159,7 +159,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,* the fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (fault_signal_pending (fault, regs))return ;
arch/parisc/mm/fault.c +1 -1
@@ -302,7 +302,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,* fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (fault_signal_pending (fault, regs))return ;
arch/powerpc/mm/copro_fault.c +1 -1
@@ -64,7 +64,7 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea, } ret =0 ; - *flt =handle_mm_fault (vma, ea, is_write ?FAULT_FLAG_WRITE :0 ); + *flt =handle_mm_fault (vma, ea, is_write ?FAULT_FLAG_WRITE :0 ,NULL );if (unlikely (*flt &VM_FAULT_ERROR )) {if (*flt &VM_FAULT_OOM ) { ret = -ENOMEM ;
arch/powerpc/mm/fault.c +1 -1
@@ -607,7 +607,7 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL ); major |= fault &VM_FAULT_MAJOR ;
arch/riscv/mm/fault.c +1 -1
@@ -109,7 +109,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, addr, flags); + fault =handle_mm_fault (vma, addr, flags,NULL );/* * If we need to retry but a fatal signal is pending, handle the
arch/s390/mm/fault.c +1 -1
@@ -478,7 +478,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (fault_signal_pending (fault, regs)) { fault =VM_FAULT_SIGNAL ;if (flags &FAULT_FLAG_RETRY_NOWAIT )
arch/sh/mm/fault.c +1 -1
@@ -482,7 +482,7 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (unlikely (fault & (VM_FAULT_RETRY |VM_FAULT_ERROR )))if (mm_fault_error (regs, error_code, address, fault))
arch/sparc/mm/fault_32.c +2 -2
@@ -234,7 +234,7 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (fault_signal_pending (fault, regs))return ;
@@ -410,7 +410,7 @@ static void force_user_fault(unsigned long address, int write)if (!(vma->vm_flags & (VM_READ |VM_EXEC ))) goto bad_area; } -switch (handle_mm_fault (vma, address, flags)) { +switch (handle_mm_fault (vma, address, flags,NULL )) {case VM_FAULT_SIGBUS :case VM_FAULT_OOM : goto do_sigbus;
arch/sparc/mm/fault_64.c +1 -1
@@ -422,7 +422,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs) goto bad_area; } - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (fault_signal_pending (fault, regs)) goto exit_exception;
arch/um/kernel/trap.c +1 -1
@@ -71,7 +71,7 @@ int handle_page_fault(unsigned long address, unsigned long ip, do { vm_fault_t fault; - fault = handle_mm_fault(vma, address, flags); + fault = handle_mm_fault(vma, address, flags, NULL); if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current)) goto out_nosemaphore;
arch/x86/mm/fault.c +1 -1
@@ -1291,7 +1291,7 @@ void do_user_addr_fault(struct pt_regs *regs,* userland). The return to userland is identified whenever * FAULT_FLAG_USER|FAULT_FLAG_KILLABLE are both set in flags. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL ); major |= fault &VM_FAULT_MAJOR ;/* Quick path to respond to signals */
arch/xtensa/mm/fault.c +1 -1
@@ -107,7 +107,7 @@ void do_page_fault(struct pt_regs *regs)* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags); + fault =handle_mm_fault (vma, address, flags,NULL );if (fault_signal_pending (fault, regs))return ;
drivers/iommu/amd/iommu_v2.c +1 -1
@@ -495,7 +495,7 @@ static void do_fault(struct work_struct *work)if (access_error (vma, fault)) goto out; - ret =handle_mm_fault (vma, address, flags); + ret =handle_mm_fault (vma, address, flags,NULL ); out:mmap_read_unlock (mm);
drivers/iommu/intel/svm.c +2 -1
@@ -872,7 +872,8 @@ static irqreturn_t prq_event_thread(int irq, void *d) goto invalid; ret =handle_mm_fault (vma, address, - req->wr_req ?FAULT_FLAG_WRITE :0 ); + req->wr_req ?FAULT_FLAG_WRITE :0 , +NULL );if (ret &VM_FAULT_ERROR ) goto invalid;
include/linux/mm.h +5 -2
@@ -39,6 +39,7 @@ struct file_ra_state;struct user_struct ;struct writeback_control ;struct bdi_writeback ; +struct pt_regs ;void init_mm_internals (void );
@@ -1659,7 +1660,8 @@ int invalidate_inode_page(struct page *page);#ifdef CONFIG_MMU extern vm_fault_t handle_mm_fault (struct vm_area_struct *vma, -unsigned long address,unsigned int flags); +unsigned long address,unsigned int flags, +struct pt_regs *regs);extern int fixup_user_fault (struct task_struct *tsk,struct mm_struct *mm,unsigned long address,unsigned int fault_flags,bool *unlocked);
@@ -1669,7 +1671,8 @@ void unmap_mapping_range(struct address_space *mapping, loff_t const holebegin, loff_t const holelen, int even_cows);#else static inline vm_fault_t handle_mm_fault (struct vm_area_struct *vma, -unsigned long address,unsigned int flags) +unsigned long address,unsigned int flags, +struct pt_regs *regs) {/* should never happen if there's no MMU */ BUG ();
mm/gup.c +2 -2
@@ -884,7 +884,7 @@ static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma, fault_flags |=FAULT_FLAG_TRIED ; } - ret =handle_mm_fault (vma, address, fault_flags); + ret =handle_mm_fault (vma, address, fault_flags,NULL );if (ret &VM_FAULT_ERROR ) {int err =vm_fault_to_errno (ret, *flags);
@@ -1238,7 +1238,7 @@ int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm, fatal_signal_pending(current))return -EINTR ; - ret =handle_mm_fault (vma, address, fault_flags); + ret =handle_mm_fault (vma, address, fault_flags,NULL ); major |= ret &VM_FAULT_MAJOR ;if (ret &VM_FAULT_ERROR ) {int err =vm_fault_to_errno (ret,0 );
mm/hmm.c +2 -1
@@ -75,7 +75,8 @@ static int hmm_vma_fault(unsigned long addr, unsigned long end, }for (; addr < end; addr +=PAGE_SIZE ) -if (handle_mm_fault (vma, addr, fault_flags) &VM_FAULT_ERROR ) +if (handle_mm_fault (vma, addr, fault_flags,NULL ) & +VM_FAULT_ERROR )return -EFAULT ;return -EBUSY ; }
mm/ksm.c +2 -1
@@ -480,7 +480,8 @@ static int break_ksm(struct vm_area_struct *vma, unsigned long addr)break ;if (PageKsm (page)) ret =handle_mm_fault (vma, addr, -FAULT_FLAG_WRITE |FAULT_FLAG_REMOTE ); +FAULT_FLAG_WRITE |FAULT_FLAG_REMOTE , +NULL );else ret =VM_FAULT_WRITE ;put_page (page);
mm/memory.c +63 -1
88 lines not on this page · raw message
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 2/25 mm/alpha: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault().
Add the missing PERF_COUNT_SW_PAGE_FAULTS perf events too. Note, the other two perf events (PERF_COUNT_SW_PAGE_FAULTS_[MAJ|MIN]) were done in handle_mm_fault().
--- arch/alpha/mm/fault.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
arch/alpha/mm/fault.c +3 -5
@@ -25,6 +25,7 @@#include <linux/interrupt.h> #include <linux/extable.h> #include <linux/uaccess.h> +#include <linux/perf_event.h> extern void die_if_kernel (char *,struct pt_regs *,long ,unsigned long *);
@@ -116,6 +117,7 @@ do_page_fault(unsigned long address, unsigned long mmcsr,#endif if (user_mode (regs)) flags |=FAULT_FLAG_USER ; +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); retry:mmap_read_lock (mm); vma =find_vma (mm, address);
@@ -148,7 +150,7 @@ do_page_fault(unsigned long address, unsigned long mmcsr,/* If for any reason at all we couldn't handle the fault, make sure we exit gracefully rather than endlessly redo the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs))return ;
@@ -164,10 +166,6 @@ do_page_fault(unsigned long address, unsigned long mmcsr, }if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) - current->maj_flt++; -else - current->min_flt++;if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 3/25 mm/arc: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
Fix PERF_COUNT_SW_PAGE_FAULTS perf event manually for page fault retries, by moving it before taking mmap_sem.
--- arch/arc/mm/fault.c | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-)
arch/arc/mm/fault.c +3 -15
@@ -105,6 +105,7 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)if (write) flags |=FAULT_FLAG_WRITE ; +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); retry:mmap_read_lock (mm);
@@ -130,7 +131,7 @@ void do_page_fault(unsigned long address, struct pt_regs *regs) goto bad_area; } - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);/* Quick path to respond to signals */ if (fault_signal_pending (fault, regs)) {
@@ -155,22 +156,9 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)* Major/minor page fault accounting * (in case of retry we only land here once) */ -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); - -if (likely (!(fault &VM_FAULT_ERROR ))) { -if (fault &VM_FAULT_MAJOR ) { - tsk->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , - regs, address); - }else { - tsk->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , - regs, address); - } - +if (likely (!(fault &VM_FAULT_ERROR )))/* Normal return path: fault Handled Gracefully */ return ; - }if (!user_mode (regs)) goto no_context;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 4/25 mm/arm: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened. To do this, we need to pass the pt_regs pointer into __do_page_fault().
Fix PERF_COUNT_SW_PAGE_FAULTS perf event manually for page fault retries, by moving it before taking mmap_sem.
--- arch/arm/mm/fault.c | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-)
arch/arm/mm/fault.c +6 -19
@@ -202,7 +202,8 @@ static inline bool access_error(unsigned int fsr, struct vm_area_struct *vma)static vm_fault_t __kprobes __do_page_fault (struct mm_struct *mm,unsigned long addr,unsigned int fsr, -unsigned int flags,struct task_struct *tsk) +unsigned int flags,struct task_struct *tsk, +struct pt_regs *regs) {struct vm_area_struct *vma;vm_fault_t fault;
@@ -224,7 +225,7 @@ __do_page_fault(struct mm_struct *mm, unsigned long addr, unsigned int fsr, goto out; } -return handle_mm_fault (vma, addr &PAGE_MASK , flags,NULL ); +return handle_mm_fault (vma, addr &PAGE_MASK , flags, regs); check_stack:/* Don't allow expansion below FIRST_USER_ADDRESS */
@@ -266,6 +267,8 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)if ((fsr &FSR_WRITE ) && !(fsr &FSR_CM )) flags |=FAULT_FLAG_WRITE ; +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, addr); +/* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code,
@@ -290,7 +293,7 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)#endif } - fault =__do_page_fault (mm, addr, fsr, flags, tsk); + fault =__do_page_fault (mm, addr, fsr, flags, tsk, regs);/* If we need to retry but a fatal signal is pending, handle the * signal first. We do not need to release the mmap_lock because
@@ -302,23 +305,7 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)return 0 ; } -/* -* Major/minor page fault accounting is only done on the -* initial attempt. If we go through a retry, it is extremely -* likely that the page will be found in page cache at that point. -*/ - -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, addr);if (!(fault &VM_FAULT_ERROR ) && flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) { - tsk->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , - regs, addr); - }else { - tsk->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , - regs, addr); - }if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ; goto retry;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 5/25 mm/arm64: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened. To do this, we pass pt_regs pointer into __do_page_fault().
--- arch/arm64/mm/fault.c | 29 ++++++----------------------- 1 file changed, 6 insertions(+), 23 deletions(-)
arch/arm64/mm/fault.c +6 -23
@@ -404,7 +404,8 @@ static void do_bad_area(unsigned long addr, unsigned int esr, struct pt_regs *re#define VM_FAULT_BADACCESS 0x020000static vm_fault_t __do_page_fault (struct mm_struct *mm,unsigned long addr, -unsigned int mm_flags,unsigned long vm_flags) +unsigned int mm_flags,unsigned long vm_flags, +struct pt_regs *regs) {struct vm_area_struct *vma =find_vma (mm, addr);
@@ -428,7 +429,7 @@ static vm_fault_t __do_page_fault(struct mm_struct *mm, unsigned long addr,*/ if (!(vma->vm_flags & vm_flags))return VM_FAULT_BADACCESS ; -return handle_mm_fault (vma, addr &PAGE_MASK , mm_flags,NULL ); +return handle_mm_fault (vma, addr &PAGE_MASK , mm_flags, regs); }static bool is_el0_instruction_abort (unsigned int esr)
@@ -450,7 +451,7 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, {const struct fault_info *inf;struct mm_struct *mm = current->mm; -vm_fault_t fault, major =0 ; +vm_fault_t fault;unsigned long vm_flags =VM_ACCESS_FLAGS ;unsigned int mm_flags =FAULT_FLAG_DEFAULT ;
@@ -516,8 +517,7 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr,#endif } - fault =__do_page_fault (mm, addr, mm_flags, vm_flags); - major |= fault &VM_FAULT_MAJOR ; + fault =__do_page_fault (mm, addr, mm_flags, vm_flags, regs);/* Quick path to respond to signals */ if (fault_signal_pending (fault, regs)) {
@@ -538,25 +538,8 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr,* Handle the "normal" (no error) case first. */ if (likely (!(fault & (VM_FAULT_ERROR |VM_FAULT_BADMAP | -VM_FAULT_BADACCESS )))) { -/* -* Major/minor page fault accounting is only done -* once. If we go through a retry, it is extremely -* likely that the page will be found in page cache at -* that point. -*/ -if (major) { - current->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , regs, - addr); - }else { - current->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , regs, - addr); - } - +VM_FAULT_BADACCESS ))))return 0 ; - }/* * If we are in kernel mode at this point, we have no context to
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 6/25 mm/csky: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
--- arch/csky/mm/fault.c | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-)
arch/csky/mm/fault.c +1 -11
@@ -151,7 +151,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long write,* the fault. */ fault =handle_mm_fault (vma, address, write ?FAULT_FLAG_WRITE :0 , -NULL ); + regs);if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
@@ -161,16 +161,6 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long write, goto bad_area;BUG (); } -if (fault &VM_FAULT_MAJOR ) { - tsk->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , regs, - address); - }else { - tsk->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , regs, - address); - } -mmap_read_unlock (mm);return ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 7/25 mm/hexagon: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
Add the missing PERF_COUNT_SW_PAGE_FAULTS perf events too. Note, the other two perf events (PERF_COUNT_SW_PAGE_FAULTS_[MAJ|MIN]) were done in handle_mm_fault().
--- arch/hexagon/mm/vm_fault.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
arch/hexagon/mm/vm_fault.c +4 -5
@@ -18,6 +18,7 @@#include <linux/signal.h> #include <linux/extable.h> #include <linux/hardirq.h> +#include <linux/perf_event.h> /* * Decode of hardware exception sends us to one of several
@@ -53,6 +54,8 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)if (user_mode (regs)) flags |=FAULT_FLAG_USER ; + +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); retry:mmap_read_lock (mm); vma =find_vma (mm, address);
@@ -88,7 +91,7 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)break ; } - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs))return ;
@@ -96,10 +99,6 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)/* The most common case -- we are done. */ if (likely (!(fault &VM_FAULT_ERROR ))) {if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) - current->maj_flt++; -else - current->min_flt++;if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ; goto retry;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 8/25 mm/ia64: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
Add the missing PERF_COUNT_SW_PAGE_FAULTS perf events too. Note, the other two perf events (PERF_COUNT_SW_PAGE_FAULTS_[MAJ|MIN]) were done in handle_mm_fault().
--- arch/ia64/mm/fault.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
arch/ia64/mm/fault.c +4 -5
@@ -14,6 +14,7 @@#include <linux/kdebug.h> #include <linux/prefetch.h> #include <linux/uaccess.h> +#include <linux/perf_event.h> #include <asm/processor.h> #include <asm/exception.h>
@@ -105,6 +106,8 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re flags |=FAULT_FLAG_USER ;if (mask &VM_WRITE ) flags |=FAULT_FLAG_WRITE ; + +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); retry:mmap_read_lock (mm);
@@ -143,7 +146,7 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re* sure we exit gracefully rather than endlessly redo the * fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs))return ;
@@ -166,10 +169,6 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re }if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) - current->maj_flt++; -else - current->min_flt++;if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 9/25 mm/m68k: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
Add the missing PERF_COUNT_SW_PAGE_FAULTS perf events too. Note, the other two perf events (PERF_COUNT_SW_PAGE_FAULTS_[MAJ|MIN]) were done in handle_mm_fault().
--- arch/m68k/mm/fault.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-)
arch/m68k/mm/fault.c +4 -10
@@ -12,6 +12,7 @@#include <linux/interrupt.h> #include <linux/module.h> #include <linux/uaccess.h> +#include <linux/perf_event.h> #include <asm/setup.h> #include <asm/traps.h>
@@ -84,6 +85,8 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,if (user_mode (regs)) flags |=FAULT_FLAG_USER ; + +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); retry:mmap_read_lock (mm);
@@ -134,7 +137,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,* the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);pr_debug ("handle_mm_fault returns %x\n" , fault);if (fault_signal_pending (fault, regs))
@@ -150,16 +153,7 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,BUG (); } -/* -* Major/minor page fault accounting is only done on the -* initial attempt. If we go through a retry, it is extremely -* likely that the page will be found in page cache at that point. -*/ if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) - current->maj_flt++; -else - current->min_flt++;if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 10/25 mm/microblaze: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
Add the missing PERF_COUNT_SW_PAGE_FAULTS perf events too. Note, the other two perf events (PERF_COUNT_SW_PAGE_FAULTS_[MAJ|MIN]) were done in handle_mm_fault().
--- arch/microblaze/mm/fault.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
arch/microblaze/mm/fault.c +4 -5
@@ -28,6 +28,7 @@#include <linux/mman.h> #include <linux/mm.h> #include <linux/interrupt.h> +#include <linux/perf_event.h> #include <asm/page.h> #include <asm/mmu.h>
@@ -121,6 +122,8 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,if (user_mode (regs)) flags |=FAULT_FLAG_USER ; +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); +/* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an
@@ -214,7 +217,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs))return ;
@@ -230,10 +233,6 @@ void do_page_fault(struct pt_regs *regs, unsigned long address, }if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (unlikely (fault &VM_FAULT_MAJOR )) - current->maj_flt++; -else - current->min_flt++;if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 11/25 mm/mips: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
Fix PERF_COUNT_SW_PAGE_FAULTS perf event manually for page fault retries, by moving it before taking mmap_sem.
--- arch/mips/mm/fault.c | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-)
arch/mips/mm/fault.c +3 -11
@@ -96,6 +96,8 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,if (user_mode (regs)) flags |=FAULT_FLAG_USER ; + +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); retry:mmap_read_lock (mm); vma =find_vma (mm, address);
@@ -152,12 +154,11 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs))return ; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address);if (unlikely (fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_OOM ) goto out_of_memory;
@@ -168,15 +169,6 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,BUG (); }if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) { -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , - regs, address); - tsk->maj_flt++; - }else { -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , - regs, address); - tsk->min_flt++; - }if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 12/25 mm/nds32: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
Fix PERF_COUNT_SW_PAGE_FAULTS perf event manually for page fault retries, by moving it before taking mmap_sem.
--- arch/nds32/mm/fault.c | 19 +++---------------- 1 file changed, 3 insertions(+), 16 deletions(-)
arch/nds32/mm/fault.c +3 -16
@@ -121,6 +121,8 @@ void do_page_fault(unsigned long entry, unsigned long addr,if (unlikely (faulthandler_disabled () || !mm)) goto no_context; +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, addr); +/* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code,
@@ -206,7 +208,7 @@ void do_page_fault(unsigned long entry, unsigned long addr,* the fault. */ - fault =handle_mm_fault (vma, addr, flags,NULL ); + fault =handle_mm_fault (vma, addr, flags, regs);/* * If we need to retry but a fatal signal is pending, handle the
@@ -228,22 +230,7 @@ void do_page_fault(unsigned long entry, unsigned long addr, goto bad_area; } -/* -* Major/minor page fault accounting is only done on the initial -* attempt. If we go through a retry, it is extremely likely that the -* page will be found in page cache at that point. -*/ -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, addr);if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) { - tsk->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ , -1 , regs, addr); - }else { - tsk->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN , -1 , regs, addr); - }if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 13/25 mm/nios2: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
Add the missing PERF_COUNT_SW_PAGE_FAULTS perf events too. Note, the other two perf events (PERF_COUNT_SW_PAGE_FAULTS_[MAJ|MIN]) were done in handle_mm_fault().
--- arch/nios2/mm/fault.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-)
arch/nios2/mm/fault.c +4 -10
@@ -24,6 +24,7 @@#include <linux/mm.h> #include <linux/extable.h> #include <linux/uaccess.h> +#include <linux/perf_event.h> #include <asm/mmu_context.h> #include <asm/traps.h>
@@ -83,6 +84,8 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,if (user_mode (regs)) flags |=FAULT_FLAG_USER ; +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); +if (!mmap_read_trylock (mm)) {if (!user_mode (regs) && !search_exception_tables (regs->ea)) goto bad_area_nosemaphore;
@@ -131,7 +134,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs))return ;
@@ -146,16 +149,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,BUG (); } -/* -* Major/minor page fault accounting is only done on the -* initial attempt. If we go through a retry, it is extremely -* likely that the page will be found in page cache at that point. -*/ if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) - current->maj_flt++; -else - current->min_flt++;if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 14/25 mm/openrisc: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
Add the missing PERF_COUNT_SW_PAGE_FAULTS perf events too. Note, the other two perf events (PERF_COUNT_SW_PAGE_FAULTS_[MAJ|MIN]) were done in handle_mm_fault().
--- arch/openrisc/mm/fault.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
arch/openrisc/mm/fault.c +4 -5
@@ -15,6 +15,7 @@#include <linux/interrupt.h> #include <linux/extable.h> #include <linux/sched/signal.h> +#include <linux/perf_event.h> #include <linux/uaccess.h> #include <asm/siginfo.h>
@@ -103,6 +104,8 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,if (in_interrupt () || !mm) goto no_context; +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); + retry:mmap_read_lock (mm); vma =find_vma (mm, address);
@@ -159,7 +162,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,* the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs))return ;
@@ -176,10 +179,6 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,if (flags &FAULT_FLAG_ALLOW_RETRY ) {/*RGD modeled on Cris */ -if (fault &VM_FAULT_MAJOR ) - tsk->maj_flt++; -else - tsk->min_flt++;if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 15/25 mm/parisc: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
Add the missing PERF_COUNT_SW_PAGE_FAULTS perf events too. Note, the other two perf events (PERF_COUNT_SW_PAGE_FAULTS_[MAJ|MIN]) were done in handle_mm_fault().
--- arch/parisc/mm/fault.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
arch/parisc/mm/fault.c +3 -5
@@ -18,6 +18,7 @@#include <linux/extable.h> #include <linux/uaccess.h> #include <linux/hugetlb.h> +#include <linux/perf_event.h> #include <asm/traps.h>
@@ -281,6 +282,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long code, acc_type =parisc_acctyp (code, regs->iir);if (acc_type &VM_WRITE ) flags |=FAULT_FLAG_WRITE ; +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); retry:mmap_read_lock (mm); vma =find_vma_prev (mm, address, &prev_vma);
@@ -302,7 +304,7 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,* fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs))return ;
@@ -323,10 +325,6 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,BUG (); }if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) - current->maj_flt++; -else - current->min_flt++;if (fault &VM_FAULT_RETRY ) {/* * No need to mmap_read_unlock(mm) as we would
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 16/25 mm/powerpc: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault().
--- arch/powerpc/mm/fault.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
arch/powerpc/mm/fault.c +3 -8
@@ -607,7 +607,7 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs); major |= fault &VM_FAULT_MAJOR ;
@@ -633,14 +633,9 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address,/* * Major/minor page fault accounting. */ -if (major) { - current->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , regs, address); +if (major)cmo_account_page_fault (); - }else { - current->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , regs, address); - } +return 0 ; }NOKPROBE_SYMBOL (__do_page_fault);
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 17/25 mm/riscv: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
--- arch/riscv/mm/fault.c | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-)
arch/riscv/mm/fault.c +1 -15
@@ -109,7 +109,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, addr, flags,NULL ); + fault =handle_mm_fault (vma, addr, flags, regs);/* * If we need to retry but a fatal signal is pending, handle the
@@ -127,21 +127,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)BUG (); } -/* -* Major/minor page fault accounting is only done on the -* initial attempt. If we go through a retry, it is extremely -* likely that the page will be found in page cache at that point. -*/ if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) { - tsk->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ , -1 , regs, addr); - }else { - tsk->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN , -1 , regs, addr); - }if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 18/25 mm/s390: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
--- arch/s390/mm/fault.c | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-)
arch/s390/mm/fault.c +1 -15
@@ -478,7 +478,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs)) { fault =VM_FAULT_SIGNAL ;if (flags &FAULT_FLAG_RETRY_NOWAIT )
@@ -488,21 +488,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)if (unlikely (fault &VM_FAULT_ERROR )) goto out_up; -/* -* Major/minor page fault accounting is only done on the -* initial attempt. If we go through a retry, it is extremely -* likely that the page will be found in page cache at that point. -*/ if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) { - tsk->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , - regs, address); - }else { - tsk->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , - regs, address); - }if (fault &VM_FAULT_RETRY ) {if (IS_ENABLED (CONFIG_PGSTE ) && gmap && (flags &FAULT_FLAG_RETRY_NOWAIT )) {
Signature · 1 line
2.26.2
Alexander Gordeev
maintainerRe: [PATCH v5 18/25] mm/s390: Use general page fault accounting
Quoted 29 lines · Peter Xu
Use the general page fault accounting by passing regs into
handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.CC Heiko Carstens <[email protected]>CC Vasily Gorbik <[email protected]>CC Christian Borntraeger <[email protected]>Reviewed-by Gerald Schaefer <[email protected]>Acked-by Gerald Schaefer <[email protected]>Signed-off-by Peter Xu <[email protected]>--- arch/s390/mm/fault.c | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-)arch/s390/mm/fault.c +1
-1@@ -478,7 +478,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs)) { fault =VM_FAULT_SIGNAL ;if (flags &FAULT_FLAG_RETRY_NOWAIT )
arch/s390/mm/fault.c static inline vm_fault_t do_exception(struct pt_regs *regs, int access) · line 491
Quoted 9 lines
arch/s390/mm/fault.c +0
-5@@ -488,21 +488,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)if (unlikely (fault &VM_FAULT_ERROR )) goto out_up; -/* -* Major/minor page fault accounting is only done on the -* initial attempt. If we go through a retry, it is extremely -* likely that the page will be found in page cache at that point. -*/ if (flags &FAULT_FLAG_ALLOW_RETRY ) {
Shouldn't this check ^^^ be dropped as well?
Since commit 4064b9827063 ("mm: allow VM_FAULT_RETRY for multiple times") FAULT_FLAG_ALLOW_RETRY never gets unset, so no need to check..
Quoted 15 lines
arch/s390/mm/fault.c +0
-9-if (fault &VM_FAULT_MAJOR ) { - tsk->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , - regs, address); - }else { - tsk->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , - regs, address); - }if (fault &VM_FAULT_RETRY ) {if (IS_ENABLED (CONFIG_PGSTE ) && gmap && (flags &FAULT_FLAG_RETRY_NOWAIT )) {Signature · 1 line
2.26.2
Peter Xu
reviewerRe: [PATCH v5 18/25] mm/s390: Use general page fault accounting
Quoted 42 lines · Alexander Gordeev
Quoted 29 lines · Peter Xu
Use the general page fault accounting by passing regs into
handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.CC Heiko Carstens <[email protected]>CC Vasily Gorbik <[email protected]>CC Christian Borntraeger <[email protected]>Reviewed-by Gerald Schaefer <[email protected]>Acked-by Gerald Schaefer <[email protected]>Signed-off-by Peter Xu <[email protected]>--- arch/s390/mm/fault.c | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-)arch/s390/mm/fault.c +1
-1@@ -478,7 +478,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs)) { fault =VM_FAULT_SIGNAL ;if (flags &FAULT_FLAG_RETRY_NOWAIT )arch/s390/mm/fault.c static inline vm_fault_t do_exception(struct pt_regs *regs, int access) · line 491
Quoted 9 lines
arch/s390/mm/fault.c +0
-5@@ -488,21 +488,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)if (unlikely (fault &VM_FAULT_ERROR )) goto out_up; -/* -* Major/minor page fault accounting is only done on the -* initial attempt. If we go through a retry, it is extremely -* likely that the page will be found in page cache at that point. -*/ if (flags &FAULT_FLAG_ALLOW_RETRY ) {Shouldn't this check ^^^ be dropped as well?
Quoted 2 lines
Since commit
4064b9827063("mm: allowVM_FAULT_RETRYfor multiple times")FAULT_FLAG_ALLOW_RETRYnever gets unset, so no need to check..
I agree, but it should be out of the scope of the accounting changes that this patch wants to address. Maybe more suitable for a work on top?
This should also exist for most of the archs too, and I'm also not sure whether compiler could be smart enough to optimize this directly since it seems to have all the knowledge.
Thanks,
Signature · 1 line
Peter Xu
Peter Xu
reviewerPATCH v5 19/25 mm/sh: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
--- arch/sh/mm/fault.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)
arch/sh/mm/fault.c +1 -10
@@ -482,22 +482,13 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (unlikely (fault & (VM_FAULT_RETRY |VM_FAULT_ERROR )))if (mm_fault_error (regs, error_code, address, fault))return ;if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) { - tsk->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , - regs, address); - }else { - tsk->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , - regs, address); - }if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 20/25 mm/sparc32: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
--- arch/sparc/mm/fault_32.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)
arch/sparc/mm/fault_32.c +1 -10
@@ -234,7 +234,7 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs))return ;
@@ -250,15 +250,6 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write, }if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) { - current->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ , -1 , regs, address); - }else { - current->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN , -1 , regs, address); - }if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 21/25 mm/sparc64: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
--- arch/sparc/mm/fault_64.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)
arch/sparc/mm/fault_64.c +1 -10
@@ -422,7 +422,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs) goto bad_area; } - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs)) goto exit_exception;
@@ -438,15 +438,6 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs) }if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) { - current->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ , -1 , regs, address); - }else { - current->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN , -1 , regs, address); - }if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 22/25 mm/x86: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault().
--- arch/x86/mm/fault.c | 17 ++--------------- 1 file changed, 2 insertions(+), 15 deletions(-)
arch/x86/mm/fault.c +2 -15
@@ -1139,7 +1139,7 @@ void do_user_addr_fault(struct pt_regs *regs,struct vm_area_struct *vma;struct task_struct *tsk;struct mm_struct *mm; -vm_fault_t fault, major =0 ; +vm_fault_t fault;unsigned int flags =FAULT_FLAG_DEFAULT ; tsk = current;
@@ -1291,8 +1291,7 @@ void do_user_addr_fault(struct pt_regs *regs,* userland). The return to userland is identified whenever * FAULT_FLAG_USER|FAULT_FLAG_KILLABLE are both set in flags. */ - fault =handle_mm_fault (vma, address, flags,NULL ); - major |= fault &VM_FAULT_MAJOR ; + fault =handle_mm_fault (vma, address, flags, regs);/* Quick path to respond to signals */ if (fault_signal_pending (fault, regs)) {
@@ -1319,18 +1318,6 @@ void do_user_addr_fault(struct pt_regs *regs,return ; } -/* -* Major/minor page fault accounting. If any of the events -* returned VM_FAULT_MAJOR, we account it as a major fault. -*/ -if (major) { - tsk->maj_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , regs, address); - }else { - tsk->min_flt++; -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , regs, address); - } -check_v8086_mode (regs, address, tsk); }NOKPROBE_SYMBOL (do_user_addr_fault);
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 23/25 mm/xtensa: Use general page fault accounting
In reply to Peter XuUse the general page fault accounting by passing regs into handle_mm_fault(). It naturally solve the issue of multiple page fault accounting when page fault retry happened.
Remove the PERF_COUNT_SW_PAGE_FAULTS_[MAJ|MIN] perf events because it's now also done in handle_mm_fault().
Move the PERF_COUNT_SW_PAGE_FAULTS event higher before taking mmap_sem for the fault, then it'll match with the rest of the archs.
--- arch/xtensa/mm/fault.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)
arch/xtensa/mm/fault.c +4 -11
@@ -72,6 +72,9 @@ void do_page_fault(struct pt_regs *regs)if (user_mode (regs)) flags |=FAULT_FLAG_USER ; + +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); + retry:mmap_read_lock (mm); vma =find_vma (mm, address);
@@ -107,7 +110,7 @@ void do_page_fault(struct pt_regs *regs)* make sure we exit gracefully rather than endlessly redo * the fault. */ - fault =handle_mm_fault (vma, address, flags,NULL ); + fault =handle_mm_fault (vma, address, flags, regs);if (fault_signal_pending (fault, regs))return ;
@@ -122,10 +125,6 @@ void do_page_fault(struct pt_regs *regs)BUG (); }if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) - current->maj_flt++; -else - current->min_flt++;if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
@@ -139,12 +138,6 @@ void do_page_fault(struct pt_regs *regs) }mmap_read_unlock (mm); -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); -if (flags &VM_FAULT_MAJOR ) -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , regs, address); -else -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , regs, address); -return ;/* Something tried to access memory that isn't in our memory map..
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 24/25 mm: Clean up the last pieces of page fault accountings
In reply to Peter XuHere're the last pieces of page fault accounting that were still done outside handle_mm_fault() where we still have regs==NULL when calling handle_mm_fault():
arch/powerpc/mm/copro_fault.c: copro_handle_mm_fault
arch/sparc/mm/fault_32.c: force_user_fault
arch/um/kernel/trap.c: handle_page_fault
mm/gup.c: faultin_page
fixup_user_fault
mm/hmm.c: hmm_vma_fault
mm/ksm.c: break_ksmSome of them has the issue of duplicated accounting for page fault retries. Some of them didn't do the accounting at all.
This patch cleans all these up by letting handle_mm_fault() to do per-task page fault accounting even if regs==NULL (though we'll still skip the perf event accountings). With that, we can safely remove all the outliers now.
There's another functional change in that now we account the page faults to the caller of gup, rather than the task_struct that passed into the gup code. More information of this can be found at [1].
After this patch, below things should never be touched again outside handle_mm_fault():
- task_struct.[maj|min]_flt
- PERF_COUNT_SW_PAGE_FAULTS_[MAJ|MIN]
[1] https://lore.kernel.org/lkml/CAHk-=wj_V2Tps2QrMn20_W0OJF9xqNh52XSGA42s-ZJ8Y+GyKw@mail.gmail.com/
--- arch/powerpc/mm/copro_fault.c | 5 ----- arch/um/kernel/trap.c | 4 ---- mm/gup.c | 13 ------------- mm/memory.c | 17 ++++++++++------- 4 files changed, 10 insertions(+), 29 deletions(-)
arch/powerpc/mm/copro_fault.c +0 -5
@@ -76,11 +76,6 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,BUG (); } -if (*flt &VM_FAULT_MAJOR ) - current->maj_flt++; -else - current->min_flt++; - out_unlock:mmap_read_unlock (mm);return ret;
arch/um/kernel/trap.c +0 -4
@@ -88,10 +88,6 @@ int handle_page_fault(unsigned long address, unsigned long ip,BUG (); }if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_MAJOR ) - current->maj_flt++; -else - current->min_flt++;if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ;
mm/gup.c +0 -13
@@ -893,13 +893,6 @@ static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,BUG (); } -if (tsk) { -if (ret &VM_FAULT_MAJOR ) - tsk->maj_flt++; -else - tsk->min_flt++; - } -if (ret &VM_FAULT_RETRY ) {if (locked && !(fault_flags &FAULT_FLAG_RETRY_NOWAIT )) *locked =0 ;
@@ -1255,12 +1248,6 @@ int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm, goto retry; } -if (tsk) { -if (major) - tsk->maj_flt++; -else - tsk->min_flt++; - }return 0 ; }EXPORT_SYMBOL_GPL (fixup_user_fault);
mm/memory.c +10 -7
@@ -4404,20 +4404,23 @@ static inline void mm_account_fault(struct pt_regs *regs,*/ major = (ret &VM_FAULT_MAJOR ) || (flags &FAULT_FLAG_TRIED ); +if (major) + current->maj_flt++; +else + current->min_flt++; +/* -* If the fault is done for GUP, regs will be NULL, and we will skip -* the fault accounting. +* If the fault is done for GUP, regs will be NULL. We only do the +* accounting for the per thread fault counters who triggered the +* fault, and we skip the perf event updates. */ if (!regs)return ; -if (major) { - current->maj_flt++; +if (major)perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MAJ ,1 , regs, address); - }else { - current->min_flt++; +else perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS_MIN ,1 , regs, address); - } }/*
Signature · 1 line
2.26.2
Peter Xu
reviewerPATCH v5 25/25 mm/gup: Remove task_struct pointer for all gup code
In reply to Peter XuAfter the cleanup of page fault accounting, gup does not need to pass task_struct around any more. Remove that parameter in the whole gup stack.
--- arch/arc/kernel/process.c | 2 +- arch/s390/kvm/interrupt.c | 2 +- arch/s390/kvm/kvm-s390.c | 2 +- arch/s390/kvm/priv.c | 8 +- arch/s390/mm/gmap.c | 4 +- drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 2 +- drivers/infiniband/core/umem_odp.c | 2 +- drivers/vfio/vfio_iommu_type1.c | 4 +- fs/exec.c | 2 +- include/linux/mm.h | 9 +- kernel/events/uprobes.c | 6 +- kernel/futex.c | 2 +- mm/gup.c | 101 ++++++++------------ mm/memory.c | 2 +- mm/process_vm_access.c | 2 +- security/tomoyo/domain.c | 2 +- virt/kvm/async_pf.c | 2 +- virt/kvm/kvm_main.c | 2 +- 18 files changed, 69 insertions(+), 87 deletions(-)
arch/arc/kernel/process.c +1 -1
@@ -91,7 +91,7 @@ SYSCALL_DEFINE3(arc_usr_cmpxchg, int *, uaddr, int, expected, int, new) goto fail;mmap_read_lock (current->mm); - ret =fixup_user_fault (current, current->mm, (unsigned long ) uaddr, + ret =fixup_user_fault (current->mm, (unsigned long ) uaddr,FAULT_FLAG_WRITE ,NULL );mmap_read_unlock (current->mm);
arch/s390/kvm/interrupt.c +1 -1
@@ -2768,7 +2768,7 @@ static struct page *get_map_page(struct kvm *kvm, u64 uaddr)struct page *page =NULL ;mmap_read_lock (kvm->mm); -get_user_pages_remote (NULL , kvm->mm, uaddr,1 ,FOLL_WRITE , +get_user_pages_remote (kvm->mm, uaddr,1 ,FOLL_WRITE , &page,NULL ,NULL );mmap_read_unlock (kvm->mm);return page;
arch/s390/kvm/kvm-s390.c +1 -1
@@ -1892,7 +1892,7 @@ static long kvm_s390_set_skeys(struct kvm *kvm, struct kvm_s390_skeys *args) r =set_guest_storage_key (current->mm, hva, keys[i],0 );if (r) { - r =fixup_user_fault (current, current->mm, hva, + r =fixup_user_fault (current->mm, hva,FAULT_FLAG_WRITE , &unlocked);if (r)break ;
arch/s390/kvm/priv.c +4 -4
@@ -273,7 +273,7 @@ static int handle_iske(struct kvm_vcpu *vcpu) rc =get_guest_storage_key (current->mm, vmaddr, &key);if (rc) { - rc =fixup_user_fault (current, current->mm, vmaddr, + rc =fixup_user_fault (current->mm, vmaddr,FAULT_FLAG_WRITE , &unlocked);if (!rc) {mmap_read_unlock (current->mm);
@@ -319,7 +319,7 @@ static int handle_rrbe(struct kvm_vcpu *vcpu)mmap_read_lock (current->mm); rc =reset_guest_reference_bit (current->mm, vmaddr);if (rc <0 ) { - rc =fixup_user_fault (current, current->mm, vmaddr, + rc =fixup_user_fault (current->mm, vmaddr,FAULT_FLAG_WRITE , &unlocked);if (!rc) {mmap_read_unlock (current->mm);
@@ -390,7 +390,7 @@ static int handle_sske(struct kvm_vcpu *vcpu) m3 & SSKE_MC);if (rc <0 ) { - rc =fixup_user_fault (current, current->mm, vmaddr, + rc =fixup_user_fault (current->mm, vmaddr,FAULT_FLAG_WRITE , &unlocked); rc = !rc ? -EAGAIN : rc; }
@@ -1094,7 +1094,7 @@ static int handle_pfmf(struct kvm_vcpu *vcpu) rc =cond_set_guest_storage_key (current->mm, vmaddr, key,NULL , nq, mr, mc);if (rc <0 ) { - rc =fixup_user_fault (current, current->mm, vmaddr, + rc =fixup_user_fault (current->mm, vmaddr,FAULT_FLAG_WRITE , &unlocked); rc = !rc ? -EAGAIN : rc; }
arch/s390/mm/gmap.c +2 -2
@@ -649,7 +649,7 @@ int gmap_fault(struct gmap *gmap, unsigned long gaddr, rc = vmaddr; goto out_up; } -if (fixup_user_fault (current, gmap->mm, vmaddr, fault_flags, +if (fixup_user_fault (gmap->mm, vmaddr, fault_flags, &unlocked)) { rc = -EFAULT ; goto out_up;
@@ -879,7 +879,7 @@ static int gmap_pte_op_fixup(struct gmap *gmap, unsigned long gaddr,BUG_ON (gmap_is_shadow (gmap)); fault_flags = (prot ==PROT_WRITE ) ?FAULT_FLAG_WRITE :0 ; -if (fixup_user_fault (current, mm, vmaddr, fault_flags, &unlocked)) +if (fixup_user_fault (mm, vmaddr, fault_flags, &unlocked))return -EFAULT ;if (unlocked)/* lost mmap_lock, caller has to retry __gmap_translate */
drivers/gpu/drm/i915/gem/i915_gem_userptr.c +1 -1
@@ -469,7 +469,7 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work) locked =1 ; } ret = pin_user_pages_remote - (work->task, mm, + (mm, obj->userptr.ptr + pinned * PAGE_SIZE, npages - pinned, flags,
drivers/infiniband/core/umem_odp.c +1 -1
@@ -439,7 +439,7 @@ int ib_umem_odp_map_dma_pages(struct ib_umem_odp *umem_odp, u64 user_virt,* complex (and doesn't gain us much performance in most use * cases). */ - npages =get_user_pages_remote (owning_process, owning_mm, + npages =get_user_pages_remote (owning_mm, user_virt, gup_num_pages, flags, local_page_list,NULL ,NULL );mmap_read_unlock (owning_mm);
drivers/vfio/vfio_iommu_type1.c +2 -2
@@ -425,7 +425,7 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm,if (ret) {bool unlocked = false; - ret =fixup_user_fault (NULL , mm, vaddr, + ret =fixup_user_fault (mm, vaddr,FAULT_FLAG_REMOTE | (write_fault ?FAULT_FLAG_WRITE :0 ), &unlocked);
@@ -453,7 +453,7 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr, flags |=FOLL_WRITE ;mmap_read_lock (mm); - ret =pin_user_pages_remote (NULL , mm, vaddr,1 , flags |FOLL_LONGTERM , + ret =pin_user_pages_remote (mm, vaddr,1 , flags |FOLL_LONGTERM , page,NULL ,NULL );if (ret ==1 ) { *pfn =page_to_pfn (page[0 ]);
fs/exec.c +1 -1
@@ -217,7 +217,7 @@ static struct page *get_arg_page(struct linux_binprm *bprm, unsigned long pos,* We are doing an exec(). 'current' is the process * doing the exec and bprm->mm is the new process's mm. */ - ret =get_user_pages_remote (current, bprm->mm, pos,1 , gup_flags, + ret =get_user_pages_remote (bprm->mm, pos,1 , gup_flags, &page,NULL ,NULL );if (ret <=0 )return NULL ;
include/linux/mm.h +4 -5
@@ -1662,7 +1662,7 @@ int invalidate_inode_page(struct page *page);extern vm_fault_t handle_mm_fault (struct vm_area_struct *vma,unsigned long address,unsigned int flags,struct pt_regs *regs); -extern int fixup_user_fault (struct task_struct *tsk,struct mm_struct *mm, +extern int fixup_user_fault (struct mm_struct *mm,unsigned long address,unsigned int fault_flags,bool *unlocked); void unmap_mapping_pages(struct address_space *mapping,
@@ -1678,8 +1678,7 @@ static inline vm_fault_t handle_mm_fault(struct vm_area_struct *vma,BUG ();return VM_FAULT_SIGBUS ; } -static inline int fixup_user_fault (struct task_struct *tsk, -struct mm_struct *mm,unsigned long address, +static inline int fixup_user_fault (struct mm_struct *mm,unsigned long address,unsigned int fault_flags,bool *unlocked) {/* should never happen if there's no MMU */
@@ -1705,11 +1704,11 @@ extern int access_remote_vm(struct mm_struct *mm, unsigned long addr,extern int __access_remote_vm (struct task_struct *tsk,struct mm_struct *mm,unsigned long addr,void *buf,int len,unsigned int gup_flags); -long get_user_pages_remote (struct task_struct *tsk,struct mm_struct *mm, +long get_user_pages_remote (struct mm_struct *mm,unsigned long start,unsigned long nr_pages,unsigned int gup_flags,struct page **pages,struct vm_area_struct **vmas,int *locked); -long pin_user_pages_remote (struct task_struct *tsk,struct mm_struct *mm, +long pin_user_pages_remote (struct mm_struct *mm,unsigned long start,unsigned long nr_pages,unsigned int gup_flags,struct page **pages,struct vm_area_struct **vmas,int *locked);
kernel/events/uprobes.c +3 -3
@@ -376,7 +376,7 @@ __update_ref_ctr(struct mm_struct *mm, unsigned long vaddr, short d)if (!vaddr || !d)return -EINVAL ; - ret =get_user_pages_remote (NULL , mm, vaddr,1 , + ret =get_user_pages_remote (mm, vaddr,1 ,FOLL_WRITE , &page, &vma,NULL );if (unlikely (ret <=0 )) {/*
@@ -477,7 +477,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm,if (is_register) gup_flags |=FOLL_SPLIT_PMD ;/* Read the page with vaddr into memory */ - ret =get_user_pages_remote (NULL , mm, vaddr,1 , gup_flags, + ret =get_user_pages_remote (mm, vaddr,1 , gup_flags, &old_page, &vma,NULL );if (ret <=0 )return ret;
@@ -2029,7 +2029,7 @@ static int is_trap_at_addr(struct mm_struct *mm, unsigned long vaddr)* but we treat this as a 'remote' access since it is * essentially a kernel access to the memory. */ - result =get_user_pages_remote (NULL , mm, vaddr,1 ,FOLL_FORCE , &page, + result =get_user_pages_remote (mm, vaddr,1 ,FOLL_FORCE , &page,NULL ,NULL );if (result <0 )return result;
kernel/futex.c +1 -1
@@ -699,7 +699,7 @@ static int fault_in_user_writeable(u32 __user *uaddr)int ret;mmap_read_lock (mm); - ret =fixup_user_fault (current, mm, (unsigned long )uaddr, + ret =fixup_user_fault (mm, (unsigned long )uaddr,FAULT_FLAG_WRITE ,NULL );mmap_read_unlock (mm);
mm/gup.c +42 -59
348 lines not on this page · raw message
mm/memory.c +1 -1
@@ -4746,7 +4746,7 @@ int __access_remote_vm(struct task_struct *tsk, struct mm_struct *mm,void *maddr;struct page *page =NULL ; - ret =get_user_pages_remote (tsk, mm, addr,1 , + ret =get_user_pages_remote (mm, addr,1 , gup_flags, &page, &vma,NULL );if (ret <=0 ) {#ifndef CONFIG_HAVE_IOREMAP_PROT
mm/process_vm_access.c +1 -1
@@ -105,7 +105,7 @@ static int process_vm_rw_single_vec(unsigned long addr,* current/current->mm */ mmap_read_lock (mm); - pinned_pages =pin_user_pages_remote (task, mm, pa, pinned_pages, + pinned_pages =pin_user_pages_remote (mm, pa, pinned_pages, flags, process_pages,NULL , &locked);if (locked)
security/tomoyo/domain.c +1 -1
@@ -914,7 +914,7 @@ bool tomoyo_dump_page(struct linux_binprm *bprm, unsigned long pos,* (represented by bprm). 'current' is the process doing * the execve(). */ -if (get_user_pages_remote (current, bprm->mm, pos,1 , +if (get_user_pages_remote (bprm->mm, pos,1 ,FOLL_FORCE , &page,NULL ,NULL ) <=0 )return false;#else
virt/kvm/async_pf.c +1 -1
@@ -61,7 +61,7 @@ static void async_pf_execute(struct work_struct *work)* access remotely. */ mmap_read_lock (mm); -get_user_pages_remote (NULL , mm, addr,1 ,FOLL_WRITE ,NULL ,NULL , +get_user_pages_remote (mm, addr,1 ,FOLL_WRITE ,NULL ,NULL , &locked);if (locked)mmap_read_unlock (mm);
virt/kvm/kvm_main.c +1 -1
@@ -1830,7 +1830,7 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma,* not call the fault handler, so do it here. */ bool unlocked = false; - r =fixup_user_fault (current, current->mm, addr, + r =fixup_user_fault (current->mm, addr, (write_fault ?FAULT_FLAG_WRITE :0 ), &unlocked);if (unlocked)
Signature · 1 line
2.26.2