mprotect() and working set sampling optimizations
Subsystems (5)
v2 cover mprotect() and working set sampling optimizations
Ok, people suggested to split out the change_protection() modification into a third patch.
This series implements an mprotect() optimization that also helps improve the quality of working set scanning:
- working set scanning gets faster
- we can scan with a touched-page rate, instead of with a virtual-memory proportional rate (within limits).
This is already part of numa/core, but wanted to send it out separately as well, to get specific feedback for the mprotect() bits.
Thanks,
Ingo
---
Ingo Molnar (1):
mm: Optimize the TLB flush of sys_mprotect() and change_protection()
users
Peter Zijlstra (2):
mm: Count the number of pages affected in change_protection() sched, numa, mm: Count WS scanning against present PTEs, not virtual
memory ranges
include/linux/hugetlb.h | 8 ++++++-- include/linux/mm.h | 6 +++--- kernel/sched/fair.c | 37 +++++++++++++++++++++---------------- mm/hugetlb.c | 10 ++++++++-- mm/mprotect.c | 46 ++++++++++++++++++++++++++++++++++------------ 5 files changed, 72 insertions(+), 35 deletions(-)
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 1/3 mm: Count the number of pages affected in change_protection()
From: Peter Zijlstra <[email protected]>
This will be used by three kinds of purposes:
- to optimize mprotect()
- to speed up working set scanning for working set areas that have not been touched
- to more accurately scan per real working set
No change in functionality from this patch.
--- include/linux/hugetlb.h | 8 ++++++-- include/linux/mm.h | 6 +++--- mm/hugetlb.c | 10 ++++++++-- mm/mprotect.c | 41 ++++++++++++++++++++++++++++++----------- 4 files changed, 47 insertions(+), 18 deletions(-)
include/linux/hugetlb.h +6 -2
@@ -87,7 +87,7 @@ struct page *follow_huge_pud(struct mm_struct *mm, unsigned long address, pud_t *pud, int write);int pmd_huge (pmd_t pmd);int pud_huge (pud_t pmd); -void hugetlb_change_protection (struct vm_area_struct *vma, +unsigned long hugetlb_change_protection (struct vm_area_struct *vma,unsigned long address,unsigned long end,pgprot_t newprot);#else /* !CONFIG_HUGETLB_PAGE */
@@ -132,7 +132,11 @@ static inline void copy_huge_page(struct page *dst, struct page *src) { } -#define hugetlb_change_protection (vma, address, end, newprot) +static inline unsigned long hugetlb_change_protection (struct vm_area_struct *vma, +unsigned long address,unsigned long end,pgprot_t newprot) +{ +return 0 ; +} static inline void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start,
include/linux/mm.h +3 -3
@@ -1099,7 +1099,7 @@ extern unsigned long move_page_tables(struct vm_area_struct *vma,extern unsigned long do_mremap (unsigned long addr,unsigned long old_len,unsigned long new_len,unsigned long flags,unsigned long new_addr); -extern void change_protection (struct vm_area_struct *vma,unsigned long start, +extern unsigned long change_protection (struct vm_area_struct *vma,unsigned long start,unsigned long end,pgprot_t newprot,int dirty_accountable); extern int mprotect_fixup(struct vm_area_struct *vma,
@@ -1581,10 +1581,10 @@ static inline pgprot_t vma_prot_none(struct vm_area_struct *vma)return pgprot_modify (vma->vm_page_prot,vm_get_page_prot (vmflags)); } -static inline void +static inline unsigned long change_prot_none (struct vm_area_struct *vma,unsigned long start,unsigned long end) { -change_protection (vma, start, end,vma_prot_none (vma),0 ); +return change_protection (vma, start, end,vma_prot_none (vma),0 ); }struct vm_area_struct *find_extend_vma (struct mm_struct *,unsigned long addr);
mm/hugetlb.c +8 -2
@@ -3014,7 +3014,7 @@ same_page:return i ? i : -EFAULT ; } -void hugetlb_change_protection (struct vm_area_struct *vma, +unsigned long hugetlb_change_protection (struct vm_area_struct *vma,unsigned long address,unsigned long end,pgprot_t newprot) {struct mm_struct *mm = vma->vm_mm;
@@ -3022,6 +3022,7 @@ void hugetlb_change_protection(struct vm_area_struct *vma,pte_t *ptep;pte_t pte;struct hstate *h =hstate_vma (vma); +unsigned long pages =0 ;BUG_ON (address >= end);flush_cache_range (vma, address, end);
@@ -3032,12 +3033,15 @@ void hugetlb_change_protection(struct vm_area_struct *vma, ptep =huge_pte_offset (mm, address);if (!ptep)continue ; -if (huge_pmd_unshare (mm, &address, ptep)) +if (huge_pmd_unshare (mm, &address, ptep)) { + pages++;continue ; + }if (!huge_pte_none (huge_ptep_get (ptep))) { pte =huge_ptep_get_and_clear (mm, address, ptep); pte =pte_mkhuge (pte_modify (pte, newprot));set_huge_pte_at (mm, address, ptep, pte); + pages++; } }spin_unlock (&mm->page_table_lock);
@@ -3049,6 +3053,8 @@ void hugetlb_change_protection(struct vm_area_struct *vma,*/ flush_tlb_range (vma, start, end);mutex_unlock (&vma->vm_file->f_mapping->i_mmap_mutex); + +return pages << h->order; } int hugetlb_reserve_pages(struct inode *inode,
mm/mprotect.c +30 -11
131 lines not on this page · raw message
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 2/3 sched, numa, mm: Count WS scanning against present PTEs, not virtual memory ranges
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
By accounting against the present PTEs, scanning speed reflects the actual present (mapped) memory.
--- kernel/sched/fair.c | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-)
kernel/sched/fair.c +21 -16
@@ -914,8 +914,8 @@ void task_numa_work(struct callback_head *work)struct task_struct *p = current;struct mm_struct *mm = p->mm;struct vm_area_struct *vma; -unsigned long offset, end; -long length; +unsigned long start, end; +long pages; WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
@@ -942,30 +942,35 @@ void task_numa_work(struct callback_head *work)if (cmpxchg (&mm->numa_next_scan, migrate, next_scan) != migrate)return ; - offset = mm->numa_scan_offset; - length = sysctl_sched_numa_scan_size; - length <<=20 ; + start = mm->numa_scan_offset; + pages = sysctl_sched_numa_scan_size; + pages <<=20 -PAGE_SHIFT ;/* MB in pages */ +if (!pages) +return ;down_write (&mm->mmap_sem); - vma =find_vma (mm, offset); + vma =find_vma (mm, start);if (!vma) {ACCESS_ONCE (mm->numa_scan_seq)++; - offset =0 ; + start =0 ; vma = mm->mmap; } -for (; vma && length >0 ; vma = vma->vm_next) { +for (; vma; vma = vma->vm_next) {if (!vma_migratable (vma))continue ; - offset =max (offset, vma->vm_start); - end =min (ALIGN (offset + length,HPAGE_SIZE ), vma->vm_end); - length -= end - offset; - -change_prot_none (vma, offset, end); - - offset = end; +do { + start =max (start, vma->vm_start); + end =ALIGN (start + (pages <<PAGE_SHIFT ),HPAGE_SIZE ); + end =min (end, vma->vm_end); + pages -=change_prot_none (vma, start, end); + start = end; +if (pages <=0 ) + goto out; + }while (end != vma->vm_end); } - mm->numa_scan_offset = offset; +out: + mm->numa_scan_offset = start;up_write (&mm->mmap_sem); }
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 3/3 mm: Optimize the TLB flush of sys_mprotect() and change_protection() users
In reply to Ingo MolnarReuse the NUMA code's 'modified page protections' count that change_protection() computes and skip the TLB flush if there's no changes to a range that sys_mprotect() modifies.
Given that mprotect() already optimizes the same-flags case I expected this optimization to dominantly trigger on CONFIG_NUMA_BALANCING=y kernels - but even with that feature disabled it triggers rather often.
There's two reasons for that:
1)
While sys_mprotect() already optimizes the same-flag case:
if (newflags == oldflags) {
*pprev = vma;
return 0;
}and this test works in many cases, but it is too sharp in some others, where it differentiates between protection values that the underlying PTE format makes no distinction about, such as PROT_EXEC == PROT_READ on x86.
2)
Even where the pte format over vma flag changes necessiates a modification of the pagetables, there might be no pagetables yet to modify: they might not be instantiated yet.
During a regular desktop bootup this optimization hits a couple of hundred times. During a Java test I measured thousands of hits.
So this optimization improves sys_mprotect() in general, not just CONFIG_NUMA_BALANCING=y kernels.
[ We could further increase the efficiency of this optimization if
change_pte_range() and change_huge_pmd() was a bit smarter about recognizing exact-same-value protection masks - when the hardware can do that safely. This would probably further speed up mprotect(). ]
--- mm/mprotect.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/mprotect.c +4 -1
@@ -145,7 +145,10 @@ static unsigned long change_protection_range(struct vm_area_struct *vma, pages +=change_pud_range (vma, pgd, addr, next, newprot, dirty_accountable); }while (pgd++, addr = next, addr != end); -flush_tlb_range (vma, start, end); + +/* Only flush the TLB if we actually modified any entries: */ +if (pages) +flush_tlb_range (vma, start, end);return pages; }
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>