Latest numa/core release, v16
Subsystems (17)
cover Latest numa/core release, v16
I'm pleased to announce the latest version of the numa/core tree.
Here are some quick, preliminary performance numbers on a 4-node, 32-way, 64 GB RAM system:
CONFIG_NUMA_BALANCING=y
-----------------------------------------------------------------------
[ seconds ] v3.7 AutoNUMA | numa/core-v16 [ vs. v3.7]
[ lower is better ] ----- -------- | ------------- -----------
|
numa01 340.3 192.3 | 139.4 +144.1%
numa01_THREAD_ALLOC 425.1 135.1 | 121.1 +251.0%
numa02 56.1 25.3 | 17.5 +220.5%
|
[ SPECjbb transactions/sec ] |
[ higher is better ] |
|
SPECjbb single-1x32 524k 507k | 638k +21.7%-----------------------------------------------------------------------
On my NUMA system the numa/core tree now significantly outperforms both the vanilla kernel and the AutoNUMA (v28) kernel, in these benchmarks. No NUMA balancing kernel has ever performed so well on this system.
It is notable that workloads where 'private' processing dominates (numa01_THREAD_ALLOC and numa02) are now very close to bare metal hard binding performance.
These are the main changes in this release:
- There are countless performance improvements. The new shared/private distinction metric we introduced in v15 is now further refined and is used in more places within the scheduler to converge in a better and more directed fashion.
- I restructured the whole tree to make it cleaner, to simplify its mm/ impact and in general to make it more mergable. It now includes either agreed-upon patches, or bare essentials that are needed to make the CONFIG_NUMA_BALANCING=y feature work. It is fully bisect tested - it builds and works at every point.
- The hard-coded "PROT_NONE" feature that reviewers complained about is now factored out and selectable on a per architecture basis. (the arch porting aspect of this is untested, but the basic fabric is there and should be pretty close to what we need.)
The generic PROT_NONE based facility can be used by architectures to prototype this feature quickly.
- I tried to pick up all fixes that were sent. Many thanks go to Hugh Dickins and Johannes Weiner! If I missed any fix or review feedback, please re-send, as the code base has changed significantly.
Future plans are to concentrate on converging 'shared' workloads even better, to address any pending review feedback, and to fix any regressions that might be remaining.
Bug reports, review feedback and suggestions are welcome!
Thanks,
Ingo
------------>
Andrea Arcangeli (1):
numa, mm: Support NUMA hinting page faults from gup/gup_fast
Ingo Molnar (9):
mm: Optimize the TLB flush of sys_mprotect() and change_protection()
users
sched, mm, numa: Create generic NUMA fault infrastructure, with
architectures overrides
sched, mm, x86: Add the ARCH_SUPPORTS_NUMA_BALANCING flag sched, numa, mm: Interleave shared tasks
sched: Implement NUMA scanning backoff
sched: Improve convergence
sched: Introduce staged average NUMA faults
sched: Track groups of shared tasks
sched: Use the best-buddy 'ideal cpu' in balancing decisions
Peter Zijlstra (11):
mm: Count the number of pages affected in change_protection() sched, numa, mm: Add last_cpu to page flags
sched: Make find_busiest_queue() a method
sched, numa, mm: Describe the NUMA scheduling problem formally mm/migrate: Introduce migrate_misplaced_page()
sched, numa, mm, arch: Add variable locality exception sched, numa, mm: Add the scanning page fault machinery
sched: Add adaptive NUMA affinity support
sched: Implement constant, per task Working Set Sampling (WSS) rate sched, numa, mm: Count WS scanning against present PTEs, not virtual
memory ranges
sched: Implement slow start for working set sampling
Rik van Riel (6):
mm/generic: Only flush the local TLB in ptep_set_access_flags() x86/mm: Only do a local tlb flush in ptep_set_access_flags() x86/mm: Introduce pte_accessible()
mm: Only flush the TLB when clearing an accessible pte x86/mm: Completely drop the TLB flush from ptep_set_access_flags() sched, numa, mm: Add credits for NUMA placement
CREDITS | 1 + Documentation/scheduler/numa-problem.txt | 236 ++++++ arch/sh/mm/Kconfig | 1 + arch/x86/Kconfig | 2 + arch/x86/include/asm/pgtable.h | 6 + arch/x86/mm/pgtable.c | 8 +- include/asm-generic/pgtable.h | 59 ++ include/linux/huge_mm.h | 12 + include/linux/hugetlb.h | 8 +- include/linux/init_task.h | 8 + include/linux/mempolicy.h | 8 + include/linux/migrate.h | 7 + include/linux/migrate_mode.h | 3 + include/linux/mm.h | 99 ++- include/linux/mm_types.h | 10 + include/linux/mmzone.h | 14 +- include/linux/page-flags-layout.h | 83 ++ include/linux/sched.h | 52 +- init/Kconfig | 81 ++ kernel/bounds.c | 4 + kernel/sched/core.c | 79 +- kernel/sched/fair.c | 1227 +++++++++++++++++++++++++----- kernel/sched/features.h | 10 + kernel/sched/sched.h | 38 +- kernel/sysctl.c | 45 +- mm/Makefile | 1 + mm/huge_memory.c | 163 ++++ mm/hugetlb.c | 10 +- mm/internal.h | 5 +- mm/memcontrol.c | 7 +- mm/memory.c | 108 ++- mm/mempolicy.c | 183 ++++- mm/migrate.c | 81 +- mm/mprotect.c | 69 +- mm/numa.c | 73 ++ mm/pgtable-generic.c | 9 +- 36 files changed, 2492 insertions(+), 318 deletions(-) create mode 100644 Documentation/scheduler/numa-problem.txt create mode 100644 include/linux/page-flags-layout.h create mode 100644 mm/numa.c
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/27 mm/generic: Only flush the local TLB in ptep_set_access_flags()
From: Rik van Riel <[email protected]>
The function ptep_set_access_flags() is only ever used to upgrade access permissions to a page - i.e. they make it less restrictive.
That means the only negative side effect of not flushing remote TLBs in this function is that other CPUs may incur spurious page faults, if they happen to access the same address, and still have a PTE with the old permissions cached in their TLB caches.
Having another CPU maybe incur a spurious page fault is faster than always incurring the cost of a remote TLB flush, so replace the remote TLB flush with a purely local one.
This should be safe on every architecture that correctly implements flush_tlb_fix_spurious_fault() to actually invalidate the local TLB entry that caused a page fault, as well as on architectures where the hardware invalidates TLB entries that cause page faults.
In the unlikely event that you are hitting what appears to be an infinite loop of page faults, and 'git bisect' took you to this changeset, your architecture needs to implement flush_tlb_fix_spurious_fault() to actually flush the TLB entry.
--- mm/pgtable-generic.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
mm/pgtable-generic.c +3 -3
@@ -12,8 +12,8 @@#ifndef __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS/* -* Only sets the access flags (dirty, accessed, and -* writable). Furthermore, we know it always gets set to a "more +* Only sets the access flags (dirty, accessed), as well as write +* permission. Furthermore, we know it always gets set to a "more * permissive" setting, which allows most architectures to optimize * this. We return whether the PTE actually changed, which in turn * instructs the caller to do things like update__mmu_cache. This
@@ -27,7 +27,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma,int changed = !pte_same (*ptep, entry);if (changed) {set_pte_at (vma->vm_mm, address, ptep, entry); -flush_tlb_page (vma, address); +flush_tlb_fix_spurious_fault (vma, address); }return changed; }
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/27 x86/mm: Only do a local tlb flush in ptep_set_access_flags()
In reply to Ingo MolnarFrom: Rik van Riel <[email protected]>
Because we only ever upgrade a PTE when calling ptep_set_access_flags(), it is safe to skip flushing entries on remote TLBs.
The worst that can happen is a spurious page fault on other CPUs, which would flush that TLB entry.
Lazily letting another CPU incur a spurious page fault occasionally is (much!) cheaper than aggressively flushing everybody else's TLB.
--- arch/x86/mm/pgtable.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
arch/x86/mm/pgtable.c +8 -1
@@ -301,6 +301,13 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd)free_page ((unsigned long )pgd); } +/* +* Used to set accessed or dirty bits in the page table entries +* on other architectures. On x86, the accessed and dirty bits +* are tracked by hardware. However, do_wp_page calls this function +* to also make the pte writeable at the same time the dirty bit is +* set. In that case we do actually need to write the PTE. +*/ int ptep_set_access_flags (struct vm_area_struct *vma,unsigned long address,pte_t *ptep,pte_t entry,int dirty)
@@ -310,7 +317,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma,if (changed && dirty) { *ptep = entry;pte_update_defer (vma->vm_mm, address, ptep); -flush_tlb_page (vma, address); +__flush_tlb_one (address); }return changed;
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/27 x86/mm: Introduce pte_accessible()
In reply to Ingo MolnarFrom: Rik van Riel <[email protected]>
We need pte_present to return true for _PAGE_PROTNONE pages, to indicate that the pte is associated with a page.
However, for TLB flushing purposes, we would like to know whether the pte points to an actually accessible page. This allows us to skip remote TLB flushes for pages that are not actually accessible.
Fill in this method for x86 and provide a safe (but slower) method on other architectures.
--- arch/x86/include/asm/pgtable.h | 6 ++++++ include/asm-generic/pgtable.h | 4 ++++ 2 files changed, 10 insertions(+)
arch/x86/include/asm/pgtable.h +6 -0
@@ -407,6 +407,12 @@ static inline int pte_present(pte_t a)return pte_flags (a) & (_PAGE_PRESENT | _PAGE_PROTNONE); } +#define pte_accessible pte_accessible +static inline int pte_accessible (pte_t a) +{ +return pte_flags (a) & _PAGE_PRESENT; +} +static inline int pte_hidden (pte_t pte) {return pte_flags (pte) & _PAGE_HIDDEN;
include/asm-generic/pgtable.h +4 -0
@@ -219,6 +219,10 @@ static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b)#define move_pte (pte, prot, old_addr, new_addr) (pte)#endif +#ifndef pte_accessible +# define pte_accessible (pte) ((void)(pte),1) +#endif +#ifndef flush_tlb_fix_spurious_fault#define flush_tlb_fix_spurious_fault (vma, address) flush_tlb_page(vma, address)#endif
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/27 mm: Only flush the TLB when clearing an accessible pte
In reply to Ingo MolnarFrom: Rik van Riel <[email protected]>
If ptep_clear_flush() is called to clear a page table entry that is accessible anyway by the CPU, eg. a _PAGE_PROTNONE page table entry, there is no need to flush the TLB on remote CPUs.
--- mm/pgtable-generic.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/pgtable-generic.c +2 -1
@@ -88,7 +88,8 @@ pte_t ptep_clear_flush(struct vm_area_struct *vma, unsigned long address, {pte_t pte; pte =ptep_get_and_clear ((vma)->vm_mm, address, ptep); -flush_tlb_page (vma, address); +if (pte_accessible (pte)) +flush_tlb_page (vma, address);return pte; }#endif
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/27 x86/mm: Completely drop the TLB flush from ptep_set_access_flags()
In reply to Ingo MolnarFrom: Rik van Riel <[email protected]>
Intel has an architectural guarantee that the TLB entry causing a page fault gets invalidated automatically. This means we should be able to drop the local TLB invalidation.
Because of the way other areas of the page fault code work, chances are good that all x86 CPUs do this. However, if someone somewhere has an x86 CPU that does not invalidate the TLB entry causing a page fault, this one-liner should be easy to revert - or a CPU model specific quirk could be added to retain this optimization on most CPUs.
[ Applied changelog massage and moved this last in the series,
to create bisection distance. ]
--- arch/x86/mm/pgtable.c | 1 - 1 file changed, 1 deletion(-)
arch/x86/mm/pgtable.c +0 -1
@@ -317,7 +317,6 @@ int ptep_set_access_flags(struct vm_area_struct *vma,if (changed && dirty) { *ptep = entry;pte_update_defer (vma->vm_mm, address, ptep); -__flush_tlb_one (address); }return changed;
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/27 mm: Count the number of pages affected in change_protection()
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
This will be used for three kinds of purposes:
- to optimize mprotect()
- to speed up working set scanning for working set areas that have not been touched
- to more accurately scan per real working set
No change in functionality from this patch.
--- include/linux/hugetlb.h | 8 +++++-- include/linux/mm.h | 3 +++ mm/hugetlb.c | 10 +++++++-- mm/mprotect.c | 58 +++++++++++++++++++++++++++++++++++++------------ 4 files changed, 61 insertions(+), 18 deletions(-)
include/linux/hugetlb.h +6 -2
@@ -87,7 +87,7 @@ struct page *follow_huge_pud(struct mm_struct *mm, unsigned long address, pud_t *pud, int write);int pmd_huge (pmd_t pmd);int pud_huge (pud_t pmd); -void hugetlb_change_protection (struct vm_area_struct *vma, +unsigned long hugetlb_change_protection (struct vm_area_struct *vma,unsigned long address,unsigned long end,pgprot_t newprot);#else /* !CONFIG_HUGETLB_PAGE */
@@ -132,7 +132,11 @@ static inline void copy_huge_page(struct page *dst, struct page *src) { } -#define hugetlb_change_protection (vma, address, end, newprot) +static inline unsigned long hugetlb_change_protection (struct vm_area_struct *vma, +unsigned long address,unsigned long end,pgprot_t newprot) +{ +return 0 ; +} static inline void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start,
include/linux/mm.h +3 -0
@@ -1078,6 +1078,9 @@ extern unsigned long move_page_tables(struct vm_area_struct *vma,extern unsigned long do_mremap (unsigned long addr,unsigned long old_len,unsigned long new_len,unsigned long flags,unsigned long new_addr); +extern unsigned long change_protection (struct vm_area_struct *vma,unsigned long start, +unsigned long end,pgprot_t newprot, +int dirty_accountable);extern int mprotect_fixup (struct vm_area_struct *vma,struct vm_area_struct **pprev,unsigned long start,unsigned long end,unsigned long newflags);
mm/hugetlb.c +8 -2
@@ -3014,7 +3014,7 @@ same_page:return i ? i : -EFAULT ; } -void hugetlb_change_protection (struct vm_area_struct *vma, +unsigned long hugetlb_change_protection (struct vm_area_struct *vma,unsigned long address,unsigned long end,pgprot_t newprot) {struct mm_struct *mm = vma->vm_mm;
@@ -3022,6 +3022,7 @@ void hugetlb_change_protection(struct vm_area_struct *vma,pte_t *ptep;pte_t pte;struct hstate *h =hstate_vma (vma); +unsigned long pages =0 ;BUG_ON (address >= end);flush_cache_range (vma, address, end);
@@ -3032,12 +3033,15 @@ void hugetlb_change_protection(struct vm_area_struct *vma, ptep =huge_pte_offset (mm, address);if (!ptep)continue ; -if (huge_pmd_unshare (mm, &address, ptep)) +if (huge_pmd_unshare (mm, &address, ptep)) { + pages++;continue ; + }if (!huge_pte_none (huge_ptep_get (ptep))) { pte =huge_ptep_get_and_clear (mm, address, ptep); pte =pte_mkhuge (pte_modify (pte, newprot));set_huge_pte_at (mm, address, ptep, pte); + pages++; } }spin_unlock (&mm->page_table_lock);
@@ -3049,6 +3053,8 @@ void hugetlb_change_protection(struct vm_area_struct *vma,*/ flush_tlb_range (vma, start, end);mutex_unlock (&vma->vm_file->f_mapping->i_mmap_mutex); + +return pages << h->order; } int hugetlb_reserve_pages(struct inode *inode,
mm/mprotect.c +44 -14
143 lines not on this page · raw message
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 7/27 mm: Optimize the TLB flush of sys_mprotect() and change_protection() users
In reply to Ingo MolnarReuse the NUMA code's 'modified page protections' count that change_protection() computes and skip the TLB flush if there's no changes to a range that sys_mprotect() modifies.
Given that mprotect() already optimizes the same-flags case I expected this optimization to dominantly trigger on CONFIG_NUMA_BALANCING=y kernels - but even with that feature disabled it triggers rather often.
There's two reasons for that:
1)
While sys_mprotect() already optimizes the same-flag case:
if (newflags == oldflags) {
*pprev = vma;
return 0;
}and this test works in many cases, but it is too sharp in some others, where it differentiates between protection values that the underlying PTE format makes no distinction about, such as PROT_EXEC == PROT_READ on x86.
2)
Even where the pte format over vma flag changes necessiates a modification of the pagetables, there might be no pagetables yet to modify: they might not be instantiated yet.
During a regular desktop bootup this optimization hits a couple of hundred times. During a Java test I measured thousands of hits.
So this optimization improves sys_mprotect() in general, not just CONFIG_NUMA_BALANCING=y kernels.
[ We could further increase the efficiency of this optimization if
change_pte_range() and change_huge_pmd() was a bit smarter about recognizing exact-same-value protection masks - when the hardware can do that safely. This would probably further speed up mprotect(). ]
--- mm/mprotect.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/mprotect.c +3 -1
@@ -153,7 +153,9 @@ static unsigned long change_protection_range(struct vm_area_struct *vma, dirty_accountable); }while (pgd++, addr = next, addr != end); -flush_tlb_range (vma, start, end); +/* Only flush the TLB if we actually modified any entries: */ +if (pages) +flush_tlb_range (vma, start, end);return pages; }
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 8/27 sched, numa, mm: Add last_cpu to page flags
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
Introduce a per-page last_cpu field, fold this into the struct page::flags field whenever possible.
The unlikely/rare 32bit NUMA configs will likely grow the page-frame.
[ Completely dropping 32bit support for CONFIG_NUMA_BALANCING would simplify
things, but it would also remove the warning if we grow enough 64bit only page-flags to push the last-cpu out. ]
--- include/linux/mm.h | 90 +++++++++++++++++++++------------------ include/linux/mm_types.h | 5 +++ include/linux/mmzone.h | 14 +----- include/linux/page-flags-layout.h | 83 ++++++++++++++++++++++++++++++++++++ kernel/bounds.c | 4 ++ mm/memory.c | 4 ++ 6 files changed, 146 insertions(+), 54 deletions(-) create mode 100644 include/linux/page-flags-layout.h
include/linux/mm.h +49 -41
@@ -581,50 +581,11 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)* sets it, so none of the operations on it need to be atomic. */ - -/* -* page->flags layout: -* -* There are three possibilities for how page->flags get -* laid out. The first is for the normal case, without -* sparsemem. The second is for sparsemem when there is -* plenty of space for node and section. The last is when -* we have run out of space and have to fall back to an -* alternate (slower) way of determining the node. -* -* No sparsemem or sparsemem vmemmap: | NODE | ZONE | ... | FLAGS | -* classic sparse with space for node:| SECTION | NODE | ZONE | ... | FLAGS | -* classic sparse no space for node: | SECTION | ZONE | ... | FLAGS | -*/ -#if defined(CONFIG_SPARSEMEM ) && !defined(CONFIG_SPARSEMEM_VMEMMAP ) -#define SECTIONS_WIDTH SECTIONS_SHIFT -#else -#define SECTIONS_WIDTH 0 -#endif - -#define ZONES_WIDTH ZONES_SHIFT - -#if SECTIONS_WIDTH +ZONES_WIDTH +NODES_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS -#define NODES_WIDTH NODES_SHIFT -#else -#ifdef CONFIG_SPARSEMEM_VMEMMAP -#error "Vmemmap: No space for nodes field in page flags" -#endif -#define NODES_WIDTH 0 -#endif - -/* Page flags: | [SECTION] | [NODE] | ZONE | ... | FLAGS | */ +/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_CPU] | ... | FLAGS | */ #define SECTIONS_PGOFF ((sizeof(unsigned long)*8) - SECTIONS_WIDTH)#define NODES_PGOFF (SECTIONS_PGOFF - NODES_WIDTH)#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH) - -/* -* We are going to use the flags for the page to node mapping if its in -* there. This includes the case where there is no node, so it is implicit. -*/ -#if !(NODES_WIDTH >0 ||NODES_SHIFT ==0 ) -#define NODE_NOT_IN_PAGE_FLAGS -#endif +#define LAST_CPU_PGOFF (ZONES_PGOFF - LAST_CPU_WIDTH)/* * Define the bit shifts to access each section. For non-existent
@@ -634,6 +595,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)#define SECTIONS_PGSHIFT (SECTIONS_PGOFF * (SECTIONS_WIDTH != 0))#define NODES_PGSHIFT (NODES_PGOFF * (NODES_WIDTH != 0))#define ZONES_PGSHIFT (ZONES_PGOFF * (ZONES_WIDTH != 0)) +#define LAST_CPU_PGSHIFT (LAST_CPU_PGOFF * (LAST_CPU_WIDTH != 0))/* NODE:ZONE or SECTION:ZONE is used to ID a zone for the buddy allocator */ #ifdef NODE_NOT_IN_PAGE_FLAGS
@@ -655,6 +617,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)#define ZONES_MASK ((1UL << ZONES_WIDTH) - 1)#define NODES_MASK ((1UL << NODES_WIDTH) - 1)#define SECTIONS_MASK ((1UL << SECTIONS_WIDTH) - 1) +#define LAST_CPU_MASK ((1UL << LAST_CPU_WIDTH) - 1)#define ZONEID_MASK ((1UL << ZONEID_SHIFT) - 1)static inline enum zone_type page_zonenum (const struct page *page)
@@ -693,6 +656,51 @@ static inline int page_to_nid(const struct page *page) }#endif +#ifdef CONFIG_NUMA_BALANCING +#ifdef LAST_CPU_NOT_IN_PAGE_FLAGS +static inline int page_xchg_last_cpu (struct page *page,int cpu) +{ +return xchg (&page->_last_cpu, cpu); +} + +static inline int page_last_cpu (struct page *page) +{ +return page->_last_cpu; +} +#else +static inline int page_xchg_last_cpu (struct page *page,int cpu) +{ +unsigned long old_flags, flags; +int last_cpu; + +do { + old_flags = flags = page->flags; + last_cpu = (flags >>LAST_CPU_PGSHIFT ) &LAST_CPU_MASK ; + + flags &= ~(LAST_CPU_MASK <<LAST_CPU_PGSHIFT ); + flags |= (cpu &LAST_CPU_MASK ) <<LAST_CPU_PGSHIFT ; + }while (unlikely (cmpxchg (&page->flags, old_flags, flags) != old_flags)); + +return last_cpu; +} + +static inline int page_last_cpu (struct page *page) +{ +return (page->flags >>LAST_CPU_PGSHIFT ) &LAST_CPU_MASK ; +} +#endif /* LAST_CPU_NOT_IN_PAGE_FLAGS */ +#else /* CONFIG_NUMA_BALANCING */ +static inline int page_xchg_last_cpu (struct page *page,int cpu) +{ +return page_to_nid (page); +} + +static inline int page_last_cpu (struct page *page) +{ +return page_to_nid (page); +} +#endif /* CONFIG_NUMA_BALANCING */ +static inline struct zone *page_zone (const struct page *page) {return &NODE_DATA (page_to_nid (page))->node_zones[page_zonenum (page)];
include/linux/mm_types.h +5 -0
@@ -12,6 +12,7 @@#include <linux/cpumask.h> #include <linux/page-debug-flags.h> #include <linux/uprobes.h> +#include <linux/page-flags-layout.h> #include <asm/page.h> #include <asm/mmu.h>
@@ -175,6 +176,10 @@ struct page {*/ void *shadow;#endif + +#ifdef LAST_CPU_NOT_IN_PAGE_FLAGS +int _last_cpu; +#endif }/* * The struct page can be forced to be double word aligned so that atomic ops
include/linux/mmzone.h +1 -13
@@ -15,7 +15,7 @@#include <linux/seqlock.h> #include <linux/nodemask.h> #include <linux/pageblock-flags.h> -#include <generated/bounds.h> +#include <linux/page-flags-layout.h> #include <linux/atomic.h> #include <asm/page.h>
@@ -318,16 +318,6 @@ enum zone_type {* match the requested limits. See gfp_zone() in include/linux/gfp.h */ -#if MAX_NR_ZONES <2 -#define ZONES_SHIFT 0 -#elif MAX_NR_ZONES <=2 -#define ZONES_SHIFT 1 -#elif MAX_NR_ZONES <=4 -#define ZONES_SHIFT 2 -#else -#error ZONES_SHIFT -- too many zones configured adjust calculation -#endif -struct zone {/* Fields commonly accessed by the page allocator */
@@ -1030,8 +1020,6 @@ static inline unsigned long early_pfn_to_nid(unsigned long pfn)* PA_SECTION_SHIFT physical address to/from section number * PFN_SECTION_SHIFT pfn to/from section number */ -#define SECTIONS_SHIFT (MAX_PHYSMEM_BITS - SECTION_SIZE_BITS) -#define PA_SECTION_SHIFT (SECTION_SIZE_BITS)#define PFN_SECTION_SHIFT (SECTION_SIZE_BITS - PAGE_SHIFT)
include/linux/page-flags-layout.h new file +83 -0
83 lines not on this page · raw message
kernel/bounds.c +4 -0
@@ -10,6 +10,7 @@#include <linux/mmzone.h> #include <linux/kbuild.h> #include <linux/page_cgroup.h> +#include <linux/log2.h> void foo (void ) {
@@ -17,5 +18,8 @@ void foo(void)DEFINE (NR_PAGEFLAGS , __NR_PAGEFLAGS);DEFINE (MAX_NR_ZONES , __MAX_NR_ZONES);DEFINE (NR_PCG_FLAGS , __NR_PCG_FLAGS); +#ifdef CONFIG_SMP +DEFINE (NR_CPUS_BITS ,ilog2 (CONFIG_NR_CPUS )); +#endif /* End of constants */ }
mm/memory.c +4 -0
@@ -67,6 +67,10 @@#include "internal.h" +#ifdef LAST_CPU_NOT_IN_PAGE_FLAGS +#warning Unfortunate NUMA config, growing page-frame for last_cpu. +#endif +#ifndef CONFIG_NEED_MULTIPLE_NODES /* use the per-pgdat data instead for discontigmem - mbligh */ unsigned long max_mapnr;
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 9/27 sched, mm, numa: Create generic NUMA fault infrastructure, with architectures overrides
In reply to Ingo MolnarThis patch is based on patches written by multiple people:
Hugh Dickins <[email protected]>
Johannes Weiner <[email protected]>
Peter Zijlstra <[email protected]>
Of the "mm/mpol: Create special PROT_NONE infrastructure" patch and its variants.
I have reworked the code so significantly that I had to drop the acks and signoffs.
In order to facilitate a lazy -- fault driven -- migration of pages, create a special transient PROT_NONE variant, we can then use the 'spurious' protection faults to drive our migrations from.
Pages that already had an effective PROT_NONE mapping will not be detected to generate these 'spuriuos' faults for the simple reason that we cannot distinguish them on their protection bits, see pte_numa().
This isn't a problem since PROT_NONE (and possible PROT_WRITE with dirty tracking) aren't used or are rare enough for us to not care about their placement.
Architectures can set the CONFIG_ARCH_WANTS_NUMA_GENERIC_PGPROT Kconfig variable, in which case they get the PROT_NONE variant. Alternatively they can provide the basic primitives themselves:
bool pte_numa(struct vm_area_struct *vma, pte_t pte); pte_t pte_mknuma(struct vm_area_struct *vma, pte_t pte); bool pmd_numa(struct vm_area_struct *vma, pmd_t pmd); pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot) unsigned long change_prot_numa(struct vm_area_struct *vma, unsigned long start, unsigned long end);
[ This non-generic angle is untested though. ]
Also-From: Johannes Weiner <[email protected]>
Also-From: Hugh Dickins <[email protected]>
Also-From: Peter Zijlstra <[email protected]>
--- include/asm-generic/pgtable.h | 55 ++++++++++++++ include/linux/huge_mm.h | 12 ++++ include/linux/mempolicy.h | 6 ++ include/linux/migrate.h | 5 ++ include/linux/mm.h | 5 ++ include/linux/sched.h | 2 + init/Kconfig | 22 ++++++ mm/Makefile | 1 + mm/huge_memory.c | 162 ++++++++++++++++++++++++++++++++++++++++++ mm/internal.h | 5 +- mm/memcontrol.c | 7 +- mm/memory.c | 85 ++++++++++++++++++++-- mm/migrate.c | 2 +- mm/mprotect.c | 7 -- mm/numa.c | 73 +++++++++++++++++++ 15 files changed, 430 insertions(+), 19 deletions(-) create mode 100644 mm/numa.c
include/asm-generic/pgtable.h +55 -0
@@ -537,6 +537,61 @@ static inline int pmd_none_or_trans_huge_or_clear_bad(pmd_t *pmd) }/* +* Is this pte used for NUMA scanning? +*/ +#ifdef CONFIG_ARCH_USES_NUMA_GENERIC_PGPROT +extern bool pte_numa (struct vm_area_struct *vma,pte_t pte); +#else +# ifndef pte_numa +static inline bool pte_numa (struct vm_area_struct *vma,pte_t pte) +{ +return false; +} +# endif +#endif + +/* +* Turn a pte into a NUMA entry: +*/ +#ifdef CONFIG_ARCH_USES_NUMA_GENERIC_PGPROT +extern pte_t pte_mknuma (struct vm_area_struct *vma,pte_t pte); +#else +# ifndef pte_mknuma +static inline pte_t pte_mknuma (struct vm_area_struct *vma,pte_t pte) +{ +return pte; +} +# endif +#endif + +/* +* Is this pmd used for NUMA scanning? +*/ +#ifdef CONFIG_ARCH_USES_NUMA_GENERIC_PGPROT_HUGEPAGE +extern bool pmd_numa (struct vm_area_struct *vma,pmd_t pmd); +#else +# ifndef pmd_numa +static inline bool pmd_numa (struct vm_area_struct *vma,pmd_t pmd) +{ +return false; +} +# endif +#endif + +/* +* Some architectures (such as x86) may need to preserve certain pgprot +* bits, without complicating generic pgprot code. +* +* Most architectures don't care: +*/ +#ifndef pgprot_modify +static inline pgprot_t pgprot_modify (pgprot_t oldprot,pgprot_t newprot) +{ +return newprot; +} +#endif + +/* * This is a noop if Transparent Hugepage Support is not built into * the kernel. Otherwise it is equivalent to * pmd_none_or_trans_huge_or_clear_bad(), and shall only be called in
include/linux/huge_mm.h +12 -0
@@ -197,4 +197,16 @@ static inline int pmd_trans_huge_lock(pmd_t *pmd, }#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#ifdef CONFIG_NUMA_BALANCING_HUGEPAGE +extern void do_huge_pmd_numa_page (struct mm_struct *mm,struct vm_area_struct *vma, +unsigned long address,pmd_t *pmd, +unsigned int flags,pmd_t orig_pmd); +#else +static inline void do_huge_pmd_numa_page (struct mm_struct *mm,struct vm_area_struct *vma, +unsigned long address,pmd_t *pmd, +unsigned int flags,pmd_t orig_pmd) +{ +} +#endif +#endif /* _LINUX_HUGE_MM_H */
include/linux/mempolicy.h +6 -0
@@ -324,4 +324,10 @@ static inline int mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol, }#endif /* CONFIG_NUMA */ + +static inline int mpol_misplaced (struct page *page,struct vm_area_struct *vma, +unsigned long address) +{ +return -1 ;/* no node preference */ +}#endif
include/linux/migrate.h +5 -0
@@ -64,4 +64,9 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#define fail_migrate_page NULL#endif /* CONFIG_MIGRATION */ +static inline +int migrate_misplaced_page (struct page *page,int node) +{ +return -EAGAIN ;/* can't migrate now */ +}#endif /* _LINUX_MIGRATE_H */
include/linux/mm.h +5 -0
@@ -1559,6 +1559,11 @@ static inline pgprot_t vm_get_page_prot(unsigned long vm_flags) }#endif +#ifdef CONFIG_ARCH_USES_NUMA_GENERIC_PGPROT +extern unsigned long +change_prot_numa (struct vm_area_struct *vma,unsigned long start,unsigned long end); +#endif +struct vm_area_struct *find_extend_vma (struct mm_struct *,unsigned long addr);int remap_pfn_range (struct vm_area_struct *,unsigned long addr,unsigned long pfn,unsigned long size,pgprot_t );
include/linux/sched.h +2 -0
@@ -1575,6 +1575,8 @@ struct task_struct {/* Future-safe accessor for struct task_struct's cpus_allowed. */ #define tsk_cpus_allowed (tsk) (&(tsk)->cpus_allowed) +static inline void task_numa_fault (int node,int cpu,int pages) { } +/* * Priority of a process goes from 0..MAX_PRIO-1, valid RT * priority is 0..MAX_RT_PRIO-1, and SCHED_NORMAL/SCHED_BATCH
init/Kconfig +22 -0
@@ -696,6 +696,28 @@ config LOG_BUF_SHIFT config HAVE_UNSTABLE_SCHED_CLOCK bool +# +# Helper Kconfig switches to express compound feature dependencies +# and thus make the .h/.c code more readable: +# +config NUMA_BALANCING_HUGEPAGE + bool + default y + depends on NUMA_BALANCING + depends on TRANSPARENT_HUGEPAGE + +config ARCH_USES_NUMA_GENERIC_PGPROT + bool + default y + depends on ARCH_WANTS_NUMA_GENERIC_PGPROT + depends on NUMA_BALANCING + +config ARCH_USES_NUMA_GENERIC_PGPROT_HUGEPAGE + bool + default y + depends on ARCH_USES_NUMA_GENERIC_PGPROT + depends on TRANSPARENT_HUGEPAGE + menuconfig CGROUPS boolean "Control Group support" depends on EVENTFD
mm/Makefile +1 -0
@@ -34,6 +34,7 @@ obj-$(CONFIG_FRONTSWAP) += frontswap.o obj-$(CONFIG_HAS_DMA) += dmapool.o obj-$(CONFIG_HUGETLBFS) += hugetlb.o obj-$(CONFIG_NUMA) += mempolicy.o +obj-$(CONFIG_ARCH_USES_NUMA_GENERIC_PGPROT) += numa.o obj-$(CONFIG_SPARSEMEM) += sparse.o obj-$(CONFIG_SPARSEMEM_VMEMMAP) += sparse-vmemmap.o obj-$(CONFIG_SLOB) += slob.o
mm/huge_memory.c +162 -0
180 lines not on this page · raw message
mm/internal.h +3 -2
@@ -212,11 +212,12 @@ static inline void mlock_migrate_page(struct page *newpage, struct page *page) {if (TestClearPageMlocked (page)) {unsigned long flags; +int nr_pages =hpage_nr_pages (page);local_irq_save (flags); -__dec_zone_page_state (page,NR_MLOCK ); +__mod_zone_page_state (page_zone (page),NR_MLOCK , -nr_pages);SetPageMlocked (newpage); -__inc_zone_page_state (newpage,NR_MLOCK ); +__mod_zone_page_state (page_zone (newpage),NR_MLOCK , nr_pages);local_irq_restore (flags); } }
mm/memcontrol.c +5 -2
@@ -3255,15 +3255,18 @@ void mem_cgroup_prepare_migration(struct page *page, struct page *newpage, struct mem_cgroup **memcgp) {struct mem_cgroup *memcg =NULL ; +unsigned int nr_pages =1 ;struct page_cgroup *pc;enum charge_type ctype; *memcgp =NULL ; -VM_BUG_ON (PageTransHuge (page));if (mem_cgroup_disabled ())return ; +if (PageTransHuge (page)) + nr_pages <<=compound_order (page); + pc =lookup_page_cgroup (page);lock_page_cgroup (pc);if (PageCgroupUsed (pc)) {
@@ -3325,7 +3328,7 @@ void mem_cgroup_prepare_migration(struct page *page, struct page *newpage,* charged to the res_counter since we plan on replacing the * old one and only one page is going to be left afterwards. */ -__mem_cgroup_commit_charge (memcg, newpage,1 , ctype, false); +__mem_cgroup_commit_charge (memcg, newpage, nr_pages, ctype, false); }/* remove redundant charge if migration failed*/
mm/memory.c +78 -7
121 lines not on this page · raw message
mm/migrate.c +1 -1
@@ -407,7 +407,7 @@ int migrate_huge_page_move_mapping(struct address_space *mapping,*/ void migrate_page_copy (struct page *newpage,struct page *page) { -if (PageHuge (page)) +if (PageHuge (page) ||PageTransHuge (page))copy_huge_page (newpage, page);else copy_highpage (newpage, page);
mm/mprotect.c +0 -7
@@ -28,13 +28,6 @@#include <asm/cacheflush.h> #include <asm/tlbflush.h> -#ifndef pgprot_modify -static inline pgprot_t pgprot_modify (pgprot_t oldprot,pgprot_t newprot) -{ -return newprot; -} -#endif -static unsigned long change_pte_range (struct mm_struct *mm,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,int dirty_accountable)
mm/numa.c new file +73 -0
73 lines not on this page · raw message
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 10/27 sched: Make find_busiest_queue() a method
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
Its a bit awkward but it was the least painful means of modifying the queue selection. Used in a later patch to conditionally use a random queue.
--- kernel/sched/fair.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-)
kernel/sched/fair.c +12 -8
@@ -3600,6 +3600,9 @@ struct lb_env {unsigned int loop;unsigned int loop_break;unsigned int loop_max; + +struct rq * (*find_busiest_queue)(struct lb_env *, +struct sched_group *); };/*
@@ -4779,13 +4782,14 @@ static int load_balance(int this_cpu, struct rq *this_rq,struct cpumask *cpus =__get_cpu_var (load_balance_tmpmask);struct lb_env env = { - .sd = sd, - .dst_cpu = this_cpu, - .dst_rq = this_rq, - .dst_grpmask =sched_group_cpus (sd->groups), - .idle = idle, - .loop_break = sched_nr_migrate_break, - .cpus = cpus, + .sd = sd, + .dst_cpu = this_cpu, + .dst_rq = this_rq, + .dst_grpmask =sched_group_cpus (sd->groups), + .idle = idle, + .loop_break = sched_nr_migrate_break, + .cpus = cpus, + .find_busiest_queue = find_busiest_queue, };cpumask_copy (cpus, cpu_active_mask);
@@ -4804,7 +4808,7 @@ redo: goto out_balanced; } - busiest =find_busiest_queue (&env, group); + busiest = env.find_busiest_queue (&env, group);if (!busiest) {schedstat_inc (sd, lb_nobusyq[idle]); goto out_balanced;
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 11/27 sched, numa, mm: Describe the NUMA scheduling problem formally
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
This is probably a first: formal description of a complex high-level computing problem, within the kernel source.
--- Documentation/scheduler/numa-problem.txt | 230 +++++++++++++++++++++++++++++++ 1 file changed, 230 insertions(+) create mode 100644 Documentation/scheduler/numa-problem.txt
Documentation/scheduler/numa-problem.txt new file +230 -0
@@ -0,0 +1,230 @@ + + +Effective NUMA scheduling problem statement, described formally: + + * minimize interconnect traffic + +For each task 't_i' we have memory, this memory can be spread over multiple +physical nodes, let us denote this as: 'p_i,k', the memory task 't_i' has on +node 'k' in [pages]. + +If a task shares memory with another task let us denote this as: +'s_i,k', the memory shared between tasks including 't_i' residing on node +'k'. + +Let 'M' be the distribution that governs all 'p' and 's', ie. the page placement. + +Similarly, lets define 'fp_i,k' and 'fs_i,k' resp. as the (average) usage +frequency over those memory regions [1/s] such that the product gives an +(average) bandwidth 'bp' and 'bs' in [pages/s]. + +(note: multiple tasks sharing memory naturally avoid duplicat accounting + because each task will have its own access frequency 'fs') + +(pjt: I think this frequency is more numerically consistent if you explicitly + restrict p/s above to be the working-set. (It also makes explicit the + requirement for <C0,M0> to change about a change in the working set.) + + Doing this does have the nice property that it lets you use your frequency + measurement as a weak-ordering for the benefit a task would receive when + we can't fit everything. + + e.g. task1 has working set 10mb, f=90% + task2 has working set 90mb, f=10% + + Both are using 9mb/s of bandwidth, but we'd expect a much larger benefit + from task1 being on the right node than task2. ) + +Let 'C' map every task 't_i' to a cpu 'c_i' and its corresponding node 'n_i': + + C: t_i -> {c_i, n_i} + +This gives us the total interconnect traffic between nodes 'k' and 'l', +'T_k,l', as: + + T_k,l = \Sum_i bp_i,l + bs_i,l + \Sum bp_j,k + bs_j,k where n_i == k, n_j == l + +And our goal is to obtain C0 and M0 such that: + + T_k,l(C0, M0) =< T_k,l(C, M) for all C, M where k != l + +(note: we could introduce 'nc(k,l)' as the cost function of accessing memory + on node 'l' from node 'k', this would be useful for bigger NUMA systems + + pjt: I agree nice to have, but intuition suggests diminishing returns on more + usual systems given factors like things like Haswell's enormous 35mb l3 + cache and QPI being able to do a direct fetch.) + +(note: do we need a limit on the total memory per node?) + + + * fairness + +For each task 't_i' we have a weight 'w_i' (related to nice), and each cpu +'c_n' has a compute capacity 'P_n', again, using our map 'C' we can formulate a +load 'L_n': + + L_n = 1/P_n * \Sum_i w_i for all c_i = n + +using that we can formulate a load difference between CPUs + + L_n,m = | L_n - L_m | + +Which allows us to state the fairness goal like: + + L_n,m(C0) =< L_n,m(C) for all C, n != m + +(pjt: It can also be usefully stated that, having converged at C0: + + | L_n(C0) - L_m(C0) | <= 4/3 * | G_n( U(t_i, t_j) ) - G_m( U(t_i, t_j) ) | + + Where G_n,m is the greedy partition of tasks between L_n and L_m. This is + the "worst" partition we should accept; but having it gives us a useful + bound on how much we can reasonably adjust L_n/L_m at a Pareto point to + favor T_n,m. ) + +Together they give us the complete multi-objective optimization problem: + + min_C,M [ L_n,m(C), T_k,l(C,M) ] + + + +Notes: + + - the memory bandwidth problem is very much an inter-process problem, in + particular there is no such concept as a process in the above problem. + + - the naive solution would completely prefer fairness over interconnect + traffic, the more complicated solution could pick another Pareto point using + an aggregate objective function such that we balance the loss of work + efficiency against the gain of running, we'd want to more or less suggest + there to be a fixed bound on the error from the Pareto line for any + such solution. + +References: + + http://en.wikipedia.org/wiki/Mathematical_optimization + http://en.wikipedia.org/wiki/Multi-objective_optimization + + +* warning, significant hand-waving ahead, improvements welcome * + + +Partial solutions / approximations: + + 1) have task node placement be a pure preference from the 'fairness' pov. + +This means we always prefer fairness over interconnect bandwidth. This reduces +the problem to: + + min_C,M [ T_k,l(C,M) ] + + 2a) migrate memory towards 'n_i' (the task's node). + +This creates memory movement such that 'p_i,k for k != n_i' becomes 0 -- +provided 'n_i' stays stable enough and there's sufficient memory (looks like +we might need memory limits for this). + +This does however not provide us with any 's_i' (shared) information. It does +however remove 'M' since it defines memory placement in terms of task +placement. + +XXX properties of this M vs a potential optimal + + 2b) migrate memory towards 'n_i' using 2 samples. + +This separates pages into those that will migrate and those that will not due +to the two samples not matching. We could consider the first to be of 'p_i' +(private) and the second to be of 's_i' (shared). + +This interpretation can be motivated by the previously observed property that +'p_i,k for k != n_i' should become 0 under sufficient memory, leaving only +'s_i' (shared). (here we loose the need for memory limits again, since it +becomes indistinguishable from shared). + +XXX include the statistical babble on double sampling somewhere near + +This reduces the problem further; we loose 'M' as per 2a, it further reduces +the 'T_k,l' (interconnect traffic) term to only include shared (since per the +above all private will be local): + + T_k,l = \Sum_i bs_i,l for every n_i = k, l != k + +[ more or less matches the state of sched/numa and describes its remaining + problems and assumptions. It should work well for tasks without significant + shared memory usage between tasks. ] + +Possible future directions: + +Motivated by the form of 'T_k,l', try and obtain each term of the sum, so we +can evaluate it; + + 3a) add per-task per node counters + +At fault time, count the number of pages the task faults on for each node. +This should give an approximation of 'p_i' for the local node and 's_i,k' for +all remote nodes. + +While these numbers provide pages per scan, and so have the unit [pages/s] they +don't count repeat access and thus aren't actually representable for our +bandwidth numberes. + + 3b) additional frequency term + +Additionally (or instead if it turns out we don't need the raw 'p' and 's' +numbers) we can approximate the repeat accesses by using the time since marking +the pages as indication of the access frequency. + +Let 'I' be the interval of marking pages and 'e' the elapsed time since the +last marking, then we could estimate the number of accesses 'a' as 'a = I / e'. +If we then increment the node counters using 'a' instead of 1 we might get +a better estimate of bandwidth terms. + + 3c) additional averaging; can be applied on top of either a/b. + +[ Rik argues that decaying averages on 3a might be sufficient for bandwidth since + the decaying avg includes the old accesses and therefore has a measure of repeat + accesses. + + Rik also argued that the sample frequency is too low to get accurate access + frequency measurements, I'm not entirely convinced, event at low sample + frequencies the avg elapsed time 'e' over multiple samples should still + give us a fair approximation of the avg access frequency 'a'. + + So doing both b&c has a fair chance of working and allowing us to distinguish + between important and less important memory accesses. + + Experimentation has shown no benefit from the added frequency term so far. ] + +This will give us 'bp_i' and 'bs_i,k' so that we can approximately compute +'T_k,l' Our optimization problem now reads: + + min_C [ \Sum_i bs_i,l for every n_i = k, l != k ] + +And includes only shared terms, this makes sense since all task private memory +will become local as per 2. + +This suggests that if there is significant shared memory, we should try and +move towards it. + + 4) move towards where 'most' memory is + +The simplest significance test is comparing the biggest shared 's_i,k' against +the private 'p_i'. If we have more shared than private, move towards it. + +This effectively makes us move towards where most our memory is and forms a +feed-back loop with 2. We migrate memory towards us and we migrate towards +where 'most' memory is. + +(Note: even if there were two tasks fully trashing the same shared memory, it + is very rare for there to be an 50/50 split in memory, lacking a perfect + split, the small will move towards the larger. In case of the perfect + split, we'll tie-break towards the lower node number.) + + 5) 'throttle' 4's node placement + +Since per 2b our 's_i,k' and 'p_i' require at least two scans to 'stabilize' +and show representative numbers, we should limit node-migration to not be +faster than this. + + n) poke holes in previous that require more stuff and describe it.
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 12/27 numa, mm: Support NUMA hinting page faults from gup/gup_fast
In reply to Ingo MolnarFrom: Andrea Arcangeli <[email protected]>
Introduce FOLL_NUMA to tell follow_page to check pte/pmd_numa. get_user_pages must use FOLL_NUMA, and it's safe to do so because it always invokes handle_mm_fault and retries the follow_page later.
KVM secondary MMU page faults will trigger the NUMA hinting page faults through gup_fast -> get_user_pages -> follow_page -> handle_mm_fault.
Other follow_page callers like KSM should not use FOLL_NUMA, or they would fail to get the pages if they use follow_page instead of get_user_pages.
[ This patch was picked up from the AutoNUMA tree. ]
--- include/linux/mm.h | 1 + mm/memory.c | 17 +++++++++++++++++ 2 files changed, 18 insertions(+)
include/linux/mm.h +1 -0
@@ -1585,6 +1585,7 @@ struct page *follow_page(struct vm_area_struct *, unsigned long address,#define FOLL_MLOCK 0x40/* mark page as mlocked */ #define FOLL_SPLIT 0x80/* don't return transhuge pages, split them */ #define FOLL_HWPOISON 0x100/* check page is hwpoisoned */ +#define FOLL_NUMA 0x200/* force NUMA hinting page fault */ typedef int (*pte_fn_t )(pte_t *pte,pgtable_t token,unsigned long addr,void *data);
mm/memory.c +17 -0
@@ -1522,6 +1522,8 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, page =follow_huge_pmd (mm, address, pmd, flags &FOLL_WRITE ); goto out; } +if ((flags &FOLL_NUMA ) &&pmd_numa (vma, *pmd)) + goto no_page_table;if (pmd_trans_huge (*pmd)) {if (flags &FOLL_SPLIT ) {split_huge_page_pmd (mm, pmd);
@@ -1551,6 +1553,8 @@ split_fallthrough: pte = *ptep;if (!pte_present (pte)) goto no_page; +if ((flags &FOLL_NUMA ) &&pte_numa (vma, pte)) + goto no_page;if ((flags &FOLL_WRITE ) && !pte_write (pte)) goto unlock;
@@ -1702,6 +1706,19 @@ int __get_user_pages(struct task_struct *tsk, struct mm_struct *mm, (VM_WRITE | VM_MAYWRITE) : (VM_READ | VM_MAYREAD); vm_flags &= (gup_flags &FOLL_FORCE ) ? (VM_MAYREAD |VM_MAYWRITE ) : (VM_READ |VM_WRITE ); + +/* +* If FOLL_FORCE and FOLL_NUMA are both set, handle_mm_fault +* would be called on PROT_NONE ranges. We must never invoke +* handle_mm_fault on PROT_NONE ranges or the NUMA hinting +* page faults would unprotect the PROT_NONE ranges if +* _PAGE_NUMA and _PAGE_PROTNONE are sharing the same pte/pmd +* bitflag. So to avoid that, don't set FOLL_NUMA if +* FOLL_FORCE is set. +*/ +if (!(gup_flags &FOLL_FORCE )) + gup_flags |=FOLL_NUMA ; + i =0 ; do {
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 13/27 mm/migrate: Introduce migrate_misplaced_page()
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
Add migrate_misplaced_page() which deals with migrating pages from faults.
This includes adding a new MIGRATE_FAULT migration mode to deal with the extra page reference required due to having to look up the page.
--- include/linux/migrate.h | 4 ++- include/linux/migrate_mode.h | 3 ++ mm/migrate.c | 79 +++++++++++++++++++++++++++++++++++++++----- 3 files changed, 77 insertions(+), 9 deletions(-)
include/linux/migrate.h +3 -1
@@ -30,6 +30,7 @@ extern int migrate_vmas(struct mm_struct *mm,extern void migrate_page_copy (struct page *newpage,struct page *page);extern int migrate_huge_page_move_mapping (struct address_space *mapping,struct page *newpage,struct page *page); +extern int migrate_misplaced_page (struct page *page,int node);#else static inline void putback_lru_pages (struct list_head *l) {}
@@ -63,10 +64,11 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#define migrate_page NULL#define fail_migrate_page NULL -#endif /* CONFIG_MIGRATION */ static inline int migrate_misplaced_page (struct page *page,int node) {return -EAGAIN ;/* can't migrate now */ } + +#endif /* CONFIG_MIGRATION */ #endif /* _LINUX_MIGRATE_H */
include/linux/migrate_mode.h +3 -0
@@ -6,11 +6,14 @@* on most operations but not ->writepage as the potential stall time * is too significant * MIGRATE_SYNC will block when migrating pages +* MIGRATE_FAULT called from the fault path to migrate-on-fault for mempolicy +* this path has an extra reference count */ enum migrate_mode {MIGRATE_ASYNC ,MIGRATE_SYNC_LIGHT ,MIGRATE_SYNC , +MIGRATE_FAULT , };#endif /* MIGRATE_MODE_H_INCLUDED */
mm/migrate.c +71 -8
122 lines not on this page · raw message
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 14/27 sched, numa, mm, arch: Add variable locality exception
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
Some architectures (ab)use NUMA to represent different memory regions all cpu-local but of different latencies, such as SuperH.
The naming comes from Mel Gorman.
--- arch/sh/mm/Kconfig | 1 + init/Kconfig | 7 +++++++ 2 files changed, 8 insertions(+)
arch/sh/mm/Kconfig +1 -0
@@ -111,6 +111,7 @@ config VSYSCALL config NUMA bool "Non Uniform Memory Access (NUMA) Support" depends on MMU && SYS_SUPPORTS_NUMA && EXPERIMENTAL + select ARCH_WANTS_NUMA_VARIABLE_LOCALITY default n help Some SH systems have many various memories scattered around
init/Kconfig +7 -0
@@ -718,6 +718,13 @@ config ARCH_USES_NUMA_GENERIC_PGPROT_HUGEPAGE depends on ARCH_USES_NUMA_GENERIC_PGPROT depends on TRANSPARENT_HUGEPAGE +# +# For architectures that (ab)use NUMA to represent different memory regions +# all cpu-local but of different latencies, such as SuperH. +# +config ARCH_WANTS_NUMA_VARIABLE_LOCALITY + bool + menuconfig CGROUPS boolean "Control Group support" depends on EVENTFD
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 15/27 sched, numa, mm: Add credits for NUMA placement
In reply to Ingo MolnarFrom: Rik van Riel <[email protected]>
The NUMA placement code has been rewritten several times, but the basic ideas took a lot of work to develop. The people who put in the work deserve credit for it. Thanks Andrea & Peter :)
[ The Documentation/scheduler/numa-problem.txt file should
probably be rewritten once we figure out the final details of what the NUMA code needs to do, and why. ]
----
This is against tip.git numa/core
--- CREDITS | 1 + kernel/sched/fair.c | 3 +++ mm/memory.c | 2 ++ 3 files changed, 6 insertions(+)
CREDITS +1 -0
@@ -125,6 +125,7 @@ D: Author of pscan that helps to fix lp/parport bugs D: Author of lil (Linux Interrupt Latency benchmark) D: Fixed the shm swap deallocation at swapoff time (try_to_unuse message) D: VM hacker +D: NUMA task placement D: Various other kernel hacks S: Imola 40026 S: Italy
kernel/sched/fair.c +3 -0
@@ -18,6 +18,9 @@* * Adaptive scheduling granularity, math enhancements by Peter Zijlstra * Copyright (C) 2007 Red Hat, Inc., Peter Zijlstra <[email protected]> +* +* NUMA placement, statistics and algorithm by Andrea Arcangeli, +* CFS balancing changes by Peter Zijlstra. Copyright (C) 2012 Red Hat, Inc. */ #include <linux/latencytop.h>
mm/memory.c +2 -0
@@ -36,6 +36,8 @@* ([email protected]) * * Aug/Sep 2004 Changed to four level page tables (Andi Kleen) +* +* 2012 - NUMA placement page faults (Andrea Arcangeli, Peter Zijlstra) */ #include <linux/kernel_stat.h>
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 16/27 sched, mm, x86: Add the ARCH_SUPPORTS_NUMA_BALANCING flag
In reply to Ingo MolnarAllow architectures to opt-in to the adaptive affinity NUMA balancing code.
--- init/Kconfig | 7 +++++++ 1 file changed, 7 insertions(+)
init/Kconfig +7 -0
@@ -725,6 +725,13 @@ config ARCH_USES_NUMA_GENERIC_PGPROT_HUGEPAGE config ARCH_WANTS_NUMA_VARIABLE_LOCALITY bool +# +# For architectures that want to enable the PROT_NONE driven, +# NUMA-affine scheduler balancing logic: +# +config ARCH_SUPPORTS_NUMA_BALANCING + bool + menuconfig CGROUPS boolean "Control Group support" depends on EVENTFD
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 17/27 sched, numa, mm: Add the scanning page fault machinery
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
Add the NUMA working set scanning/hinting page fault machinery, with no policy yet.
[ The earliest versions had the mpol_misplaced() function from
Lee Schermerhorn - this was heavily modified later on. ]
--- include/linux/init_task.h | 8 +++ include/linux/mempolicy.h | 6 +- include/linux/mm_types.h | 4 ++ include/linux/sched.h | 41 ++++++++++++-- init/Kconfig | 73 +++++++++++++++++++----- kernel/sched/core.c | 15 +++++ kernel/sysctl.c | 31 ++++++++++- mm/huge_memory.c | 1 + mm/mempolicy.c | 137 ++++++++++++++++++++++++++++++++++++++++++++++ 9 files changed, 294 insertions(+), 22 deletions(-)
include/linux/init_task.h +8 -0
@@ -143,6 +143,13 @@ extern struct task_group root_task_group;#define INIT_TASK_COMM "swapper" +#ifdef CONFIG_NUMA_BALANCING +# define INIT_TASK_NUMA (tsk) \ + .numa_shared = -1, +#else +# define INIT_TASK_NUMA (tsk) +#endif +/* * INIT_TASK is used to set up the first task table, touch at * your own risk!. Base=0, limit=0x1fffff (=2MB)
@@ -210,6 +217,7 @@ extern struct task_group root_task_group; INIT_TRACE_RECURSION \ INIT_TASK_RCU_PREEMPT(tsk) \ INIT_CPUSET_SEQ \ + INIT_TASK_NUMA(tsk) \ }
include/linux/mempolicy.h +4 -2
@@ -198,6 +198,8 @@ static inline int vma_migratable(struct vm_area_struct *vma) return 1; } +extern int mpol_misplaced (struct page *,struct vm_area_struct *,unsigned long ); +#else struct mempolicy {};
@@ -323,11 +325,11 @@ static inline int mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol,return 0 ; } -#endif /* CONFIG_NUMA */ -static inline int mpol_misplaced (struct page *page,struct vm_area_struct *vma,unsigned long address) {return -1 ;/* no node preference */ } + +#endif /* CONFIG_NUMA */ #endif
include/linux/mm_types.h +4 -0
@@ -403,6 +403,10 @@ struct mm_struct {#ifdef CONFIG_CPUMASK_OFFSTACK struct cpumask cpumask_allocation;#endif +#ifdef CONFIG_NUMA_BALANCING +unsigned long numa_next_scan; +int numa_scan_seq; +#endif struct uprobes_state uprobes_state; };
include/linux/sched.h +37 -4
74 lines not on this page · raw message
init/Kconfig +59 -14
85 lines not on this page · raw message
kernel/sched/core.c +15 -0
@@ -1544,6 +1544,21 @@ static void __sched_fork(struct task_struct *p)#ifdef CONFIG_PREEMPT_NOTIFIERS INIT_HLIST_HEAD (&p->preempt_notifiers);#endif + +#ifdef CONFIG_NUMA_BALANCING +if (p->mm &&atomic_read (&p->mm->mm_users) ==1 ) { + p->mm->numa_next_scan = jiffies; + p->mm->numa_scan_seq =0 ; + } + + p->numa_shared =-1 ; + p->node_stamp =0ULL ; + p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq :0 ; + p->numa_migrate_seq =2 ; + p->numa_faults =NULL ; + p->numa_scan_period = sysctl_sched_numa_scan_period_min; + p->numa_work.next = &p->numa_work; +#endif /* CONFIG_NUMA_BALANCING */ }/*
kernel/sysctl.c +29 -2
@@ -256,9 +256,11 @@ static int min_sched_granularity_ns = 100000; /* 100 usecs */static int max_sched_granularity_ns =NSEC_PER_SEC ;/* 1 second */ static int min_wakeup_granularity_ns;/* 0 usecs */ static int max_wakeup_granularity_ns =NSEC_PER_SEC ;/* 1 second */ +#ifdef CONFIG_SMP static int min_sched_tunable_scaling =SCHED_TUNABLESCALING_NONE ;static int max_sched_tunable_scaling =SCHED_TUNABLESCALING_END -1 ; -#endif +#endif /* CONFIG_SMP */ +#endif /* CONFIG_SCHED_DEBUG */ #ifdef CONFIG_COMPACTION static int min_extfrag_threshold;
@@ -301,6 +303,7 @@ static struct ctl_table kern_table[] = { .extra1 = &min_wakeup_granularity_ns, .extra2 = &max_wakeup_granularity_ns, }, +#ifdef CONFIG_SMP { .procname = "sched_tunable_scaling", .data = &sysctl_sched_tunable_scaling,
@@ -347,7 +350,31 @@ static struct ctl_table kern_table[] = { .extra1 = &zero, .extra2 = &one, }, -#endif +#endif /* CONFIG_SMP */ +#ifdef CONFIG_NUMA_BALANCING + { + .procname = "sched_numa_scan_period_min_ms", + .data = &sysctl_sched_numa_scan_period_min, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec, + }, + { + .procname = "sched_numa_scan_period_max_ms", + .data = &sysctl_sched_numa_scan_period_max, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec, + }, + { + .procname = "sched_numa_settle_count", + .data = &sysctl_sched_numa_settle_count, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec, + }, +#endif /* CONFIG_NUMA_BALANCING */ +#endif /* CONFIG_SCHED_DEBUG */ { .procname = "sched_rt_period_us", .data = &sysctl_sched_rt_period,
mm/huge_memory.c +1 -0
@@ -1456,6 +1456,7 @@ static void __split_huge_page_refcount(struct page *page) page_tail->mapping = page->mapping; page_tail->index = page->index + i; +page_xchg_last_cpu (page,page_last_cpu (page_tail));BUG_ON (!PageAnon (page_tail));BUG_ON (!PageUptodate (page_tail));
mm/mempolicy.c +137 -0
143 lines not on this page · raw message
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 18/27 sched: Add adaptive NUMA affinity support
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
The principal ideas behind this patch are the fundamental difference between shared and privately used memory and the very strong desire to only rely on per-task behavioral state for scheduling decisions.
We define 'shared memory' as all user memory that is frequently accessed by multiple tasks and conversely 'private memory' is the user memory used predominantly by a single task.
To approximate the above strict definition we recognise that task placement is dominantly per cpu and thus using cpu granular page access state is a natural fit. Thus we introduce page::last_cpu as the cpu that last accessed a page.
Using this, we can construct two per-task node-vectors, 'S_i' and 'P_i' reflecting the amount of shared and privately used pages of this task respectively. Pages for which two consecutive 'hits' are of the same cpu are assumed private and the others are shared.
[ This means that we will start evaluating this state when the
task has not migrated for at least 2 scans, see NUMA_SETTLE ]
Using these vectors we can compute the total number of shared/private pages of this task and determine which dominates.
[ Note that for shared tasks we only see '1/n' the total number
of shared pages for the other tasks will take the other faults; where 'n' is the number of tasks sharing the memory. So for an equal comparison we should divide total private by 'n' as well, but we don't have 'n' so we pick 2. ]
We can also compute which node holds most of our memory, running on this node will be called 'ideal placement' (As per previous patches we will prefer to pull memory towards wherever we run.)
We change the load-balancer to prefer moving tasks in order of:
1) !numa tasks and numa tasks in the direction of more faults
2) allow !ideal tasks getting worse in the direction of faults
3) allow private tasks to get worse
4) allow shared tasks to get worse
This order ensures we prefer increasing memory locality but when we do have to make hard decisions we prefer spreading private over shared, because spreading shared tasks significantly increases the interconnect bandwidth since not all memory can follow.
We also add an extra 'lateral' force to the load balancer that perturbs the state when otherwise 'fairly' balanced. This ensures we don't get 'stuck' in a state which is fair but undesired from a memory location POV (see can_do_numa_run()).
Lastly, we allow shared tasks to defeat the default spreading of tasks such that, when possible, they can aggregate on a single node.
Shared tasks aggregate for the very simple reason that there has to be a single node that holds most of their memory and a second most, etc.. and tasks want to move up the faults ladder.
Enable it on x86. A number of other architectures are most likely fine too - but they should enable and test this feature explicitly.
--- Documentation/scheduler/numa-problem.txt | 20 +- arch/x86/Kconfig | 2 + include/linux/sched.h | 1 + kernel/sched/core.c | 53 +- kernel/sched/fair.c | 975 +++++++++++++++++++++++++------ kernel/sched/features.h | 8 + kernel/sched/sched.h | 38 +- 7 files changed, 900 insertions(+), 197 deletions(-)
Documentation/scheduler/numa-problem.txt +13 -7
@@ -133,6 +133,8 @@ XXX properties of this M vs a potential optimal 2b) migrate memory towards 'n_i' using 2 samples. +XXX include the statistical babble on double sampling somewhere near + This separates pages into those that will migrate and those that will not due to the two samples not matching. We could consider the first to be of 'p_i' (private) and the second to be of 's_i' (shared).
@@ -142,7 +144,17 @@ This interpretation can be motivated by the previously observed property that 's_i' (shared). (here we loose the need for memory limits again, since it becomes indistinguishable from shared). -XXX include the statistical babble on double sampling somewhere near + 2c) use cpu samples instead of node samples + +The problem with sampling on node granularity is that one looses 's_i' for +the local node, since one cannot distinguish between two accesses from the +same node. + +By increasing the granularity to per-cpu we gain the ability to have both an +'s_i' and 'p_i' per node. Since we do all task placement per-cpu as well this +seems like a natural match. The line where we overcommit cpus is where we loose +granularity again, but when we loose overcommit we naturally spread tasks. +Therefore it should work out nicely. This reduces the problem further; we loose 'M' as per 2a, it further reduces the 'T_k,l' (interconnect traffic) term to only include shared (since per the
@@ -150,12 +162,6 @@ above all private will be local): T_k,l = \Sum_i bs_i,l for every n_i = k, l != k -[ more or less matches the state of sched/numa and describes its remaining - problems and assumptions. It should work well for tasks without significant - shared memory usage between tasks. ] - -Possible future directions: - Motivated by the form of 'T_k,l', try and obtain each term of the sum, so we can evaluate it;
arch/x86/Kconfig +2 -0
@@ -22,6 +22,8 @@ config X86 def_bool y select HAVE_AOUT if X86_32 select HAVE_UNSTABLE_SCHED_CLOCK + select ARCH_SUPPORTS_NUMA_BALANCING + select ARCH_WANTS_NUMA_GENERIC_PGPROT select HAVE_IDE select HAVE_OPROFILE select HAVE_PCSPKR_PLATFORM
include/linux/sched.h +1 -0
@@ -823,6 +823,7 @@ enum cpu_idle_type {#define SD_ASYM_PACKING 0x0800/* Place busy groups earlier in the domain */ #define SD_PREFER_SIBLING 0x1000/* Prefer to place tasks in a sibling domain */ #define SD_OVERLAP 0x2000/* sched_domains of this level overlap */ +#define SD_NUMA 0x4000/* cross-node balancing */ extern int __weak arch_sd_sibiling_asym_packing (void );
kernel/sched/core.c +50 -3
101 lines not on this page · raw message
kernel/sched/fair.c +791 -184
1,305 lines not on this page · raw message
kernel/sched/features.h +8 -0
@@ -66,3 +66,11 @@ SCHED_FEAT(TTWU_QUEUE, true) SCHED_FEAT(FORCE_SD_OVERLAP, false) SCHED_FEAT(RT_RUNTIME_SHARE, true) SCHED_FEAT(LB_MIN, false) + +#ifdef CONFIG_NUMA_BALANCING +/* Do the working set probing faults: */ +SCHED_FEAT(NUMA, true) +SCHED_FEAT(NUMA_FAULTS_UP, true) +SCHED_FEAT(NUMA_FAULTS_DOWN, false) +SCHED_FEAT(NUMA_SETTLE, true) +#endif
kernel/sched/sched.h +35 -3
@@ -3,6 +3,7 @@#include <linux/mutex.h> #include <linux/spinlock.h> #include <linux/stop_machine.h> +#include <linux/slab.h> #include "cpupri.h"
@@ -420,17 +421,29 @@ struct rq {unsigned long cpu_power;unsigned char idle_balance; -/* For active balancing */ int post_schedule; + +/* For active balancing */ int active_balance; -int push_cpu; -struct cpu_stop_work active_balance_work; +int ab_dst_cpu; +int ab_flags; +int ab_failed; +int ab_idle; +struct cpu_stop_work ab_work; +/* cpu of this runqueue: */ int cpu;int online;struct list_head cfs_tasks; +#ifdef CONFIG_NUMA_BALANCING +unsigned long numa_weight; +unsigned long nr_numa_running; +unsigned long nr_ideal_running; +#endif +unsigned long nr_shared_running;/* 0 on non-NUMA */ +u64 rt_avg;u64 age_stamp;u64 idle_stamp;
@@ -501,6 +514,18 @@ DECLARE_PER_CPU(struct rq, runqueues);#define cpu_curr (cpu) (cpu_rq(cpu)->curr)#define raw_rq () (&__raw_get_cpu_var(runqueues)) +#ifdef CONFIG_NUMA_BALANCING +extern void sched_setnuma (struct task_struct *p,int node,int shared); +static inline void task_numa_free (struct task_struct *p) +{ +kfree (p->numa_faults); +} +#else /* CONFIG_NUMA_BALANCING */ +static inline void task_numa_free (struct task_struct *p) +{ +} +#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SMP #define rcu_dereference_check_sched_domain (p) \
@@ -544,6 +569,7 @@ static inline struct sched_domain *highest_flag_domain(int cpu, int flag) DECLARE_PER_CPU(struct sched_domain *, sd_llc);DECLARE_PER_CPU (int, sd_llc_id); +DECLARE_PER_CPU(struct sched_domain *, sd_node);extern int group_balance_cpu (struct sched_group *sg);
@@ -663,6 +689,12 @@ extern struct static_key sched_feat_keys[__SCHED_FEAT_NR];#define sched_feat (x) (sysctl_sched_features & (1UL << __SCHED_FEAT_##x))#endif /* SCHED_DEBUG && HAVE_JUMP_LABEL */ +#ifdef CONFIG_NUMA_BALANCING +#define sched_feat_numa (x) sched_feat(x) +#else +#define sched_feat_numa (x) (0) +#endif +static inline u64 global_rt_period (void ) {return (u64 )sysctl_sched_rt_period *NSEC_PER_USEC ;
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 19/27 sched: Implement constant, per task Working Set Sampling (WSS) rate
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
Previously, to probe the working set of a task, we'd use a very simple and crude method: mark all of its address space PROT_NONE.
That method has various (obvious) disadvantages:
- it samples the working set at dissimilar rates, giving some tasks a sampling quality advantage over others.
- creates performance problems for tasks with very large working sets
- over-samples processes with large address spaces but which only very rarely execute
Improve that method by keeping a rotating offset into the address space that marks the current position of the scan, and advance it by a constant rate (in a CPU cycles execution proportional manner). If the offset reaches the last mapped address of the mm then it then it starts over at the first address.
The per-task nature of the working set sampling functionality in this tree allows such constant rate, per task, execution-weight proportional sampling of the working set, with an adaptive sampling interval/frequency that goes from once per 100 msecs up to just once per 1.6 seconds. The current sampling volume is 256 MB per interval.
As tasks mature and converge their working set, so does the sampling rate slow down to just a trickle, 256 MB per 1.6 seconds of CPU time executed.
This, beyond being adaptive, also rate-limits rarely executing systems and does not over-sample on overloaded systems.
[ In AutoNUMA speak, this patch deals with the effective sampling
rate of the 'hinting page fault'. AutoNUMA's scanning is currently rate-limited, but it is also fundamentally single-threaded, executing in the knuma_scand kernel thread, so the limit in AutoNUMA is global and does not scale up with the number of CPUs, nor does it scan tasks in an execution proportional manner.
So the idea of rate-limiting the scanning was first implemented in the AutoNUMA tree via a global rate limit. This patch goes beyond that by implementing an execution rate proportional working set sampling rate that is not implemented via a single global scanning daemon. ]
[ Dan Carpenter pointed out a possible NULL pointer dereference in the
first version of this patch. ]
--- include/linux/mm_types.h | 1 + include/linux/sched.h | 1 + kernel/sched/fair.c | 41 +++++++++++++++++++++++++++++------------ kernel/sysctl.c | 7 +++++++ 4 files changed, 38 insertions(+), 12 deletions(-)
include/linux/mm_types.h +1 -0
@@ -405,6 +405,7 @@ struct mm_struct {#endif #ifdef CONFIG_NUMA_BALANCING unsigned long numa_next_scan; +unsigned long numa_scan_offset;int numa_scan_seq;#endif struct uprobes_state uprobes_state;
include/linux/sched.h +1 -0
@@ -2047,6 +2047,7 @@ extern enum sched_tunable_scaling sysctl_sched_tunable_scaling;extern unsigned int sysctl_sched_numa_scan_period_min;extern unsigned int sysctl_sched_numa_scan_period_max; +extern unsigned int sysctl_sched_numa_scan_size;extern unsigned int sysctl_sched_numa_settle_count;#ifdef CONFIG_SCHED_DEBUG
kernel/sched/fair.c +29 -12
61 lines not on this page · raw message
kernel/sysctl.c +7 -0
@@ -367,6 +367,13 @@ static struct ctl_table kern_table[] = { .proc_handler = proc_dointvec, }, { + .procname = "sched_numa_scan_size_mb", + .data = &sysctl_sched_numa_scan_size, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec, + }, + { .procname = "sched_numa_settle_count", .data = &sysctl_sched_numa_settle_count, .maxlen = sizeof(unsigned int),
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 20/27 sched, numa, mm: Count WS scanning against present PTEs, not virtual memory ranges
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
By accounting against the present PTEs, scanning speed reflects the actual present (mapped) memory.
--- kernel/sched/fair.c | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-)
kernel/sched/fair.c +21 -16
@@ -914,8 +914,8 @@ void task_numa_work(struct callback_head *work)struct task_struct *p = current;struct mm_struct *mm = p->mm;struct vm_area_struct *vma; -unsigned long offset, end; -long length; +unsigned long start, end; +long pages; WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
@@ -942,30 +942,35 @@ void task_numa_work(struct callback_head *work)if (cmpxchg (&mm->numa_next_scan, migrate, next_scan) != migrate)return ; - offset = mm->numa_scan_offset; - length = sysctl_sched_numa_scan_size; - length <<=20 ; + start = mm->numa_scan_offset; + pages = sysctl_sched_numa_scan_size; + pages <<=20 -PAGE_SHIFT ;/* MB in pages */ +if (!pages) +return ;down_write (&mm->mmap_sem); - vma =find_vma (mm, offset); + vma =find_vma (mm, start);if (!vma) {ACCESS_ONCE (mm->numa_scan_seq)++; - offset =0 ; + start =0 ; vma = mm->mmap; } -for (; vma && length >0 ; vma = vma->vm_next) { +for (; vma; vma = vma->vm_next) {if (!vma_migratable (vma))continue ; - offset =max (offset, vma->vm_start); - end =min (ALIGN (offset + length,HPAGE_SIZE ), vma->vm_end); - length -= end - offset; - -change_prot_numa (vma, offset, end); - - offset = end; +do { + start =max (start, vma->vm_start); + end =ALIGN (start + (pages <<PAGE_SHIFT ),HPAGE_SIZE ); + end =min (end, vma->vm_end); + pages -=change_prot_numa (vma, start, end); + start = end; +if (pages <=0 ) + goto out; + }while (end != vma->vm_end); } - mm->numa_scan_offset = offset; +out: + mm->numa_scan_offset = start;up_write (&mm->mmap_sem); }
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 21/27 sched: Implement slow start for working set sampling
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
Add a 1 second delay before starting to scan the working set of a task and starting to balance it amongst nodes.
[ note that before the constant per task WSS sampling rate patch
the initial scan would happen much later still, in effect that patch caused this regression. ]
The theory is that short-run tasks benefit very little from NUMA
placement: they come and go, and they better stick to the node they were started on. As tasks mature and rebalance to other CPUs and nodes, so does their NUMA placement have to change and so does it start to matter more and more.
In practice this change fixes an observable kbuild regression:
# [ a perf stat --null --repeat 10 test of ten bzImage builds to /dev/shm ]
!NUMA:
45.291088843 seconds time elapsed ( +- 0.40% ) 45.154231752 seconds time elapsed ( +- 0.36% )
+NUMA, no slow start:
46.172308123 seconds time elapsed ( +- 0.30% ) 46.343168745 seconds time elapsed ( +- 0.25% )
+NUMA, 1 sec slow start:
45.224189155 seconds time elapsed ( +- 0.25% ) 45.160866532 seconds time elapsed ( +- 0.17% )
and it also fixes an observable perf bench (hackbench) regression:
# perf stat --null --repeat 10 perf bench sched messaging
-NUMA:
-NUMA: 0.246225691 seconds time elapsed ( +- 1.31% ) +NUMA no slow start: 0.252620063 seconds time elapsed ( +- 1.13% )
+NUMA 1sec delay: 0.248076230 seconds time elapsed ( +- 1.35% )
The implementation is simple and straightforward, most of the patch deals with adding the /proc/sys/kernel/sched_numa_scan_delay_ms tunable knob.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 2 +- kernel/sched/fair.c | 16 ++++++++++------ kernel/sysctl.c | 7 +++++++ 4 files changed, 19 insertions(+), 7 deletions(-)
include/linux/sched.h +1 -0
@@ -2045,6 +2045,7 @@ enum sched_tunable_scaling { };extern enum sched_tunable_scaling sysctl_sched_tunable_scaling; +extern unsigned int sysctl_sched_numa_scan_delay;extern unsigned int sysctl_sched_numa_scan_period_min;extern unsigned int sysctl_sched_numa_scan_period_max;extern unsigned int sysctl_sched_numa_scan_size;
kernel/sched/core.c +1 -1
@@ -1556,7 +1556,7 @@ static void __sched_fork(struct task_struct *p) p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq :0 ; p->numa_migrate_seq =2 ; p->numa_faults =NULL ; - p->numa_scan_period = sysctl_sched_numa_scan_period_min; + p->numa_scan_period = sysctl_sched_numa_scan_delay; p->numa_work.next = &p->numa_work;#endif /* CONFIG_NUMA_BALANCING */ }
kernel/sched/fair.c +10 -6
@@ -823,11 +823,12 @@ static void account_numa_dequeue(struct rq *rq, struct task_struct *p) }/* -* numa task sample period in ms: 5s +* Scan @scan_size MB every @scan_period after an initial @scan_delay. */ -unsigned int sysctl_sched_numa_scan_period_min =100 ; -unsigned int sysctl_sched_numa_scan_period_max =100 *16 ; -unsigned int sysctl_sched_numa_scan_size =256 ;/* MB */ +unsigned int sysctl_sched_numa_scan_delay =1000 ;/* ms */ +unsigned int sysctl_sched_numa_scan_period_min =100 ;/* ms */ +unsigned int sysctl_sched_numa_scan_period_max =100 *16 ;/* ms */ +unsigned int sysctl_sched_numa_scan_size =256 ;/* MB */ /* * Wait for the 2-sample stuff to settle before migrating again
@@ -938,10 +939,12 @@ void task_numa_work(struct callback_head *work)if (time_before (now, migrate))return ; - next_scan = now +2 *msecs_to_jiffies (sysctl_sched_numa_scan_period_min); + next_scan = now +msecs_to_jiffies (sysctl_sched_numa_scan_period_min);if (cmpxchg (&mm->numa_next_scan, migrate, next_scan) != migrate)return ; + current->numa_scan_period +=jiffies_to_msecs (2 ); + start = mm->numa_scan_offset; pages = sysctl_sched_numa_scan_size; pages <<=20 -PAGE_SHIFT ;/* MB in pages */
@@ -998,7 +1001,8 @@ void task_tick_numa(struct rq *rq, struct task_struct *curr) period = (u64 )curr->numa_scan_period *NSEC_PER_MSEC ;if (now - curr->node_stamp > period) { - curr->node_stamp = now; + curr->node_stamp += period; + curr->numa_scan_period = sysctl_sched_numa_scan_period_min;/* * We are comparing runtime to wall clock time here, which
kernel/sysctl.c +7 -0
@@ -353,6 +353,13 @@ static struct ctl_table kern_table[] = {#endif /* CONFIG_SMP */ #ifdef CONFIG_NUMA_BALANCING { + .procname = "sched_numa_scan_delay_ms", + .data = &sysctl_sched_numa_scan_delay, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec, + }, + { .procname = "sched_numa_scan_period_min_ms", .data = &sysctl_sched_numa_scan_period_min, .maxlen = sizeof(unsigned int),
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 22/27 sched, numa, mm: Interleave shared tasks
In reply to Ingo MolnarInterleave tasks that are 'shared' - i.e. whose memory access patterns indicate that they are intensively sharing memory with other tasks.
If such a task ends up converging then it switches back into the lazy node-local policy.
--- mm/mempolicy.c | 54 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 40 insertions(+), 14 deletions(-)
mm/mempolicy.c +40 -14
@@ -111,12 +111,30 @@ enum zone_type policy_zone = 0;/* * run-time system-wide default policy => local allocation */ -static struct mempolicy default_policy = { - .refcnt =ATOMIC_INIT (1 ),/* never free it */ - .mode =MPOL_PREFERRED , - .flags =MPOL_F_LOCAL , + +static struct mempolicy default_policy_local = { + .refcnt =ATOMIC_INIT (1 ),/* never free it */ + .mode =MPOL_PREFERRED , + .flags =MPOL_F_LOCAL , +}; + +/* +* .v.nodes is set by numa_policy_init(): +*/ +static struct mempolicy default_policy_shared = { + .refcnt =ATOMIC_INIT (1 ),/* never free it */ + .mode =MPOL_INTERLEAVE , + .flags =0 , }; +static struct mempolicy *default_policy (void ) +{ +if (task_numa_shared (current) ==1 ) +return &default_policy_shared; + +return &default_policy_local; +} + static const struct mempolicy_operations { int (*create)(struct mempolicy *pol, const nodemask_t *nodes); /*
@@ -789,7 +807,7 @@ out:static void get_policy_nodemask (struct mempolicy *p,nodemask_t *nodes) {nodes_clear (*nodes); -if (p == &default_policy) +if (p ==default_policy ())return ;switch (p->mode) {
@@ -864,7 +882,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,return -EINVAL ;if (!pol) - pol = &default_policy;/* indicates default behavior */ + pol =default_policy ();/* indicates default behavior */ if (flags &MPOL_F_NODE ) {if (flags &MPOL_F_ADDR ) {
@@ -880,7 +898,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, goto out; } }else { - *policy = pol == &default_policy ?MPOL_DEFAULT : + *policy = pol ==default_policy () ?MPOL_DEFAULT : pol->mode;/* * Internal mempolicy flags must be masked off before exposing
@@ -1568,7 +1586,7 @@ struct mempolicy *get_vma_policy(struct task_struct *task, } }if (!pol) - pol = &default_policy; + pol =default_policy ();return pol; }
@@ -1974,7 +1992,7 @@ struct page *alloc_pages_current(gfp_t gfp, unsigned order)unsigned int cpuset_mems_cookie;if (!pol ||in_interrupt () || (gfp & __GFP_THISNODE)) - pol = &default_policy; + pol =default_policy (); retry_cpuset: cpuset_mems_cookie =get_mems_allowed ();
@@ -2255,7 +2273,6 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned longint best_nid =-1 , page_nid;int cpu_last_access, this_cpu;struct mempolicy *pol; -unsigned long pgoff;struct zone *zone;BUG_ON (!vma);
@@ -2271,13 +2288,20 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned longswitch (pol->mode) {case MPOL_INTERLEAVE : + { +int shift; +BUG_ON (addr >= vma->vm_end);BUG_ON (addr < vma->vm_start); - pgoff = vma->vm_pgoff; - pgoff += (addr - vma->vm_start) >>PAGE_SHIFT ; - best_nid =offset_il_node (pol, vma, pgoff); +if (transparent_hugepage_enabled (vma) || vma->vm_flags &VM_HUGETLB ) + shift =HPAGE_SHIFT ; +else + shift =PAGE_SHIFT ; + + best_nid =interleave_nid (pol, vma, addr, shift);break ; + }case MPOL_PREFERRED :if (pol->flags &MPOL_F_LOCAL )
@@ -2492,6 +2516,8 @@ void __init numa_policy_init(void)sizeof (struct sp_node ), 0, SLAB_PANIC, NULL); + default_policy_shared.v.nodes = node_online_map; +/* * Set interleaving policy for system init. Interleaving is only * enabled across suitably sized nodes (default is >= 16MB), or
@@ -2712,7 +2738,7 @@ int mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol, int no_context)*/ VM_BUG_ON (maxlen <strlen ("interleave" ) +strlen ("relative" ) +16 ); -if (!pol || pol == &default_policy) +if (!pol || pol ==default_policy ()) mode =MPOL_DEFAULT ;else mode = pol->mode;
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 23/27 sched: Implement NUMA scanning backoff
In reply to Ingo MolnarBack off slowly from scanning, up to sysctl_sched_numa_scan_period_max (1.6 seconds). Scan faster again if we were forced to switch to another node.
This makes sure that workload in equilibrium don't get scanned as often as workloads that are still converging.
--- kernel/sched/core.c | 6 ++++++ kernel/sched/fair.c | 8 +++++++- 2 files changed, 13 insertions(+), 1 deletion(-)
kernel/sched/core.c +6 -0
@@ -6024,6 +6024,12 @@ void sched_setnuma(struct task_struct *p, int node, int shared)if (on_rq)enqueue_task (rq, p,0 );task_rq_unlock (rq, p, &flags); + +/* +* Reset the scanning period. If the task converges +* on this node then we'll back off again: +*/ + p->numa_scan_period = sysctl_sched_numa_scan_period_min; }#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/fair.c +7 -1
@@ -865,8 +865,10 @@ static void task_numa_placement(struct task_struct *p) } } -if (max_node != p->numa_max_node) +if (max_node != p->numa_max_node) {sched_setnuma (p, max_node,task_numa_shared (p)); + goto out_backoff; + } p->numa_migrate_seq++; if (sched_feat(NUMA_SETTLE) &&
@@ -882,7 +884,11 @@ static void task_numa_placement(struct task_struct *p)if (shared !=task_numa_shared (p)) {sched_setnuma (p, p->numa_max_node, shared); p->numa_migrate_seq =0 ; + goto out_backoff; } +return ; +out_backoff: + p->numa_scan_period =min (p->numa_scan_period *2 , sysctl_sched_numa_scan_period_max); }/*
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 24/27 sched: Improve convergence
In reply to Ingo Molnar - break out of can_do_numa_run() earlier if we can make no progress
- don't flip between siblings that often
- turn on bidirectional fault balancing
- improve the flow in task_numa_work()
--- kernel/sched/fair.c | 46 ++++++++++++++++++++++++++++++++-------------- kernel/sched/features.h | 2 +- 2 files changed, 33 insertions(+), 15 deletions(-)
kernel/sched/fair.c +32 -14
@@ -917,12 +917,12 @@ void task_numa_fault(int node, int last_cpu, int pages)*/ void task_numa_work (struct callback_head *work) { +long pages_total, pages_left, pages_changed;unsigned long migrate, next_scan, now = jiffies; +unsigned long start0, start, end;struct task_struct *p = current;struct mm_struct *mm = p->mm;struct vm_area_struct *vma; -unsigned long start, end; -long pages; WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
@@ -951,35 +951,42 @@ void task_numa_work(struct callback_head *work) current->numa_scan_period +=jiffies_to_msecs (2 ); - start = mm->numa_scan_offset; - pages = sysctl_sched_numa_scan_size; - pages <<=20 -PAGE_SHIFT ;/* MB in pages */ -if (!pages) + start0 = start = end = mm->numa_scan_offset; + pages_total = sysctl_sched_numa_scan_size; + pages_total <<=20 -PAGE_SHIFT ;/* MB in pages */ +if (!pages_total)return ; + pages_left = pages_total; +down_write (&mm->mmap_sem); vma =find_vma (mm, start);if (!vma) {ACCESS_ONCE (mm->numa_scan_seq)++; - start =0 ; - vma = mm->mmap; + end =0 ; + vma =find_vma (mm, end); }for (; vma; vma = vma->vm_next) {if (!vma_migratable (vma))continue ;do { - start =max (start, vma->vm_start); - end =ALIGN (start + (pages <<PAGE_SHIFT ),HPAGE_SIZE ); + start =max (end, vma->vm_start); + end =ALIGN (start + (pages_left <<PAGE_SHIFT ),HPAGE_SIZE ); end =min (end, vma->vm_end); - pages -=change_prot_numa (vma, start, end); - start = end; -if (pages <=0 ) + pages_changed =change_prot_numa (vma, start, end); + +WARN_ON_ONCE (pages_changed > pages_total); +BUG_ON (pages_changed <0 ); + + pages_left -= pages_changed; +if (pages_left <=0 ) goto out; }while (end != vma->vm_end); } out: - mm->numa_scan_offset = start; + mm->numa_scan_offset = end; +up_write (&mm->mmap_sem); }
@@ -3306,6 +3313,13 @@ static int select_idle_sibling(struct task_struct *p, int target)int i;/* +* For NUMA tasks constant, reliable placement is more important +* than flipping tasks between siblings: +*/ +if (task_numa_shared (p) >=0 ) +return target; + +/* * If the task is going to be woken-up on this cpu and if it is * already idle, then it is the right target. */
@@ -4581,6 +4595,10 @@ static bool can_do_numa_run(struct lb_env *env, struct sd_lb_stats *sds)* If we got capacity allow stacking up on shared tasks. */ if ((sds->this_shared_running < sds->this_group_capacity) && sds->numa_shared_running) { +/* There's no point in trying to move if all are here already: */ +if (sds->numa_shared_running == sds->this_shared_running) +return false; + env->flags |=LBF_NUMA_SHARED ;return true; }
kernel/sched/features.h +1 -1
@@ -71,6 +71,6 @@ SCHED_FEAT(LB_MIN, false)/* Do the working set probing faults: */ SCHED_FEAT(NUMA, true) SCHED_FEAT(NUMA_FAULTS_UP, true) -SCHED_FEAT(NUMA_FAULTS_DOWN, false) +SCHED_FEAT(NUMA_FAULTS_DOWN, true) SCHED_FEAT(NUMA_SETTLE, true)#endif
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 25/27 sched: Introduce staged average NUMA faults
In reply to Ingo MolnarThe current way of building the p->numa_faults[2][node] faults statistics has a sampling artifact:
The continuous and immediate nature of propagating new fault stats to the numa_faults array creates a 'pulsating' dynamic, that starts at the average value at the beginning of the scan, increases monotonically until we finish the scan to about twice the average, and then drops back to half of its value due to the running average.
Since we rely on these values to balance tasks, the pulsating nature resulted in false migrations and general noise in the stats.
To solve this, introduce buffering of the current scan via p->task_numa_faults_curr[]. The array is co-allocated with the p->task_numa[] for efficiency reasons, but it is otherwise an ordinary separate array.
At the end of the scan we propagate the latest stats into the average stats value. Most of the balancing code stays unmodified.
The cost of this change is that we delay the effects of the latest round of faults by 1 scan - but using the partial faults info was creating artifacts.
This instantly stabilized the page fault stats and improved numa02-alike workloads by making them faster to converge.
--- include/linux/sched.h | 1 + kernel/sched/fair.c | 20 +++++++++++++++++--- 2 files changed, 18 insertions(+), 3 deletions(-)
include/linux/sched.h +1 -0
@@ -1511,6 +1511,7 @@ struct task_struct {u64 node_stamp;/* migration stamp */ unsigned long numa_weight;unsigned long *numa_faults; +unsigned long *numa_faults_curr;struct callback_head numa_work;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/fair.c +17 -3
@@ -852,12 +852,26 @@ static void task_numa_placement(struct task_struct *p) p->numa_scan_seq = seq; +/* +* Update the fault average with the result of the latest +* scan: +*/ for (node =0 ; node < nr_node_ids; node++) { faults =0 ;for (priv =0 ; priv <2 ; priv++) { - faults += p->numa_faults[2 *node + priv]; - total[priv] += p->numa_faults[2 *node + priv]; - p->numa_faults[2 *node + priv] /=2 ; +unsigned int new_faults; +unsigned int idx; + + idx =2 *node + priv; + new_faults = p->numa_faults_curr[idx]; + p->numa_faults_curr[idx] =0 ; + +/* Keep a simple running average: */ + p->numa_faults[idx] += new_faults; + p->numa_faults[idx] /=2 ; + + faults += p->numa_faults[idx]; + total[priv] += p->numa_faults[idx]; }if (faults > max_faults) { max_faults = faults;
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 26/27 sched: Track groups of shared tasks
In reply to Ingo MolnarTo be able to cluster memory-related tasks more efficiently, introduce a new metric that tracks the 'best' buddy task
Track our "memory buddies": the tasks we actively share memory with.
Firstly we establish the identity of some other task that we are sharing memory with by looking at rq[page::last_cpu].curr - i.e. we check the task that is running on that CPU right now.
This is not entirely correct as this task might have scheduled or migrate ther - but statistically there will be correlation to the tasks that we share memory with, and correlation is all we need.
We map out the relation itself by filtering out the highest address ask that is below our own task address, per working set scan iteration.
This creates a natural ordering relation between groups of tasks:
t1 < t2 < t3 < t4
t1->memory_buddy == NULL
t2->memory_buddy == t1
t3->memory_buddy == t2
t4->memory_buddy == t3The load-balancer can then use this information to speed up NUMA convergence, by moving such tasks together if capacity and load constraints allow it.
(This is all statistical so there are no preemption or locking worries.)
--- include/linux/sched.h | 5 ++ kernel/sched/core.c | 5 ++ kernel/sched/fair.c | 144 ++++++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 151 insertions(+), 3 deletions(-)
include/linux/sched.h +5 -0
@@ -1513,6 +1513,11 @@ struct task_struct {unsigned long *numa_faults;unsigned long *numa_faults_curr;struct callback_head numa_work; + +struct task_struct *shared_buddy, *shared_buddy_curr; +unsigned long shared_buddy_faults, shared_buddy_faults_curr; +int ideal_cpu, ideal_cpu_curr; +#endif /* CONFIG_NUMA_BALANCING */ struct rcu_head rcu;
kernel/sched/core.c +5 -0
@@ -1558,6 +1558,11 @@ static void __sched_fork(struct task_struct *p) p->numa_faults =NULL ; p->numa_scan_period = sysctl_sched_numa_scan_delay; p->numa_work.next = &p->numa_work; + + p->shared_buddy =NULL ; + p->shared_buddy_faults =0 ; + p->ideal_cpu =-1 ; + p->ideal_cpu_curr =-1 ;#endif /* CONFIG_NUMA_BALANCING */ }
kernel/sched/fair.c +141 -3
176 lines not on this page · raw message
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 27/27 sched: Use the best-buddy 'ideal cpu' in balancing decisions
In reply to Ingo MolnarNow that we have a notion of (one of the) best CPUs we interrelate with in terms of memory usage, use that information to improve can_migrate_task() balancing decisions: allow the migration to occur even if we locally cache-hot, if we are on another node and want to migrate towards our best buddy's node.
( Note that this is not hard affinity - if imbalance persists long
enough then the scheduler will eventually balance tasks anyway, to maximize CPU utilization. )
--- kernel/sched/fair.c | 35 ++++++++++++++++++++++++++++++++--- kernel/sched/features.h | 2 ++ 2 files changed, 34 insertions(+), 3 deletions(-)
kernel/sched/fair.c +32 -3
@@ -840,6 +840,14 @@ static void task_numa_migrate(struct task_struct *p, int next_cpu) p->numa_migrate_seq =0 ; } +static int task_ideal_cpu (struct task_struct *p) +{ +if (!sched_feat (IDEAL_CPU )) +return -1 ; + +return p->ideal_cpu; +} +/* * Called for every full scan - here we consider switching to a new * shared buddy, if the one we found during this scan is good enough:
@@ -1028,7 +1036,7 @@ out_hit:* but don't stop the discovery of process level sharing * either: */ -if (this_task->mm == last_task->mm) +if (sched_feat (IDEAL_CPU_THREAD_BIAS ) && this_task->mm == last_task->mm) pages *=2 ; this_task->shared_buddy_faults_curr += pages;
@@ -1189,6 +1197,7 @@ void task_tick_numa(struct rq *rq, struct task_struct *curr) }#else /* !CONFIG_NUMA_BALANCING: */ #ifdef CONFIG_SMP +static inline int task_ideal_cpu (struct task_struct *p) {return -1 ; }static inline void account_numa_enqueue (struct rq *rq,struct task_struct *p) { }#endif static inline void account_numa_dequeue (struct rq *rq,struct task_struct *p) { }
@@ -4064,6 +4073,7 @@ struct lb_env {static void move_task (struct task_struct *p,struct lb_env *env) {deactivate_task (env->src_rq, p,0 ); +set_task_cpu (p, env->dst_cpu);activate_task (env->dst_rq, p,0 );check_preempt_curr (env->dst_rq, p,0 );
@@ -4242,15 +4252,17 @@ static bool can_migrate_numa_task(struct task_struct *p, struct lb_env *env)* * LBF_NUMA_RUN -- numa only, only allow improvement * LBF_NUMA_SHARED -- shared only +* LBF_NUMA_IDEAL -- ideal only * * LBF_KEEP_SHARED -- do not touch shared tasks */ /* a numa run can only move numa tasks about to improve things */ if (env->flags &LBF_NUMA_RUN ) { -if (task_numa_shared (p) <0 ) +if (task_numa_shared (p) <0 &&task_ideal_cpu (p) <0 )return false; -/* can only pull shared tasks */ + +/* If we are only allowed to pull shared tasks: */ if ((env->flags &LBF_NUMA_SHARED ) && !task_numa_shared (p))return false; }else {
@@ -4307,6 +4319,23 @@ static int can_migrate_task(struct task_struct *p, struct lb_env *env)if (!can_migrate_running_task (p, env))return false; +#ifdef CONFIG_NUMA_BALANCING +/* If we are only allowed to pull ideal tasks: */ +if ((task_ideal_cpu (p) >=0 ) && (p->shared_buddy_faults >1000 )) { +int ideal_node; +int dst_node; + +BUG_ON (env->dst_cpu <0 ); + + ideal_node =cpu_to_node (p->ideal_cpu); + dst_node =cpu_to_node (env->dst_cpu); + +if (ideal_node == dst_node) +return true; +return false; + } +#endif +if (env->sd->flags &SD_NUMA )return can_migrate_numa_task (p, env);
kernel/sched/features.h +2 -0
@@ -66,6 +66,8 @@ SCHED_FEAT(TTWU_QUEUE, true) SCHED_FEAT(FORCE_SD_OVERLAP, false) SCHED_FEAT(RT_RUNTIME_SHARE, true) SCHED_FEAT(LB_MIN, false) +SCHED_FEAT(IDEAL_CPU, true) +SCHED_FEAT(IDEAL_CPU_THREAD_BIAS, false)#ifdef CONFIG_NUMA_BALANCING /* Do the working set probing faults: */
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 18 lines · Ingo Molnar
I'm pleased to announce the latest version of the numa/core tree.
Here are some quick, preliminary performance numbers on a 4-node, 32-way, 64 GB RAM system:
CONFIG_NUMA_BALANCING=y
-----------------------------------------------------------------------[ seconds ] v3.7 AutoNUMA | numa/core-v16 [ vs. v3.7] [ lower is better ] ----- -------- | ------------- ----------- | numa01 340.3 192.3 | 139.4 +144.1% numa01_THREAD_ALLOC 425.1 135.1 | 121.1 +251.0% numa02 56.1 25.3 | 17.5 +220.5% | [ SPECjbb transactions/sec ] | [ higher is better ] | | SPECjbb single-1x32 524k 507k | 638k +21.7%
Quoted 2 lines
-----------------------------------------------------------------------
I was not able to run a full sets of tests today as I was distracted so all I have is a multi JVM comparison. I'll keep it shorter than average
3.7.0 3.7.0
rc5-stats-v4r2 rc5-schednuma-v16r1
TPut 1 101903.00 ( 0.00%) 77651.00 (-23.80%)
TPut 2 213825.00 ( 0.00%) 160285.00 (-25.04%)
TPut 3 307905.00 ( 0.00%) 237472.00 (-22.87%)
TPut 4 397046.00 ( 0.00%) 302814.00 (-23.73%)
TPut 5 477557.00 ( 0.00%) 364281.00 (-23.72%)
TPut 6 542973.00 ( 0.00%) 420810.00 (-22.50%)
TPut 7 540466.00 ( 0.00%) 448976.00 (-16.93%)
TPut 8 543226.00 ( 0.00%) 463568.00 (-14.66%)
TPut 9 513351.00 ( 0.00%) 468238.00 ( -8.79%)
TPut 10 484126.00 ( 0.00%) 457018.00 ( -5.60%)
TPut 11 467440.00 ( 0.00%) 457999.00 ( -2.02%)
TPut 12 430423.00 ( 0.00%) 447928.00 ( 4.07%)
TPut 13 445803.00 ( 0.00%) 434823.00 ( -2.46%)
TPut 14 427388.00 ( 0.00%) 430667.00 ( 0.77%)
TPut 15 437183.00 ( 0.00%) 423746.00 ( -3.07%)
TPut 16 423245.00 ( 0.00%) 416259.00 ( -1.65%)
TPut 17 417666.00 ( 0.00%) 407186.00 ( -2.51%)
TPut 18 413046.00 ( 0.00%) 398197.00 ( -3.59%)This version of the patches manages to cripple performance entirely. I do not have a single JVM comparison available as the machine has been in use during the day. I accept that it is very possible that the single JVM figures are better.
SPECJBB PEAKS
3.7.0 3.7.0
rc5-stats-v4r2 rc5-schednuma-v16r1
Expctd Warehouse 12.00 ( 0.00%) 12.00 ( 0.00%)
Expctd Peak Bops 430423.00 ( 0.00%) 447928.00 ( 4.07%)
Actual Warehouse 8.00 ( 0.00%) 9.00 ( 12.50%)
Actual Peak Bops 543226.00 ( 0.00%) 468238.00 (-13.80%)Peaks at a higher number of warehouses but actual peak throughput is hurt badly.
MMTests Statistics: duration
3.7.0 3.7.0
rc5-stats-v4r2rc5-schednuma-v16r1User 203918.23 176061.11
System 141.06 24846.23
Elapsed 4979.65 4964.45
System CPU usage.
MMTests Statistics: vmstat
No meaningful stats are available with your series.
Quoted 4 lines
On my NUMA system the numa/core tree now significantly outperforms both the vanilla kernel and the AutoNUMA (v28) kernel, in these benchmarks. No NUMA balancing kernel has ever performed so well on this system.
Maybe on your machine and maybe on your specjbb configuration it works well. But for my machine and for a multi JVM configuration, it hurts quite badly.
Quoted 11 lines
It is notable that workloads where 'private' processing dominates (
numa01_THREAD_ALLOCand numa02) are now very close to bare metal hard binding performance.These are the main changes in this release:
- There are countless performance improvements. The new shared/private distinction metric we introduced in v15 is now further refined and is used in more places within the scheduler to converge in a better and more directed fashion.
Great.
Quoted 6 lines
- I restructured the whole tree to make it cleaner, to simplify its
mm/impact and in general to make it more mergable. It now includes either agreed-upon patches, or bare essentials that are needed to make theCONFIG_NUMA_BALANCING=yfeature work. It is fully bisect tested - it builds and works at every point.
It is a misrepresentation to say that all these patches have been agreed upon.
You are still using MIGRATE_FAULT which has not been agreed upon at all.
While you have renamed change_prot_none to change_prot_numa, it still effectively hard-codes PROT_NONE. Even if an architecture redefines pte_numa to use a bit other than _PAGE_PROTNONE it'll still not work because change_protection() will not recognise it.
I still maintain that THP native migration was introduced too early now it's worse because you've collapsed it with another patch. The risk is that you might be depending on THP migration to reduce overhead for the autonumabench test cases. I've said already that I believe that the correct thing to do here is to handle regular PMDs in batch where possible and add THP native migration as an optimisation on top. This avoids us accidentally depending on THP to reduce system CPU usage.
While the series may be bisectable, it still is an all-or-nothing approach. Consider "sched, numa, mm: Add the scanning page fault machinery" for example. Despite its name it is very much orientated around schednuma and adds the fields schednuma requires. This is a small example. The bigger example continues to be "sched: Add adaptive NUMA affinity support" which is a monolithic patch that introduces .... basically everything. It would be extremely difficult to retrofit an alternative policy on top of this and to make an apples-to-apples comparison. My whole point about bisection was that we would have three major points that could be bisected
1. vanilla kernel
2. one set of optimisations, basic stats
3. basic placement policy
4. more optimisations if necessary
5. complex placement policy
The complex placement policy it would either be schednuma, autonuma or some combination and it could be evaluated in terms of a basic placement policy -- how much better does it perform? How much system overhead does it add? Otherwise it's too easy to fall into a trap where a complex placement policy for the tested workloads hides all the cost of the underlying machinery and then falls apart when tested by a larger number of users. If/when the placement policy fails the system gets majorly bogged down and it'll not be possible to break up the series in any meaningful way to see where the problem was introduced. I'm running out of creative ways to repeat myself on this.
Quoted 8 lines
- The hard-coded "
PROT_NONE" feature that reviewers complained about is now factored out and selectable on a per architecture basis. (the arch porting aspect of this is untested, but the basic fabric is there and should be pretty close to what we need.)The generic
PROT_NONEbased facility can be used by architectures to prototype this feature quickly.
They'll also need to alter change_protection() or reimplement it. It's still effectively hard-coded although it's getting better in this regard.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 22 lines · Mel Gorman
Quoted 18 lines · Ingo Molnar
I'm pleased to announce the latest version of the numa/core tree.
Here are some quick, preliminary performance numbers on a 4-node, 32-way, 64 GB RAM system:
CONFIG_NUMA_BALANCING=y
-----------------------------------------------------------------------[ seconds ] v3.7 AutoNUMA | numa/core-v16 [ vs. v3.7] [ lower is better ] ----- -------- | ------------- ----------- | numa01 340.3 192.3 | 139.4 +144.1% numa01_THREAD_ALLOC 425.1 135.1 | 121.1 +251.0% numa02 56.1 25.3 | 17.5 +220.5% | [ SPECjbb transactions/sec ] | [ higher is better ] | | SPECjbb single-1x32 524k 507k | 638k +21.7%Quoted 2 lines
-----------------------------------------------------------------------
Quoted 6 lines
I was not able to run a full sets of tests today as I was distracted so all I have is a multi JVM comparison. I'll keep it shorter than average
3.7.0 3.7.0 rc5-stats-v4r2 rc5-schednuma-v16r1
Thanks for the testing - I'll wait for your full results to see whether the other regressions you reported before are fixed/improved.
Exactly what tree/commit does "rc5-schednuma-v16r1" mean?
I am starting to have doubts about your testing methods. There does seem to be some big disconnect between your testing and mine - and I think we should clear that up by specifying exactly *what* you have tested. Did you rebase my tree in any fashion?
You can find what I tested in tip:master and I'd encourage you to test that too.
Other people within Red Hat have tested these same workloads as well, on similarly sided (and even larger) systems as yours, and they have already reported to me (much) improved numbers, including improvements in the multi-JVM SPECjbb load that you are concentrating on ...
Quoted 9 lines
Quoted 6 lines
- I restructured the whole tree to make it cleaner, to simplify its
mm/impact and in general to make it more mergable. It now includes either agreed-upon patches, or bare essentials that are needed to make theCONFIG_NUMA_BALANCING=yfeature work. It is fully bisect tested - it builds and works at every point.It is a misrepresentation to say that all these patches have been agreed upon.
That has not been claimed, please read the sentence above.
Quoted 2 lines
You are still using
MIGRATE_FAULTwhich has not been agreed upon at all.
See my followup patch yesterday.
Quoted 5 lines
While you have renamed
change_prot_nonetochange_prot_numa, it still effectively hard-codesPROT_NONE. Even if an architecture redefinespte_numato use a bit other than_PAGE_PROTNONEit'll still not work becausechange_protection()will not recognise it.
This is not how it works.
The new generic PROT_NONE scheme is that an architecture that wants to reuse the generic PROT_NONE code can define:
select ARCH_SUPPORTS_NUMA_BALANCING
and can set:
select ARCH_WANTS_NUMA_GENERIC_PGPROT
and it will get the generic code very easily. This is what x86 uses now. No architecture changes needed beyond these two lines of Kconfig enablement.
If an architecture wants to provide its own open-coded, optimizd variant, it can do so by not defining ARCH_SUPPORTS_NUMA_BALANCING, and by offering the following methods:
bool pte_numa(struct vm_area_struct *vma, pte_t pte);
pte_t pte_mknuma(struct vm_area_struct *vma, pte_t pte);
bool pmd_numa(struct vm_area_struct *vma, pmd_t pmd);
pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)
unsigned long change_prot_numa(struct vm_area_struct *vma, unsigned long start, unsigned longI have not tested the arch-defined portion but barring trivial problems it should work. We can extend the list of methods if you think more is needed, and we can offer library functions for architectures that want to share some but not all generic code - I'm personally happy with x86 using change_protection().
I think this got bikeshed painted enough already.
Quoted 8 lines
I still maintain that THP native migration was introduced too early now it's worse because you've collapsed it with another patch. The risk is that you might be depending on THP migration to reduce overhead for the autonumabench test cases. I've said already that I believe that the correct thing to do here is to handle regular PMDs in batch where possible and add THP native migration as an optimisation on top. This avoids us accidentally depending on THP to reduce system CPU usage.
Have you disabled THP in your testing of numa/core???
I think we need to stop the discussion now and clear up exactly *what* you have tested. Commit ID and an exact description of testing methodology please ...
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 29 lines · Ingo Molnar
Quoted 26 lines · Mel Gorman
Quoted 18 lines · Ingo Molnar
I'm pleased to announce the latest version of the numa/core tree.
Here are some quick, preliminary performance numbers on a 4-node, 32-way, 64 GB RAM system:
CONFIG_NUMA_BALANCING=y
-----------------------------------------------------------------------[ seconds ] v3.7 AutoNUMA | numa/core-v16 [ vs. v3.7] [ lower is better ] ----- -------- | ------------- ----------- | numa01 340.3 192.3 | 139.4 +144.1% numa01_THREAD_ALLOC 425.1 135.1 | 121.1 +251.0% numa02 56.1 25.3 | 17.5 +220.5% | [ SPECjbb transactions/sec ] | [ higher is better ] | | SPECjbb single-1x32 524k 507k | 638k +21.7%Quoted 2 lines
-----------------------------------------------------------------------
I was not able to run a full sets of tests today as I was distracted so all I have is a multi JVM comparison. I'll keep it shorter than average
Quoted 7 lines
Quoted 2 lines
3.7.0 3.7.0 rc5-stats-v4r2 rc5-schednuma-v16r1Thanks for the testing - I'll wait for your full results to see whether the other regressions you reported before are fixed/improved.
Ok.
In response to one of your later questions, I found that I had in fact disabled THP without properly reporting it. I had configured MMTests to run specjbb with "base" pages which is the default taken from the specjvm configuration file. I had noted in different reports that no THP was used and that THP was not a factor for those tests but had not properly checked why that was.
However, this also means that *all* the tests disabled THP. Vanilla kernel, balancenuma and autonuma all ran with THP disabled. The comparisons are still valid but failing to report the lack of THP was a major mistake.
Quoted 2 lines
Exactly what tree/commit does "rc5-schednuma-v16r1" mean?
Kernel 3.7-rc5
tip/sched/core from when it was last pulled
your patches as posted in this thread
JVM is Oracle JVM
java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b10) Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
I had been thinking that differences in the JVMs might be a factor but it almost certainly because I had configured mmtests to disable THP for the specjbb test. Again, I'm sorry that this was not properly identified and reported by me but will reiterate that all tests were run identically.
Quoted 4 lines
I am starting to have doubts about your testing methods. There does seem to be some big disconnect between your testing and mine - and I think we should clear that up by specifying exactly *what* you have tested.
And the difference was in test methods - specifically I disabled THP.
FWIW, what I'm testing is implemented via mmtests 0.07 which I released a few days ago. I use the same configuration file each time and the kernel .config only differs in the name of the numa balancing option it needs to enable. The THP disabling was a screw-up but it was the same screw up every time.
Quoted 2 lines
Did you rebase my tree in any fashion?
No.
However, what I will be running next will be rebased on top of rc6. I am not expecting that to make any difference.
Quoted 3 lines
You can find what I tested in tip:master and I'd encourage you to test that too.
I will.
Quoted 6 lines
Other people within Red Hat have tested these same workloads as well, on similarly sided (and even larger) systems as yours, and they have already reported to me (much) improved numbers, including improvements in the multi-JVM SPECjbb load that you are concentrating on ...
It almost certainly came down to THP and not the JVM. Can you or one other other testers try with THP disabled and see what they find?
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
- I restructured the whole tree to make it cleaner, to simplify its
mm/impact and in general to make it more mergable. It now includes either agreed-upon patches, or bare essentials that are needed to make theCONFIG_NUMA_BALANCING=yfeature work. It is fully bisect tested - it builds and works at every point.
Quoted 5 lines
Quoted 2 lines
It is a misrepresentation to say that all these patches have been agreed upon.
That has not been claimed, please read the sentence above.
Ok yes, it is one or the other.
Quoted 5 lines
Quoted 2 lines
You are still using
MIGRATE_FAULTwhich has not been agreed upon at all.See my followup patch yesterday.
I missed the follow-up patch. I was looking at the series as-posted.
Quoted 26 lines
Quoted 5 lines
While you have renamed
change_prot_nonetochange_prot_numa, it still effectively hard-codesPROT_NONE. Even if an architecture redefinespte_numato use a bit other than_PAGE_PROTNONEit'll still not work becausechange_protection()will not recognise it.This is not how it works.
The new generic
PROT_NONEscheme is that an architecture that wants to reuse the genericPROT_NONEcode can define:select
ARCH_SUPPORTS_NUMA_BALANCINGand can set:
select
ARCH_WANTS_NUMA_GENERIC_PGPROTand it will get the generic code very easily. This is what x86 uses now. No architecture changes needed beyond these two lines of
Kconfigenablement.If an architecture wants to provide its own open-coded, optimizd variant, it can do so by not defining
ARCH_SUPPORTS_NUMA_BALANCING, and by offering the following methods:
Quoted 6 lines
bool pte_numa(struct vm_area_struct *vma, pte_t pte);
pte_t pte_mknuma(struct vm_area_struct *vma, pte_t pte);
bool pmd_numa(struct vm_area_struct *vma, pmd_t pmd);
pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)
unsigned long change_prot_numa(struct vm_area_struct *vma, unsigned long start, unsigned longAh ok. The arch would indeed need change_prot_numa().
Quoted 6 lines
I have not tested the arch-defined portion but barring trivial problems it should work. We can extend the list of methods if you think more is needed, and we can offer library functions for architectures that want to share some but not all generic code - I'm personally happy with x86 using
change_protection().
FWIW, I don't think the list needs to be extended. I'm still disappointed that the pte_numa and friend implementations are relatively heavy and require a function call to vm_get_page_prot(). I'm also disappointed that regularly PMDs cannot be easily handled in batch without further modification.
On the batch PMD thing, I'm in the process of converting to using change_protection. Preserving all the logic from the old change_prot_numa() is a bit of a mess right now but so far I've found that at least two major factors are;
1. If page_mapcount > 1 pages are marked pte_numa then the overhead increases a lot. This is partially due to the placement policy being really dumb and unable to handle shared pages.
2. With a partial series (I don't have test results with a full series yet) I find that if I do not set pmd_numa and batch handle the faults that it takes 5 times longer to complete the numa01 test. This is even slower than the vanilla kernel and is probably fault overhead.
In both cases the overhead could be overcome by having a smart placement policy and then scanning backoff logic to reduce the number of faults incurred. If for some reason the scanning rate cannot be reduced because the placement policy bounces pages around nodes en the CPU usage will go through the roof setting all the PTEs and handling the faults.
Quoted 11 lines
I think this got bikeshed painted enough already.
Quoted 8 lines
I still maintain that THP native migration was introduced too early now it's worse because you've collapsed it with another patch. The risk is that you might be depending on THP migration to reduce overhead for the autonumabench test cases. I've said already that I believe that the correct thing to do here is to handle regular PMDs in batch where possible and add THP native migration as an optimisation on top. This avoids us accidentally depending on THP to reduce system CPU usage.
Quoted 2 lines
Have you disabled THP in your testing of numa/core???
As it turned out, yes, I did disable THP. The vmstats I posted in other reports all showed that no THPs were used because of this.
If your internal testers run with THP disabled, it would be interesting to know if they see similar regressions. If they do, it implies that the performance of schednuma depends on THP. If they do not, something else is also making a big difference.
The temptation is to just ignore the THP problem but if workloads cannot use THP or the system gets into a state where it cannot allocate THP then performance will be badly hurt. It'll be hard to spot that the lack of THP is what is causing the regression.
Quoted 4 lines
I think we need to stop the discussion now and clear up exactly *what* you have tested. Commit ID and an exact description of testing methodology please ...
Kernel and JVM used have already been mentioned. Testing methodology is as implemented in mmtests 0.07. The configuration file for multi JVMs is configs/config-global-dhp__jvm-specjbb and sets SPECJBB_PAGESIZES="base" which is what disabled THP.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines · Mel Gorman
Ok.
In response to one of your later questions, I found that I had in fact disabled THP without properly reporting it. [...]
Hugepages is a must for most forms of NUMA/HPC. This alone questions the relevance of most of your prior numa/core testing results. I now have to strongly dispute your other conclusions as well.
Just a look at 'perf top' output should have told you the story.
Yet time and time again you readily reported bad 'schednuma' results for a slow 4K memory model that neither we nor other NUMA testers I talked to actually used, without stopping to look why that was so...
[ I suspect that if such terabytes-of-data workloads are forced
through such a slow 4K pages model then there's a bug or mis-tuning in our code that explains the level of additional slowdown you saw - we'll fix that.
But you should know that behavior under the slow 4K model tells very little about the true scheduling and placement quality of the patches... ]
Please report proper THP-enabled numbers before continuing.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 9 lines · Ingo Molnar
Quoted 4 lines · Mel Gorman
Ok.
In response to one of your later questions, I found that I had in fact disabled THP without properly reporting it. [...]
Hugepages is a must for most forms of NUMA/HPC.
Requiring huge pages to avoid a regression is a mistake.
Quoted 5 lines
This alone
questions the relevance of most of your prior numa/core testing results. I now have to strongly dispute your other conclusions as well.
I'll freely admit that disabling THP for specjbb was a mistake and I should have caught why at the start. However, the autonumabench figures reported for the last release had THP enabled as had the kernel build benchmark figures.
Quoted 2 lines
Just a look at 'perf top' output should have told you the story.
I knew THP were not in use and said so in earlier reports. Take this for example -- https://lkml.org/lkml/2012/11/16/207 . For specjbb, note that the THP fault alloc figures are close to 0 and due to that I said "THP is not really a factor for this workload". What I failed to do was identify why THP was not in use.
Quoted 5 lines
Yet time and time again you readily reported bad 'schednuma' results for a slow 4K memory model that neither we nor other NUMA testers I talked to actually used, without stopping to look why that was so...
Again, I apologise for the THP mistake. The fact remains that the other implementations did not suffer a performance slowdown due to the same mistake.
Quoted 11 lines
[ I suspect that if such terabytes-of-data workloads are forced
through such a slow 4K pages model then there's a bug or mis-tuning in our code that explains the level of additional slowdown you saw - we'll fix that.But you should know that behavior under the slow 4K model tells very little about the true scheduling and placement quality of the patches... ]
Please report proper THP-enabled numbers before continuing.
Will do. Are THP-disabled benchmark results to be ignored?
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines · Mel Gorman
Quoted 3 lines · Ingo Molnar
* Mel Gorman <[email protected]> wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Ok.
In response to one of your later questions, I found that I had in fact disabled THP without properly reporting it. [...]
Hugepages is a must for most forms of NUMA/HPC.
Requiring huge pages to avoid a regression is a mistake.
Not all architectures support THP. Not all workloads will end up using THP effectively.
Mel, would you have numa/core profiles from !THP runs, so we can find out the cause of the regression?
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 12 lines · Rik van Riel
Quoted 11 lines · Mel Gorman
Quoted 9 lines · Ingo Molnar
Quoted 4 lines · Mel Gorman
Ok.
In response to one of your later questions, I found that I had in fact disabled THP without properly reporting it. [...]
Hugepages is a must for most forms of NUMA/HPC.
Quoted 8 lines
Quoted 1 line
Requiring huge pages to avoid a regression is a mistake.
Not all architectures support THP. Not all workloads will end up using THP effectively.
Mel, would you have numa/core profiles from !THP runs, so we can find out the cause of the regression?
Unfortunately not. I'll queue up a profile run again when I can but as the !profile runs are still going it could take a while. It might be the weekend before they even start.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Ingo Molnar
Hugepages is a must for most forms of NUMA/HPC. This alone questions the relevance of most of your prior numa/core testing results. I now have to strongly dispute your other conclusions as well.
Ingo, stop doing this kind of crap.
Let's make it clear: if the NUMA patches continue to regress performance for reasonable loads (and that very much includes "no THP") then they won't be merged.
You seem to be in total denial. Every time Mel sends out results that show that your patches MAKE PERFORMANCE WORSE you blame Mel, or blame the load, and never seem to admit that performance got worse.
Stop it. That kind of "head-in-the-sand" behavior is not conducive to good code, and I have absolutely *zero* interest in merging a branch that has been tested with only your load on only your machine, and performs better on that *one* load, and then regresses on other loads.
Seriously. If you can't make the non-THP case go faster, don't even bother sending out the patches.
Similarly, if you cannot take performance results from others, don't even bother sending out the patches. If all you care about is your own special case, then keep the patches on your own machine, and stop bothering others with your patches.
So stop ignoring the feedback, and stop shooting the messenger. Look at the numbers, and admit that there is something that needs to be fixed.
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Ingo Molnar
Hugepages is a must for most forms of NUMA/HPC. This alone questions the relevance of most of your prior numa/core testing results. I now have to strongly dispute your other conclusions as well.
Ingo, stop doing this kind of crap.
Quoted 8 lines
Let's make it clear: if the NUMA patches continue to regress performance for reasonable loads (and that very much includes "no THP") then they won't be merged.
You seem to be in total denial. Every time Mel sends out results that show that your patches MAKE PERFORMANCE WORSE you blame Mel, or blame the load, and never seem to admit that performance got worse.
No doubt numa/core should not regress with THP off or on and I'll fix that.
As a background, here's how SPECjbb gets slower on mainline (v3.7-rc6) if you boot Mel's kernel config and turn THP forcibly off:
(avg: 502395 ops/sec)
(avg: 505902 ops/sec)
(avg: 509271 ops/sec)
# echo never > /sys/kernel/mm/transparent_hugepage/enabled
(avg: 376989 ops/sec)
(avg: 379463 ops/sec)
(avg: 378131 ops/sec)
A ~30% slowdown.
[ How do I know? I asked for Mel's kernel config days ago and
actually booted Mel's very config in the past few days, spending hours on testing it on 4 separate NUMA systems, trying to find Mel's regression. In the past Mel was a reliable tester so I blindly trusted his results. Was that some weird sort of denial on my part? :-) ]
Every time a regression is reported I take it seriously - and there were performance regression reports against numa/core not just from Mel and I'm sure there will be more in the future. For example I'm taking David Rijentje's fresh performance regression report seriously as well.
What I have some problem with is Mel sending me his kernel config as the thing he tested, and which config included:
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
but he apparently went and explicitly disabled THP on top of that - which was just a weird choice of 'negative test tuning' to keep unreported. That made me waste quite some time booting and debugging his config and made the finding of the root cause of the testing difference unnecessarily hard for me.
Again, that's not an excuse for the performance regression in the numa/core tree in any way and I'll fix it.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 17 lines · Ingo Molnar
No doubt numa/core should not regress with THP off or on and I'll fix that.
As a background, here's how SPECjbb gets slower on mainline (v3.7-rc6) if you boot Mel's kernel config and turn THP forcibly off:
(avg: 502395 ops/sec)
(avg: 505902 ops/sec)
(avg: 509271 ops/sec)# echo never > /sys/kernel/mm/transparent_hugepage/enabled(avg: 376989 ops/sec)
(avg: 379463 ops/sec)
(avg: 378131 ops/sec)
Quoted 9 lines
A ~30% slowdown.
[ How do I know? I asked for Mel's kernel config days ago and
actually booted Mel's very config in the past few days, spending hours on testing it on 4 separate NUMA systems, trying to find Mel's regression. In the past Mel was a reliable tester so I blindly trusted his results. Was that some weird sort of denial on my part? :-) ]
I confirm that numa/core regresses significantly more without thp than the 6.3% regression I reported with thp in terms of throughput on the same system. numa/core at 01aa90068b12 ("sched: Use the best-buddy 'ideal cpu' in balancing decisions") had 99389.49 SPECjbb2005 bops whereas ec05a2311c35 ("Merge branch 'sched/urgent' into sched/core") had 122246.90 SPECjbb2005 bops, a 23.0% regression.
perf top -U for >=0.70% at 01aa90068b12 ("sched: Use the best-buddy 'ideal cpu' in balancing decisions"):
16.34% [kernel] [k] page_fault
12.15% [kernel] [k] down_read_trylock
9.21% [kernel] [k] up_read
7.58% [kernel] [k] handle_pte_fault
6.10% [kernel] [k] handle_mm_fault
4.35% [kernel] [k] retint_swapgs
3.99% [kernel] [k] find_vma
3.95% [kernel] [k] __do_page_fault
3.81% [kernel] [k] mpol_misplaced
3.41% [kernel] [k] get_vma_policy
2.68% [kernel] [k] task_numa_fault
1.82% [kernel] [k] pte_numa
1.65% [kernel] [k] do_page_fault
1.46% [kernel] [k] _raw_spin_lock
1.28% [kernel] [k] do_wp_page
1.26% [kernel] [k] vm_normal_page
1.25% [kernel] [k] unlock_page
1.01% [kernel] [k] change_protection
0.80% [kernel] [k] getnstimeofday
0.79% [kernel] [k] ktime_get
0.76% [kernel] [k] __wake_up_bit
0.74% [kernel] [k] rcu_check_callbacks and at ec05a2311c35 ("Merge branch 'sched/urgent' into sched/core"):
22.01% [kernel] [k] page_fault
6.54% [kernel] [k] rcu_check_callbacks
5.04% [kernel] [k] getnstimeofday
4.12% [kernel] [k] ktime_get
3.55% [kernel] [k] read_tsc
3.37% [kernel] [k] task_tick_fair
2.61% [kernel] [k] emulate_vsyscall
2.22% [kernel] [k] __do_page_fault
1.78% [kernel] [k] run_timer_softirq
1.71% [kernel] [k] write_ok_or_segv
1.55% [kernel] [k] copy_user_generic_string
1.48% [kernel] [k] __bad_area_nosemaphore
1.27% [kernel] [k] retint_swapgs
1.26% [kernel] [k] spurious_fault
1.15% [kernel] [k] update_rq_clock
1.12% [kernel] [k] update_cfs_shares
1.09% [kernel] [k] _raw_spin_lock
1.08% [kernel] [k] update_curr
1.07% [kernel] [k] error_entry
1.05% [kernel] [k] x86_pmu_disable_all
0.88% [kernel] [k] sys_gettimeofday
0.88% [kernel] [k] __do_softirq
0.87% [kernel] [k] _raw_spin_lock_irq
0.84% [kernel] [k] hrtimer_forward
0.81% [kernel] [k] ktime_get_update_offsets
0.79% [kernel] [k] __update_cpu_load
0.77% [kernel] [k] acct_update_integrals
0.77% [kernel] [k] hrtimer_interrupt
0.75% [kernel] [k] perf_adjust_freq_unthr_context.part.81
0.73% [kernel] [k] do_gettimeofday
0.73% [kernel] [k] apic_timer_interrupt
0.72% [kernel] [k] timerqueue_add
0.70% [kernel] [k] tick_sched_timerThis is in comparison to my earlier perftop results which were with thp enabled. Keep in mind that this system has a NUMA configuration of
$ cat /sys/devices/system/node/node*/distance 10 20 20 30 20 10 20 20 20 20 10 20 30 20 20 10
so perhaps you would have better luck reproducing the problem using the new ability to fake the distance in between nodes that Peter introduced in 94c0dd3278dd ("x86/numa: Allow specifying node_distance() for numa=fake") with numa=fake=4:10,20,20,30,20,10,20,20,20,20,10,20,30,20,20,10 ?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · David Rientjes
This is in comparison to my earlier perftop results which were with thp enabled. Keep in mind that this system has a NUMA configuration of
$ cat /sys/devices/system/node/node*/distance 10 20 20 30 20 10 20 20 20 20 10 20 30 20 20 10
You could check whether the basic topology is scheduled right by running Andrea's autonumabench:
git://gitorious.org/autonuma-benchmark/autonuma-benchmark.git
and do a "start.sh -A" and send the mainline versus numa/core results.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · David Rientjes
I confirm that numa/core regresses significantly more without thp than the 6.3% regression I reported with thp in terms of throughput on the same system. numa/core at
01aa90068b12("sched: Use the best-buddy 'ideal cpu' in balancing decisions") had 99389.49 SPECjbb2005 bops whereasec05a2311c35("Merge branch 'sched/urgent' into sched/core") had 122246.90 SPECjbb2005 bops, a 23.0% regression.
What is the base performance figure with THP disabled? Your baseline was:
sched/core at ec05a2311c35: 136918.34 SPECjbb2005
Would be interesting to see how that kernel reacts to THP off.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Ingo Molnar
Quoted 7 lines
I confirm that numa/core regresses significantly more without thp than the 6.3% regression I reported with thp in terms of throughput on the same system. numa/core at
01aa90068b12("sched: Use the best-buddy 'ideal cpu' in balancing decisions") had 99389.49 SPECjbb2005 bops whereasec05a2311c35("Merge branch 'sched/urgent' into sched/core") had 122246.90 SPECjbb2005 bops, a 23.0% regression.
Quoted 7 lines
What is the base performance figure with THP disabled? Your baseline was:
sched/core at ec05a2311c35: 136918.34 SPECjbb2005Would be interesting to see how that kernel reacts to THP off.
In summary, the benchmarks that I've collected thus far are:
THP enabled:
numa/core at ec05a2311c35: 136918.34 SPECjbb2005 bops
numa/core at 01aa90068b12: 128315.19 SPECjbb2005 bops (-6.3%)
THP disabled:
numa/core at ec05a2311c35: 122246.90 SPECjbb2005 bops
numa/core at 01aa90068b12: 99389.49 SPECjbb2005 bops (-23.0%)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 29 lines · David Rientjes
Quoted 17 lines · Ingo Molnar
No doubt numa/core should not regress with THP off or on and I'll fix that.
As a background, here's how SPECjbb gets slower on mainline (v3.7-rc6) if you boot Mel's kernel config and turn THP forcibly off:
(avg: 502395 ops/sec)
(avg: 505902 ops/sec)
(avg: 509271 ops/sec)# echo never > /sys/kernel/mm/transparent_hugepage/enabled(avg: 376989 ops/sec)
(avg: 379463 ops/sec)
(avg: 378131 ops/sec)Quoted 9 lines
A ~30% slowdown.
[ How do I know? I asked for Mel's kernel config days ago and
actually booted Mel's very config in the past few days, spending hours on testing it on 4 separate NUMA systems, trying to find Mel's regression. In the past Mel was a reliable tester so I blindly trusted his results. Was that some weird sort of denial on my part? :-) ]
Quoted 7 lines
I confirm that numa/core regresses significantly more without thp than the 6.3% regression I reported with thp in terms of throughput on the same system. numa/core at
01aa90068b12("sched: Use the best-buddy 'ideal cpu' in balancing decisions") had 99389.49 SPECjbb2005 bops whereasec05a2311c35("Merge branch 'sched/urgent' into sched/core") had 122246.90 SPECjbb2005 bops, a 23.0% regression.
I also see different regressions and gains depending on the number of warehouses. For low number of warehouses without THP the regression was severe but flat for higher number of warehouses. I explained in another mail that specjbb reports based on peak figures and regressions outside the peak can be missed as a result so we should watch out for that.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 46 lines
Quoted 10 lines
Ingo, stop doing this kind of crap.
Let's make it clear: if the NUMA patches continue to regress performance for reasonable loads (and that very much includes "no THP") then they won't be merged.
You seem to be in total denial. Every time Mel sends out results that show that your patches MAKE PERFORMANCE WORSE you blame Mel, or blame the load, and never seem to admit that performance got worse.
No doubt numa/core should not regress with THP off or on and I'll fix that.
As a background, here's how SPECjbb gets slower on mainline (v3.7-rc6) if you boot Mel's kernel config and turn THP forcibly off:
(avg: 502395 ops/sec)
(avg: 505902 ops/sec)
(avg: 509271 ops/sec)# echo never > /sys/kernel/mm/transparent_hugepage/enabled(avg: 376989 ops/sec)
(avg: 379463 ops/sec)
(avg: 378131 ops/sec)A ~30% slowdown.
[ How do I know? I asked for Mel's kernel config days ago and
actually booted Mel's very config in the past few days, spending hours on testing it on 4 separate NUMA systems, trying to find Mel's regression. In the past Mel was a reliable tester so I blindly trusted his results. Was that some weird sort of denial on my part? :-) ]Every time a regression is reported I take it seriously - and there were performance regression reports against numa/core not just from Mel and I'm sure there will be more in the future. For example I'm taking David Rijentje's fresh performance regression report seriously as well.
What I have some problem with is Mel sending me his kernel config as the thing he tested, and which config included:
Quoted 6 lines
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=ybut he apparently went and explicitly disabled THP on top of that - which was just a weird choice of 'negative test tuning' to keep unreported.
I've already apologised for this omission and I'll apologise again if necessary. The whole point of implementing MM Tests was to mitigate exactly this sort of situation where the testing conditions are suspicious so the configuration file and scripts can be rechecked. I was assuming the lack of THP usage was due to the JVM using shared memory because a few years ago the JVM I was using at the time did this if configured for hugepage usage. Assumptions made a complete ass out of me here. When I did the recheck of what the JVM was actually doing and reexamined the configuration, I saw the mistake and admitted it immediately.
I want to be absolutely clear that it was unintentional to disable THP like this which is why I did not report it. I did not deliberately hide such information because that would be completely unacceptable. The root of the mistake is actually from a few years ago where tests would be configured to run with base pages, huge pages and transparent hugepages -- similar to how we might currently test vanilla kernel, hard bindings and automatic migration. Because of their history, some mmtest scripts support running with multiple page sizes and I neglected to properly identify this and retrofit a "default hugepage" configuration.
I've also been already clear that this was done for *all* the specjbb tests. It was still a mistake but it was evenly applied.
I've added two extra configuration files to run specjbb single and multi JVMs with THP enabled. It takes about 1.5 to 2 hours to complete a single test which means a full battery of tests for autonuma, vanilla kernel and schednuma will take a little over 24 hours (4 specjbb tests, autobench and a few basic performance tests like kernbench, hackbench etc). They will not be running back-to-back as the machine is not dedicated to this. I'll report when they're done.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 2 lines · Mel Gorman
I've added two extra configuration files to run specjbb single and multi JVMs with THP enabled. It takes about 1.5 to 2 hours to complete a single
1.5 to 2 hours if running to the full set of warehouses required for a compliant run. Configuration is current limited to a smaller number of threads so each specjbb should complete between 40 minutes and an hour.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/1 mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
In reply to Ingo MolnarQuoted 11 lines · Ingo Molnar
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Ingo Molnar
Hugepages is a must for most forms of NUMA/HPC. This alone questions the relevance of most of your prior numa/core testing results. I now have to strongly dispute your other conclusions as well.
Ingo, stop doing this kind of crap.
Quoted 7 lines
Quoted 3 lines
Let's make it clear: if the NUMA patches continue to regress performance for reasonable loads (and that very much includes "no THP") then they won't be merged.
[...]
No doubt numa/core should not regress with THP off or on and I'll fix that.
Once it was clear how Mel's workload was configured I could reproduce it immediately myself as well and the fix was easy and
straightforward: the attached patch should do the trick.
(Lightly tested.)
Updated 32-warehouse SPECjbb test benchmarks on a 4x4 64 GB system:
mainline: 395 k/sec
numa/core +patch: 512 k/sec [ +29.6% ]
mainline +THP: 524 k/sec
numa/core +patch +THP: 654 k/sec [ +24.8% ]
So here on my box the reported 32-warehouse SPECjbb regressions are fixed to the best of my knowledge, and numa/core is now a nice unconditional speedup over mainline.
CONFIG_NUMA_BALANCING=y brings roughly as much of a speedup to mainline as CONFIG_TRANSPARENT_HUGEPAGE=y itself - and the combination of the two features brings roughly a combination of
speedups: +65%, which looks pretty impressive.
This fix had no impact on the good "+THP +NUMA" results that were reproducible with -v16 already.
Mel, David, could you give this patch too a whirl? It should improve !THP workloads.
( The 4x JVM regression is still an open bug I think - I'll
re-check and fix that one next, no need to re-report it, I'm on it. )
Thanks,
Ingo
----------------------------->
Subject: mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
From: Ingo Molnar <[email protected]>
Date: Tue Nov 20 15:48:26 CET 2012
Reduce the 4K page fault count by looking around and processing nearby pages if possible.
To keep the logic simple and straightforward we do a couple of simplifications:
- we only scan in the HPAGE_SIZE range of the faulting address
- we only go as far as the vma allows us
Also simplify the do_numa_page() flow while at it and fix the previous double faulting we incurred due to not properly fixing up freshly migrated ptes.
--- mm/memory.c | 101 +++++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 66 insertions(+), 35 deletions(-)
mm/memory.c +66 -35
@@ -3455,64 +3455,94 @@ static int do_nonlinear_fault(struct mm_return __do_fault (mm, vma, address, pmd, pgoff, flags, orig_pte); } -static int do_numa_page (struct mm_struct *mm,struct vm_area_struct *vma, +static int __do_numa_page (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,pte_t *ptep,pmd_t *pmd, -unsigned int flags,pte_t entry) +unsigned int flags,pte_t entry,spinlock_t *ptl) { -struct page *page =NULL ; -int node, page_nid =-1 ; -int last_cpu =-1 ; -spinlock_t *ptl; - - ptl =pte_lockptr (mm, pmd); -spin_lock (ptl); -if (unlikely (!pte_same (*ptep, entry))) - goto out_unlock; +struct page *page; +int new_node; page =vm_normal_page (vma, address, entry);if (page) { -get_page (page); - page_nid =page_to_nid (page); - last_cpu =page_last_cpu (page); - node =mpol_misplaced (page, vma, address); -if (node !=-1 && node != page_nid) +int page_nid =page_to_nid (page); +int last_cpu =page_last_cpu (page); + + new_node =mpol_misplaced (page, vma, address); +if (new_node !=-1 && new_node != page_nid) goto migrate; +task_numa_fault (page_nid, last_cpu,1 ); } -out_pte_upgrade_unlock: +out_pte_upgrade:flush_cache_page (vma, address,pte_pfn (entry)); -ptep_modify_prot_start (mm, address, ptep); entry =pte_modify (entry, vma->vm_page_prot); +if (pte_dirty (entry)) + entry =pte_mkwrite (entry);ptep_modify_prot_commit (mm, address, ptep, entry); -/* No TLB flush needed because we upgraded the PTE */ -update_mmu_cache (vma, address, ptep); - -out_unlock: -pte_unmap_unlock (ptep, ptl); - -if (page) { -task_numa_fault (page_nid, last_cpu,1 ); -put_page (page); - } out:return 0 ; migrate: +get_page (page);pte_unmap_unlock (ptep, ptl); -if (migrate_misplaced_page (page, node)) { +migrate_misplaced_page (page, new_node); + +/* Re-check after migration: */ + + ptl =pte_lockptr (mm, pmd); +spin_lock (ptl); + entry =ACCESS_ONCE (*ptep); + +if (!pte_numa (vma, entry)) goto out; - } - page =NULL ; - ptep =pte_offset_map_lock (mm, pmd, address, &ptl); -if (!pte_same (*ptep, entry)) - goto out_unlock; + page =vm_normal_page (vma, address, entry); + goto out_pte_upgrade; +} - goto out_pte_upgrade_unlock; +/* +* Add a simple loop to also fetch ptes within the same pmd: +*/ +static int do_numa_page (struct mm_struct *mm,struct vm_area_struct *vma, +unsigned long addr0,pte_t *ptep0,pmd_t *pmd, +unsigned int flags,pte_t entry0) +{ +unsigned long addr0_pmd = addr0 &PMD_MASK ; +unsigned long addr_start; +unsigned long addr; +spinlock_t *ptl; +int entries =0 ; +pte_t *ptep; + + addr_start =max (addr0_pmd, vma->vm_start); + ptep =pte_offset_map (pmd, addr_start); + + ptl =pte_lockptr (mm, pmd); +spin_lock (ptl); + +for (addr = addr_start; addr < vma->vm_end; addr +=PAGE_SIZE , ptep++) { +pte_t entry; + + entry =ACCESS_ONCE (*ptep); + +if ((addr &PMD_MASK ) != addr0_pmd) +break ; +if (!pte_present (entry)) +continue ; +if (!pte_numa (vma, entry)) +continue ; + +__do_numa_page (mm, vma, addr, ptep, pmd, flags, entry, ptl); + entries++; + } + +pte_unmap_unlock (ptep, ptl); + +return 0 ; }/*
@@ -3536,6 +3566,7 @@ int handle_pte_fault(struct mm_struct *mspinlock_t *ptl; entry =ACCESS_ONCE (*pte); +if (!pte_present (entry)) {if (pte_none (entry)) {if (vma->vm_ops) {
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 1/1 mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
Ok, the patch withstood a bit more testing as well. Below is a v2 version of it, with a couple of cleanups (no functional changes).
Thanks,
Ingo
----------------->
Subject: mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
From: Ingo Molnar <[email protected]>
Date: Tue Nov 20 15:48:26 CET 2012
Reduce the 4K page fault count by looking around and processing nearby pages if possible.
To keep the logic and cache overhead simple and straightforward we do a couple of simplifications:
- we only scan in the HPAGE_SIZE range of the faulting address
- we only go as far as the vma allows us
Also simplify the do_numa_page() flow while at it and fix the previous double faulting we incurred due to not properly fixing up freshly migrated ptes.
--- mm/memory.c | 99 ++++++++++++++++++++++++++++++++++++++---------------------- 1 file changed, 64 insertions(+), 35 deletions(-)
mm/memory.c +64 -35
@@ -3455,64 +3455,93 @@ static int do_nonlinear_fault(struct mm_return __do_fault (mm, vma, address, pmd, pgoff, flags, orig_pte); } -static int do_numa_page (struct mm_struct *mm,struct vm_area_struct *vma, +static int __do_numa_page (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,pte_t *ptep,pmd_t *pmd, -unsigned int flags,pte_t entry) +unsigned int flags,pte_t entry,spinlock_t *ptl) { -struct page *page =NULL ; -int node, page_nid =-1 ; -int last_cpu =-1 ; -spinlock_t *ptl; - - ptl =pte_lockptr (mm, pmd); -spin_lock (ptl); -if (unlikely (!pte_same (*ptep, entry))) - goto out_unlock; +struct page *page; +int new_node; page =vm_normal_page (vma, address, entry);if (page) { -get_page (page); - page_nid =page_to_nid (page); - last_cpu =page_last_cpu (page); - node =mpol_misplaced (page, vma, address); -if (node !=-1 && node != page_nid) +int page_nid =page_to_nid (page); +int last_cpu =page_last_cpu (page); + +task_numa_fault (page_nid, last_cpu,1 ); + + new_node =mpol_misplaced (page, vma, address); +if (new_node !=-1 && new_node != page_nid) goto migrate; } -out_pte_upgrade_unlock: +out_pte_upgrade:flush_cache_page (vma, address,pte_pfn (entry)); -ptep_modify_prot_start (mm, address, ptep); entry =pte_modify (entry, vma->vm_page_prot); +if (pte_dirty (entry)) + entry =pte_mkwrite (entry);ptep_modify_prot_commit (mm, address, ptep, entry); -/* No TLB flush needed because we upgraded the PTE */ -update_mmu_cache (vma, address, ptep); - -out_unlock: -pte_unmap_unlock (ptep, ptl); - -if (page) { -task_numa_fault (page_nid, last_cpu,1 ); -put_page (page); - } out:return 0 ; migrate: +get_page (page);pte_unmap_unlock (ptep, ptl); -if (migrate_misplaced_page (page, node)) { +migrate_misplaced_page (page, new_node);/* Drops the page reference */ + +/* Re-check after migration: */ + + ptl =pte_lockptr (mm, pmd); +spin_lock (ptl); + entry =ACCESS_ONCE (*ptep); + +if (!pte_numa (vma, entry)) goto out; - } - page =NULL ; - ptep =pte_offset_map_lock (mm, pmd, address, &ptl); -if (!pte_same (*ptep, entry)) - goto out_unlock; + goto out_pte_upgrade; +} + +/* +* Add a simple loop to also fetch ptes within the same pmd: +*/ +static int do_numa_page (struct mm_struct *mm,struct vm_area_struct *vma, +unsigned long addr0,pte_t *ptep0,pmd_t *pmd, +unsigned int flags,pte_t entry0) +{ +unsigned long addr0_pmd; +unsigned long addr_start; +unsigned long addr; +spinlock_t *ptl; +pte_t *ptep; + + addr0_pmd = addr0 &PMD_MASK ; + addr_start =max (addr0_pmd, vma->vm_start); - goto out_pte_upgrade_unlock; + ptep =pte_offset_map (pmd, addr_start); + ptl =pte_lockptr (mm, pmd); +spin_lock (ptl); + +for (addr = addr_start; addr < vma->vm_end; addr +=PAGE_SIZE , ptep++) { +pte_t entry; + + entry =ACCESS_ONCE (*ptep); + +if ((addr &PMD_MASK ) != addr0_pmd) +break ; +if (!pte_present (entry)) +continue ; +if (!pte_numa (vma, entry)) +continue ; + +__do_numa_page (mm, vma, addr, ptep, pmd, flags, entry, ptl); + } + +pte_unmap_unlock (ptep, ptl); + +return 0 ; }/*
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH, v2] mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
Quoted 6 lines · Ingo Molnar
Subject: mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
From: Ingo Molnar <[email protected]>
Date: Tue Nov 20 15:48:26 CET 2012Reduce the 4K page fault count by looking around and processing nearby pages if possible.
This is essentially what autonuma does with PMD level NUMA faults, so we know this idea works.
Performance measurements will show us how much of an impact it makes, since I don't think we have never done apples to apples comparisons with just this thing toggled :)
The patch looks good for me, just a nit-pick on the comment to do_numa_page().
Other than that:
Quoted 84 lines
Index:
linux/mm/memory.c
===================================================================mm/memory.c +24
-31@@ -3455,64 +3455,93 @@ static int do_nonlinear_fault(struct mm_return __do_fault (mm, vma, address, pmd, pgoff, flags, orig_pte); } -static int do_numa_page (struct mm_struct *mm,struct vm_area_struct *vma, +static int __do_numa_page (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,pte_t *ptep,pmd_t *pmd, -unsigned int flags,pte_t entry) +unsigned int flags,pte_t entry,spinlock_t *ptl) { -struct page *page =NULL ; -int node, page_nid =-1 ; -int last_cpu =-1 ; -spinlock_t *ptl; - - ptl =pte_lockptr (mm, pmd); -spin_lock (ptl); -if (unlikely (!pte_same (*ptep, entry))) - goto out_unlock; +struct page *page; +int new_node; page =vm_normal_page (vma, address, entry);if (page) { -get_page (page); - page_nid =page_to_nid (page); - last_cpu =page_last_cpu (page); - node =mpol_misplaced (page, vma, address); -if (node !=-1 && node != page_nid) +int page_nid =page_to_nid (page); +int last_cpu =page_last_cpu (page); + +task_numa_fault (page_nid, last_cpu,1 ); + + new_node =mpol_misplaced (page, vma, address); +if (new_node !=-1 && new_node != page_nid) goto migrate; } -out_pte_upgrade_unlock: +out_pte_upgrade:flush_cache_page (vma, address,pte_pfn (entry)); -ptep_modify_prot_start (mm, address, ptep); entry =pte_modify (entry, vma->vm_page_prot); +if (pte_dirty (entry)) + entry =pte_mkwrite (entry);ptep_modify_prot_commit (mm, address, ptep, entry); -/* No TLB flush needed because we upgraded the PTE */ -update_mmu_cache (vma, address, ptep); - -out_unlock: -pte_unmap_unlock (ptep, ptl); - -if (page) { -task_numa_fault (page_nid, last_cpu,1 ); -put_page (page); - } out:return 0 ; migrate: +get_page (page);pte_unmap_unlock (ptep, ptl); -if (migrate_misplaced_page (page, node)) { +migrate_misplaced_page (page, new_node);/* Drops the page reference */ + +/* Re-check after migration: */ + + ptl =pte_lockptr (mm, pmd); +spin_lock (ptl); + entry =ACCESS_ONCE (*ptep); + +if (!pte_numa (vma, entry)) goto out; - } - page =NULL ;
mm/memory.c static int do_nonlinear_fault(struct mm_ · line 3509
Quoted 10 lines
mm/memory.c +6
-3- ptep =pte_offset_map_lock (mm, pmd, address, &ptl); -if (!pte_same (*ptep, entry)) - goto out_unlock; + goto out_pte_upgrade; +} + +/* +* Add a simple loop to also fetch ptes within the same pmd: +*/
That's not a very useful comment. How about something like:
/*
* Also fault over nearby ptes from within the same pmd and vma,
* in order to minimize the overhead from page fault exceptions
* and TLB flushes.
*/
Quoted 40 lines
mm/memory.c +34
-1+static int do_numa_page (struct mm_struct *mm,struct vm_area_struct *vma, +unsigned long addr0,pte_t *ptep0,pmd_t *pmd, +unsigned int flags,pte_t entry0) +{ +unsigned long addr0_pmd; +unsigned long addr_start; +unsigned long addr; +spinlock_t *ptl; +pte_t *ptep; + + addr0_pmd = addr0 &PMD_MASK ; + addr_start =max (addr0_pmd, vma->vm_start); - goto out_pte_upgrade_unlock; + ptep =pte_offset_map (pmd, addr_start); + ptl =pte_lockptr (mm, pmd); +spin_lock (ptl); + +for (addr = addr_start; addr < vma->vm_end; addr +=PAGE_SIZE , ptep++) { +pte_t entry; + + entry =ACCESS_ONCE (*ptep); + +if ((addr &PMD_MASK ) != addr0_pmd) +break ; +if (!pte_present (entry)) +continue ; +if (!pte_numa (vma, entry)) +continue ; + +__do_numa_page (mm, vma, addr, ptep, pmd, flags, entry, ptl); + } + +pte_unmap_unlock (ptep, ptl); + +return 0 ; }/*
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH, v2] mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
Quoted 3 lines · Rik van Riel
Performance measurements will show us how much of an impact it makes, since I don't think we have never done apples to apples comparisons with just this thing toggled :)
I've done a couple of quick measurements to characterise it: as expected this patch simply does not matter much when THP is enabled - and most testers I worked with had THP enabled.
Testing with THP off hurst most NUMA workloads dearly and tells very little about the real NUMA story of these workloads. If you turn off THP you are living with a constant ~25% regression - just check the THP and no-THP numbers I posted:
[ 32-warehouse SPECjbb test benchmarks ]
mainline: 395 k/sec
mainline +THP: 524 k/sec
numa/core +patch: 512 k/sec [ +29.6% ]
numa/core +patch +THP: 654 k/sec [ +24.8% ]
The group of testers who had THP disabled was thus very low - maybe only Mel alone? The testers I worked with all had THP enabled.
I'd encourage everyone to report unusual 'tweaks' done before tests are reported - no matter how well intended the purpose of that tweak. There's just so many config variations we can test and we obviously check the most logically and most scalably configured system variants first.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: [PATCH, v2] mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
Quoted 24 lines · Ingo Molnar
Quoted 3 lines · Rik van Riel
Performance measurements will show us how much of an impact it makes, since I don't think we have never done apples to apples comparisons with just this thing toggled :)
I've done a couple of quick measurements to characterise it: as expected this patch simply does not matter much when THP is enabled - and most testers I worked with had THP enabled.
Testing with THP off hurst most NUMA workloads dearly and tells very little about the real NUMA story of these workloads. If you turn off THP you are living with a constant ~25% regression - just check the THP and no-THP numbers I posted:
[ 32-warehouse SPECjbb test benchmarks ]
mainline: 395 k/sec
mainline +THP: 524 k/secnuma/core +patch: 512 k/sec [ +29.6% ]
numa/core +patch +THP: 654 k/sec [ +24.8% ]
Quoted 7 lines
The group of testers who had THP disabled was thus very low - maybe only Mel alone? The testers I worked with all had THP enabled.
I'd encourage everyone to report unusual 'tweaks' done before tests are reported - no matter how well intended the purpose of that tweak.
Jeez, it was an oversight. Do I need to send a card or something?
Quoted 4 lines
There's just so many config variations we can test and we obviously check the most logically and most scalably configured system variants first.
I managed to not break the !THP case for the most part in balancenuma for the cases I looked at. As stated elsewhere not all machines can support THP that care about HPC -- ppc64 is a major example. THPs are not always available, particularly on the node you are trying to migrate to is fragmented. You can just fail the migration in this case of course but unless you are willing to compact, this situation can persist for a long time. You get to keep THP but on a remote node. If we are to ever choose to split THP to get better placement then we must be able to cope with the !THP case from the start. Lastly, not all workloads can use THP if they depend heavily on large files or shared memory.
Focusing on the THP case initially will produce better figures but I worry it will eventually kick us in the shins and be hard to back out of.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH, v2] mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
In reply to Rik van RielQuoted 8 lines · Rik van Riel
mm/memory.c static int do_nonlinear_fault(struct mm_ · line 3509
Quoted 5 lines
mm/memory.c +5
-0+} + +/* +* Add a simple loop to also fetch ptes within the same pmd: +*/ That's not a very useful comment. How about something like:
Quoted 5 lines
/*
* Also fault over nearby ptes from within the same pmd and vma,
* in order to minimize the overhead from page fault exceptions
* and TLB flushes.
*/
There's no TLB flushes here. But I'm fine with the other part so I've updated the comment to say:
/*
* Also fault over nearby ptes from within the same pmd and vma,
* in order to minimize the overhead from page fault exceptions:
*/
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH, v2] mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
In reply to Ingo MolnarQuoted 13 lines · Ingo Molnar
Reduce the 4K page fault count by looking around and processing nearby pages if possible.
To keep the logic and cache overhead simple and straightforward we do a couple of simplifications:
- we only scan in the
HPAGE_SIZErange of the faulting address
- we only go as far as the vma allows usAlso simplify the
do_numa_page()flow while at it and fix the previous double faulting we incurred due to not properly fixing up freshly migrated ptes.
Quoted 8 lines
Suggested-by Mel Gorman <[email protected]>Cc Linus Torvalds <[email protected]>Cc Andrew Morton <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Rik van Riel <[email protected]>Cc Hugh Dickins <[email protected]>Signed-off-by Ingo Molnar <[email protected]>
Ok, this is significantly better, it almost cut the regression in half on my system. With THP enabled:
numa/core at ec05a2311c35: 136918.34 SPECjbb2005 bops numa/core at 01aa90068b12: 128315.19 SPECjbb2005 bops (-6.3%) numa/core at 01aa90068b12 + patch: 132523.06 SPECjbb2005 bops (-3.2%)
Here's the newest perftop, which is radically different than before (not nearly the number of newly-added numa/core functions in the biggest consumers) but still incurs significant overhead from page faults.
92.18% perf-6697.map [.] 0x00007fe2c5afd079
1.20% libjvm.so [.] instanceKlass::oop_push_contents(PSPromotionManag
1.05% libjvm.so [.] PSPromotionManager::drain_stacks_depth(bool)
0.78% libjvm.so [.] PSPromotionManager::copy_to_survivor_space(oopDes
0.59% libjvm.so [.] PSPromotionManager::claim_or_forward_internal_dep
0.49% [kernel] [k] page_fault
0.27% libjvm.so [.] Copy::pd_disjoint_words(HeapWord*, HeapWord*, unsigned lo
0.27% libc-2.3.6.so [.] __gettimeofday
0.19% libjvm.so [.] CardTableExtension::scavenge_contents_parallel(ObjectStar
0.16% [kernel] [k] getnstimeofday
0.14% [kernel] [k] _raw_spin_lock
0.13% [kernel] [k] generic_smp_call_function_interrupt
0.11% [kernel] [k] ktime_get
0.11% [kernel] [k] rcu_check_callbacks
0.10% [kernel] [k] read_tsc
0.09% libjvm.so [.] os::javaTimeMillis()
0.09% [kernel] [k] clear_page_c
0.08% [kernel] [k] flush_tlb_func
0.08% [kernel] [k] ktime_get_update_offsets
0.07% [kernel] [k] task_tick_fair
0.06% [kernel] [k] emulate_vsyscall
0.06% libjvm.so [.] oopDesc::size_given_klass(Klass*)
0.06% [kernel] [k] __do_page_fault
0.04% [kernel] [k] __bad_area_nosemaphore
0.04% perf [.] 0x000000000003310b
0.04% libjvm.so [.] objArrayKlass::oop_push_contents(PSPromotionManager*, oop
0.04% [kernel] [k] run_timer_softirq
0.04% [kernel] [k] copy_user_generic_string
0.03% [kernel] [k] task_numa_fault
0.03% [kernel] [k] smp_call_function_many
0.03% [kernel] [k] retint_swapgs
0.03% [kernel] [k] update_cfs_shares
0.03% [kernel] [k] error_sti
0.03% [kernel] [k] _raw_spin_lock_irq
0.03% [kernel] [k] update_curr
0.02% [kernel] [k] write_ok_or_segv
0.02% [kernel] [k] call_function_interrupt
0.02% [kernel] [k] __do_softirq
0.02% [kernel] [k] acct_update_integrals
0.02% [kernel] [k] x86_pmu_disable_all
0.02% [kernel] [k] apic_timer_interrupt
0.02% [kernel] [k] tick_sched_timer--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH, v2] mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
Quoted 2 lines · David Rientjes
Ok, this is significantly better, it almost cut the regression in half on my system. [...]
The other half still seems to be related to the emulation faults that I fixed in the other patch:
Quoted 2 lines
0.49% [kernel] [k] page_fault
0.06% [kernel] [k] emulate_vsyscall Plus TLB flush costs:
Quoted 2 lines
0.13% [kernel] [k] generic_smp_call_function_interrupt
0.08% [kernel] [k] flush_tlb_funcfor which you should try the third patch I sent.
So please try all my fixes - the easiest way to do that would be to try the latest tip:master that has all related fixes integrated and send me a new perf top output - most page fault and TLB flush overhead should be gone from the profile.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: [PATCH, v2] mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
In reply to Ingo MolnarQuoted 35 lines · Ingo Molnar
Ok, the patch withstood a bit more testing as well. Below is a v2 version of it, with a couple of cleanups (no functional changes).
Thanks,
Ingo
----------------->
Subject: mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
From: Ingo Molnar <[email protected]>
Date: Tue Nov 20 15:48:26 CET 2012Reduce the 4K page fault count by looking around and processing nearby pages if possible.
To keep the logic and cache overhead simple and straightforward we do a couple of simplifications:
- we only scan in the
HPAGE_SIZErange of the faulting address
- we only go as far as the vma allows usAlso simplify the
do_numa_page()flow while at it and fix the previous double faulting we incurred due to not properly fixing up freshly migrated ptes.Suggested-by Mel Gorman <[email protected]>Cc Linus Torvalds <[email protected]>Cc Andrew Morton <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Rik van Riel <[email protected]>Cc Hugh Dickins <[email protected]>Signed-off-by Ingo Molnar <[email protected]>
Quoted 4 lines
--- mm/memory.c | 99 ++++++++++++++++++++++++++++++++++++++---------------------- 1 file changed, 64 insertions(+), 35 deletions(-)
This is functionally similar to what balancenuma does but there is one key difference worth noting. I only mark the PMD pmd_numa if all the pages pointed to by the updated[*] PTEs underneath are on the same node. The intention is that if the workload is converged on a PMD boundary then a migration of all the pages underneath will be remote->local copies. If the workload is not converged on a PMD boundary and you handle all the faults then you are potentially incurring remote->remote copies.
It also means that if the workload is not converged on the PMD boundary then a PTE fault is just one page. With yours, it will be the full PMD every time, right?
[*] Note I said only the updated ptes are checked. I do not check every
PTE underneath. I could but felt the benefit would be marginal.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH, v2] mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
In reply to Ingo MolnarThis patch cause boot hang on our SNB EP 2 sockets machine with some segmentation fault.
revert it recovers booting.
============
[ 8.290147] Freeing unused kernel memory: 1264k freed [ 8.306140] Freeing unused kernel memory: 1592k freed [ 8.342668] init[250]: segfault at 20da510 ip 00000000020da510 sp 00007fff26788040 error 15[ 8.350983] usb 2-1: New USB device found, idVendor=8087, idProduct=0024 [ 8.350987] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 8.351266] hub 2-1:1.0: USB hub found [ 8.351346] hub 2-1:1.0: 8 ports detected
Segmentation fault
[ 8.626633] usb 2-1.4: new full-speed USB device number 3 using ehci_hcd [ 8.721391] usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10 [ 8.729536] usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 8.738540] usb 2-1.4: Product: Virtual Keyboard and Mouse [ 8.745134] usb 2-1.4: Manufacturer: American Megatrends Inc. [ 8.752026] usb 2-1.4: SerialNumber: serial [ 8.758877] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input1 [ 8.774428] hid-generic 0003:046B:FF10.0001: input,hidraw0: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0 [ 8.793393] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input2 [ 8.809140] hid-generic 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1 [ 8.899511] usb 2-1.7: new low-speed USB device number 4 using ehci_hcd [ 9.073473] usb 2-1.7: New USB device found, idVendor=0557, idProduct=2220 [ 9.081633] usb 2-1.7: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 9.090643] usb 2-1.7: Product: ATEN CS-1758/54 [ 9.096258] usb 2-1.7: Manufacturer: ATEN [ 9.134093] input: ATEN ATEN CS-1758/54 as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.7/2-1.7:1.0/input/input3 [ 9.146804] hid-generic 0003:0557:2220.0003: input,hidraw2: USB HID v1.10 Keyboard [ATEN ATEN CS-1758/54] on usb-0000:00:1d.0-1.7/input0 [ 9.184396] input: ATEN ATEN CS-1758/54 as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.7/2-1.7:1.1/input/input4 [ 9.197210] hid-generic 0003:0557:2220.0004: input,hidraw3: USB HID
v1.10 Mouse [ATEN ATEN CS-1758/54] on usb-0000:00:1d.0-1.7/input1
<hang here>
Quoted 178 lines · Ingo Molnar
Ok, the patch withstood a bit more testing as well. Below is a v2 version of it, with a couple of cleanups (no functional changes).
Thanks,
Ingo
----------------->
Subject: mm, numa: Turn 4K pte NUMA faults into effective hugepage ones
From: Ingo Molnar <[email protected]>
Date: Tue Nov 20 15:48:26 CET 2012Reduce the 4K page fault count by looking around and processing nearby pages if possible.
To keep the logic and cache overhead simple and straightforward we do a couple of simplifications:
- we only scan in the
HPAGE_SIZErange of the faulting address
- we only go as far as the vma allows usAlso simplify the
do_numa_page()flow while at it and fix the previous double faulting we incurred due to not properly fixing up freshly migrated ptes.Suggested-by Mel Gorman <[email protected]>Cc Linus Torvalds <[email protected]>Cc Andrew Morton <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Rik van Riel <[email protected]>Cc Hugh Dickins <[email protected]>Signed-off-by Ingo Molnar <[email protected]>--- mm/memory.c | 99 ++++++++++++++++++++++++++++++++++++++---------------------- 1 file changed, 64 insertions(+), 35 deletions(-)Index:
linux/mm/memory.c
===================================================================mm/memory.c +64
-36@@ -3455,64 +3455,93 @@ static int do_nonlinear_fault(struct mm_return __do_fault (mm, vma, address, pmd, pgoff, flags, orig_pte); } -static int do_numa_page (struct mm_struct *mm,struct vm_area_struct *vma, +static int __do_numa_page (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,pte_t *ptep,pmd_t *pmd, -unsigned int flags,pte_t entry) +unsigned int flags,pte_t entry,spinlock_t *ptl) { -struct page *page =NULL ; -int node, page_nid =-1 ; -int last_cpu =-1 ; -spinlock_t *ptl; - - ptl =pte_lockptr (mm, pmd); -spin_lock (ptl); -if (unlikely (!pte_same (*ptep, entry))) - goto out_unlock; +struct page *page; +int new_node; page =vm_normal_page (vma, address, entry);if (page) { -get_page (page); - page_nid =page_to_nid (page); - last_cpu =page_last_cpu (page); - node =mpol_misplaced (page, vma, address); -if (node !=-1 && node != page_nid) +int page_nid =page_to_nid (page); +int last_cpu =page_last_cpu (page); + +task_numa_fault (page_nid, last_cpu,1 ); + + new_node =mpol_misplaced (page, vma, address); +if (new_node !=-1 && new_node != page_nid) goto migrate; } -out_pte_upgrade_unlock: +out_pte_upgrade:flush_cache_page (vma, address,pte_pfn (entry)); -ptep_modify_prot_start (mm, address, ptep); entry =pte_modify (entry, vma->vm_page_prot); +if (pte_dirty (entry)) + entry =pte_mkwrite (entry);ptep_modify_prot_commit (mm, address, ptep, entry); -/* No TLB flush needed because we upgraded the PTE */ -update_mmu_cache (vma, address, ptep); - -out_unlock: -pte_unmap_unlock (ptep, ptl); - -if (page) { -task_numa_fault (page_nid, last_cpu,1 ); -put_page (page); - } out:return 0 ; migrate: +get_page (page);pte_unmap_unlock (ptep, ptl); -if (migrate_misplaced_page (page, node)) { +migrate_misplaced_page (page, new_node);/* Drops the page reference */ + +/* Re-check after migration: */ + + ptl =pte_lockptr (mm, pmd); +spin_lock (ptl); + entry =ACCESS_ONCE (*ptep); + +if (!pte_numa (vma, entry)) goto out; - } - page =NULL ; - ptep =pte_offset_map_lock (mm, pmd, address, &ptl); -if (!pte_same (*ptep, entry)) - goto out_unlock; + goto out_pte_upgrade; +} + +/* +* Add a simple loop to also fetch ptes within the same pmd: +*/ +static int do_numa_page (struct mm_struct *mm,struct vm_area_struct *vma, +unsigned long addr0,pte_t *ptep0,pmd_t *pmd, +unsigned int flags,pte_t entry0) +{ +unsigned long addr0_pmd; +unsigned long addr_start; +unsigned long addr; +spinlock_t *ptl; +pte_t *ptep; + + addr0_pmd = addr0 &PMD_MASK ; + addr_start =max (addr0_pmd, vma->vm_start); - goto out_pte_upgrade_unlock; + ptep =pte_offset_map (pmd, addr_start); + ptl =pte_lockptr (mm, pmd); +spin_lock (ptl); + +for (addr = addr_start; addr < vma->vm_end; addr +=PAGE_SIZE , ptep++) { +pte_t entry; + + entry =ACCESS_ONCE (*ptep); + +if ((addr &PMD_MASK ) != addr0_pmd) +break ; +if (!pte_present (entry)) +continue ; +if (!pte_numa (vma, entry)) +continue ; + +__do_numa_page (mm, vma, addr, ptep, pmd, flags, entry, ptl); + } + +pte_unmap_unlock (ptep, ptl); + +return 0 ; }/* -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Signature · 8 lines
Thanks
Alex
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>numa/core regressions fixed - more testers wanted
In reply to Ingo MolnarQuoted 3 lines · Ingo Molnar
( The 4x JVM regression is still an open bug I think - I'll
re-check and fix that one next, no need to re-report it, I'm on it. )
So I tested this on !THP too and the combined numbers are now:
|
[ SPECjbb multi-4x8 ] |
[ tx/sec ] v3.7 | numa/core-v16
[ higher is better ] ----- | -------------
|
+THP: 639k | 655k +2.5%
-THP: 510k | 517k +1.3%So it's not a regression anymore, regardless of whether THP is enabled or disabled.
The current updated table of performance results is:
-------------------------------------------------------------------------
[ seconds ] v3.7 AutoNUMA | numa/core-v16 [ vs. v3.7]
[ lower is better ] ----- -------- | ------------- -----------
|
numa01 340.3 192.3 | 139.4 +144.1%
numa01_THREAD_ALLOC 425.1 135.1 | 121.1 +251.0%
numa02 56.1 25.3 | 17.5 +220.5%
|
[ SPECjbb transactions/sec ] |
[ higher is better ] |
|
SPECjbb 1x32 +THP 524k 507k | 638k +21.7%
SPECjbb 1x32 !THP 395k | 512k +29.6%
|
-----------------------------------------------------------------------
|
[ SPECjbb multi-4x8 ] |
[ tx/sec ] v3.7 | numa/core-v16
[ higher is better ] ----- | -------------
|
+THP: 639k | 655k +2.5%
-THP: 510k | 517k +1.3%So I think I've addressed all regressions reported so far - if anyone can still see something odd, please let me know so I can reproduce and fix it ASAP.
Next I'll work on making multi-JVM more of an improvement, and I'll also address any incoming regression reports.
Those of you who would like to test all the latest patches are welcome to pick up latest bits at tip:master:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git master
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: numa/core regressions fixed - more testers wanted
Quoted 44 lines · Ingo Molnar
Quoted 3 lines · Ingo Molnar
( The 4x JVM regression is still an open bug I think - I'll
re-check and fix that one next, no need to re-report it, I'm on it. )So I tested this on !THP too and the combined numbers are now:
| [ SPECjbb multi-4x8 ] | [ tx/sec ] v3.7 | numa/core-v16 [ higher is better ] ----- | ------------- | +THP: 639k | 655k +2.5% -THP: 510k | 517k +1.3%So it's not a regression anymore, regardless of whether THP is enabled or disabled.
The current updated table of performance results is:
-------------------------------------------------------------------------
[ seconds ] v3.7 AutoNUMA | numa/core-v16 [ vs. v3.7] [ lower is better ] ----- -------- | ------------- ----------- | numa01 340.3 192.3 | 139.4 +144.1% numa01_THREAD_ALLOC 425.1 135.1 | 121.1 +251.0% numa02 56.1 25.3 | 17.5 +220.5% | [ SPECjbb transactions/sec ] | [ higher is better ] | | SPECjbb 1x32 +THP 524k 507k | 638k +21.7% SPECjbb 1x32 !THP 395k | 512k +29.6% | ----------------------------------------------------------------------- | [ SPECjbb multi-4x8 ] | [ tx/sec ] v3.7 | numa/core-v16 [ higher is better ] ----- | ------------- | +THP: 639k | 655k +2.5% -THP: 510k | 517k +1.3%
Quoted 3 lines
So I think I've addressed all regressions reported so far - if anyone can still see something odd, please let me know so I can reproduce and fix it ASAP.
I can confirm single JVM JBB is working well for me. I see a 30% improvement over autoNUMA. What I can't make sense of is some perf stats (taken at 80 warehouses on 4 x WST-EX, 512GB memory):
tips numa/core:
5,429,632,865 node-loads
3,806,419,082 node-load-misses(70.1%)
2,486,756,884 node-stores
2,042,557,277 node-store-misses(82.1%)
2,878,655,372 node-prefetches
2,201,441,900 node-prefetch-misses autoNUMA:
4,538,975,144 node-loads
2,666,374,830 node-load-misses(58.7%)
2,148,950,354 node-stores
1,682,942,931 node-store-misses(78.3%)
2,191,139,475 node-prefetches
1,633,752,109 node-prefetch-misses The percentage of misses is higher for numa/core. I would have expected the performance increase be due to lower "node-misses", but perhaps I am misinterpreting the perf data.
One other thing I noticed was both tests are not even using all CPU (75-80%), so I suspect there's a JVM scalability issue with this workload at this number of cpu threads (80). This is a IBM JVM, so there may be some differences. I am curious if any of the others testing JBB are getting 100% cpu utilization at their warehouse peak.
So, while the performance results are encouraging, I would like to correlate it with some kind of perf data that confirms why we think it's better.
Quoted 3 lines
Next I'll work on making multi-JVM more of an improvement, and I'll also address any incoming regression reports.
I have issues with multiple KVM VMs running either JBB or dbench-in-tmpfs, and I suspect whatever I am seeing is similar to whatever multi-jvm in baremetal is. What I typically see is no real convergence of a single node for resource usage for any of the VMs. For example, when running 8 VMs, 10 vCPUs each, a VM may have the following resource usage:
host cpu usage from cpuacct cgroup:
/cgroup/cpuacct/libvirt/qemu/at-vm01
node00 node01 node02 node03
199056918180|005% 752455339099|020% 1811704146176|049% 888803723722|024%
And VM memory placement in host(in pages):
node00 node01 node02 node03
107566|023% 115245|025% 117807|025% 119414|025%
Conversely, autoNUMA usually has 98+% for cpu and memory in one of the host nodes for each of these VMs. AutoNUMA is about 30% better in these tests.
That is data for the entire run time, and "not converged" could possibly mean, "converged but moved around", but I doubt that's what happening.
Here's perf data for the dbench VMs:
numa/core:
468,634,508 node-loads
210,598,643 node-load-misses(44.9%)
172,735,053 node-stores
107,535,553 node-store-misses(51.1%)
208,064,103 node-prefetches
160,858,933 node-prefetch-misses autoNUMA:
666,498,425 node-loads
222,643,141 node-load-misses(33.4%)
219,003,566 node-stores
99,243,370 node-store-misses(45.3%)
315,439,315 node-prefetches
254,888,403 node-prefetch-misses These seems to make a little more sense to me, but the percentages for autoNUMA still seem a little high (but at least lower then numa/core). I need to take a manually pinned measurement to compare.
Quoted 4 lines
Those of you who would like to test all the latest patches are welcome to pick up latest bits at tip:master:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitmaster
I've been running on numa/core, but I'll switch to master and try these again.
Thanks,
-Andrew Theurer
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: numa/core regressions fixed - more testers wanted
Quoted 3 lines · Andrew Theurer
I can confirm single JVM JBB is working well for me. I see a 30% improvement over autoNUMA. What I can't make sense of is some perf stats (taken at 80 warehouses on 4 x WST-EX, 512GB memory):
AutoNUMA does not have native THP migration, that may explain some of the difference.
Quoted 18 lines
tips numa/core:
5,429,632,865 node-loads 3,806,419,082 node-load-misses(70.1%) 2,486,756,884 node-stores 2,042,557,277 node-store-misses(82.1%) 2,878,655,372 node-prefetches 2,201,441,900 node-prefetch-missesautoNUMA:
4,538,975,144 node-loads 2,666,374,830 node-load-misses(58.7%) 2,148,950,354 node-stores 1,682,942,931 node-store-misses(78.3%) 2,191,139,475 node-prefetches 1,633,752,109 node-prefetch-misses
Quoted 3 lines
The percentage of misses is higher for numa/core. I would have expected the performance increase be due to lower "node-misses", but perhaps I am misinterpreting the perf data.
Lack of native THP migration may be enough to explain the performance difference, despite autonuma having better node locality.
Quoted 9 lines
Quoted 2 lines
Next I'll work on making multi-JVM more of an improvement, and I'll also address any incoming regression reports.
I have issues with multiple KVM VMs running either JBB or dbench-in-tmpfs, and I suspect whatever I am seeing is similar to whatever multi-jvm in baremetal is. What I typically see is no real convergence of a single node for resource usage for any of the VMs. For example, when running 8 VMs, 10 vCPUs each, a VM may have the following resource usage:
This is an issue. I have tried understanding the new local/shared and shared task grouping code, but have not wrapped my mind around that code yet.
I will have to look at that code a few more times, and ask more questions of Ingo and Peter (and maybe ask some of the same questions again - I see that some of my comments were addressed in the next version of the patch, but the email never got a reply).
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: numa/core regressions fixed - more testers wanted
Quoted 8 lines · Rik van Riel
Quoted 3 lines · Andrew Theurer
I can confirm single JVM JBB is working well for me. I see a 30% improvement over autoNUMA. What I can't make sense of is some perf stats (taken at 80 warehouses on 4 x WST-EX, 512GB memory):
AutoNUMA does not have native THP migration, that may explain some of the difference.
When I made some fixes to the sched/numa native THP migration, I did also try porting that (with Hannes's memcg fixes) to AutoNUMA.
Here's the patch below: it appeared to be working just fine, but you might find that it doesn't quite apply to whatever tree you're using. I started from 3.6 autonuma28fast in aa.git, but had folded in some of the equally applicable TLB flush optimizations too.
There's also a little "Hack, remove after THP native migration" retuning in mm/huge_memory.c which should probably be removed too.
No signoffs, but it's from work by Peter and Ingo and Hannes and Hugh.
---
include/linux/huge_mm.h | 4 - mm/autonuma.c | 59 +++++----------- mm/huge_memory.c | 140 +++++++++++++++++++++++++++++++++----- mm/internal.h | 5 - mm/memcontrol.c | 7 + mm/memory.c | 4 - mm/migrate.c | 2 7 files changed, 158 insertions(+), 63 deletions(-)
include/linux/huge_mm.h +2 -2
@@ -11,8 +11,8 @@ extern int copy_huge_pmd(struct mm_strucextern int do_huge_pmd_wp_page (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,pmd_t *pmd,pmd_t orig_pmd); -extern int huge_pmd_numa_fixup (struct mm_struct *mm,unsigned long addr, -pmd_t pmd,pmd_t *pmdp); +extern int huge_pmd_numa_fixup (struct mm_struct *mm,struct vm_area_struct *vma, +unsigned long address,pmd_t *pmd,pmd_t orig_pmd);extern pgtable_t get_pmd_huge_pte (struct mm_struct *mm); extern struct page *follow_trans_huge_pmd(struct mm_struct *mm, unsigned long addr,
mm/autonuma.c +22 -37
134 lines not on this page · raw message
mm/huge_memory.c +123 -17
164 lines not on this page · raw message
mm/internal.h +3 -2
@@ -216,11 +216,12 @@ static inline void mlock_migrate_page(st {if (TestClearPageMlocked (page)) {unsigned long flags; +int nr_pages =hpage_nr_pages (page);local_irq_save (flags); -__dec_zone_page_state (page,NR_MLOCK ); +__mod_zone_page_state (page_zone (page),NR_MLOCK , -nr_pages);SetPageMlocked (newpage); -__inc_zone_page_state (newpage,NR_MLOCK ); +__mod_zone_page_state (page_zone (newpage),NR_MLOCK , nr_pages);local_irq_restore (flags); } }
mm/memcontrol.c +5 -2
@@ -3261,15 +3261,18 @@ void mem_cgroup_prepare_migration(struct struct mem_cgroup **memcgp) {struct mem_cgroup *memcg =NULL ; +unsigned int nr_pages =1 ;struct page_cgroup *pc;enum charge_type ctype; *memcgp =NULL ; -VM_BUG_ON (PageTransHuge (page));if (mem_cgroup_disabled ())return ; +if (PageTransHuge (page)) + nr_pages <<=compound_order (page); + pc =lookup_page_cgroup (page);lock_page_cgroup (pc);if (PageCgroupUsed (pc)) {
@@ -3331,7 +3334,7 @@ void mem_cgroup_prepare_migration(struct* charged to the res_counter since we plan on replacing the * old one and only one page is going to be left afterwards. */ -__mem_cgroup_commit_charge (memcg, newpage,1 , ctype, false); +__mem_cgroup_commit_charge (memcg, newpage, nr_pages, ctype, false); }/* remove redundant charge if migration failed*/
mm/memory.c +2 -2
@@ -3546,8 +3546,8 @@ retry:barrier ();if (pmd_trans_huge (orig_pmd)) {if (pmd_numa (*pmd)) -return huge_pmd_numa_fixup (mm, address, - orig_pmd, pmd); +return huge_pmd_numa_fixup (mm, vma, address, + pmd, orig_pmd);if (flags &FAULT_FLAG_WRITE && !pmd_write (orig_pmd) && !pmd_trans_splitting (orig_pmd)) {
mm/migrate.c +1 -1
@@ -407,7 +407,7 @@ int migrate_huge_page_move_mapping(struc*/ void migrate_page_copy (struct page *newpage,struct page *page) { -if (PageHuge (page)) +if (PageHuge (page) ||PageTransHuge (page))copy_huge_page (newpage, page);else copy_highpage (newpage, page);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: numa/core regressions fixed - more testers wanted
Quoted 9 lines · Hugh Dickins
Quoted 8 lines · Rik van Riel
Quoted 3 lines · Andrew Theurer
I can confirm single JVM JBB is working well for me. I see a 30% improvement over autoNUMA. What I can't make sense of is some perf stats (taken at 80 warehouses on 4 x WST-EX, 512GB memory):
AutoNUMA does not have native THP migration, that may explain some of the difference.
Plus, numa/core is sucking the milk of TLB-flsh-optimization from Rik.
BTW, I want to see results of numa/core without such TLB boosts.
Quoted 4 lines
When I made some fixes to the sched/numa native THP migration, I did also try porting that (with Hannes's memcg fixes) to AutoNUMA.
Thanks a ton;)
mm/huge_memory.c out: · line 1096
Quoted 3 lines
mm/huge_memory.c +3
-0+ + new_page =alloc_pages_node (numa_node_id (), + (GFP_TRANSHUGE |GFP_THISNODE ) & ~__GFP_WAIT,HPAGE_PMD_ORDER );
Such a brand new page is selected to be migration target, why?
Hillf
Quoted 2 lines
(quoted diff) +2
-0+ if (!new_page) + goto alloc_fail;
Re: numa/core regressions fixed - more testers wanted
In reply to Hugh DickinsQuoted 13 lines · Hugh Dickins
Quoted 8 lines · Rik van Riel
Quoted 3 lines · Andrew Theurer
I can confirm single JVM JBB is working well for me. I see a 30% improvement over autoNUMA. What I can't make sense of is some perf stats (taken at 80 warehouses on 4 x WST-EX, 512GB memory):
AutoNUMA does not have native THP migration, that may explain some of the difference.
When I made some fixes to the sched/numa native THP migration, I did also try porting that (with Hannes's memcg fixes) to AutoNUMA.
Quoted 7 lines
Here's the patch below: it appeared to be working just fine, but you might find that it doesn't quite apply to whatever tree you're using. I started from 3.6 autonuma28fast in aa.git, but had folded in some of the equally applicable TLB flush optimizations too.
There's also a little "Hack, remove after THP native migration" retuning in
mm/huge_memory.cwhich should probably be removed too.
Thanks, this worked for me. The autoNUMA SPECjbb result is now much closer, just 4% lower than the numa/core result. The number of anon and anon-huge pages are now nearly the same.
-Andrew Theurer
Mel Gorman
reviewerRe: numa/core regressions fixed - more testers wanted
In reply to Andrew TheurerQuoted 45 lines · Andrew Theurer
Quoted 44 lines · Ingo Molnar
Quoted 3 lines · Ingo Molnar
( The 4x JVM regression is still an open bug I think - I'll
re-check and fix that one next, no need to re-report it, I'm on it. )So I tested this on !THP too and the combined numbers are now:
| [ SPECjbb multi-4x8 ] | [ tx/sec ] v3.7 | numa/core-v16 [ higher is better ] ----- | ------------- | +THP: 639k | 655k +2.5% -THP: 510k | 517k +1.3%So it's not a regression anymore, regardless of whether THP is enabled or disabled.
The current updated table of performance results is:
-------------------------------------------------------------------------
[ seconds ] v3.7 AutoNUMA | numa/core-v16 [ vs. v3.7] [ lower is better ] ----- -------- | ------------- ----------- | numa01 340.3 192.3 | 139.4 +144.1% numa01_THREAD_ALLOC 425.1 135.1 | 121.1 +251.0% numa02 56.1 25.3 | 17.5 +220.5% | [ SPECjbb transactions/sec ] | [ higher is better ] | | SPECjbb 1x32 +THP 524k 507k | 638k +21.7% SPECjbb 1x32 !THP 395k | 512k +29.6% | ----------------------------------------------------------------------- | [ SPECjbb multi-4x8 ] | [ tx/sec ] v3.7 | numa/core-v16 [ higher is better ] ----- | ------------- | +THP: 639k | 655k +2.5% -THP: 510k | 517k +1.3%
Quoted 8 lines
Quoted 3 lines
So I think I've addressed all regressions reported so far - if anyone can still see something odd, please let me know so I can reproduce and fix it ASAP.
I can confirm single JVM JBB is working well for me. I see a 30% improvement over autoNUMA. What I can't make sense of is some perf stats (taken at 80 warehouses on 4 x WST-EX, 512GB memory):
I'm curious about possible effects with profiling. Can you rerun just this test without any profiling and see if the gain is the same? My own tests are running monitors but they only fire every 10 seconds and are not running profiles.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: numa/core regressions fixed - more testers wanted
Quoted 55 lines · Mel Gorman
Quoted 45 lines · Andrew Theurer
Quoted 44 lines · Ingo Molnar
Quoted 3 lines · Ingo Molnar
( The 4x JVM regression is still an open bug I think - I'll
re-check and fix that one next, no need to re-report it, I'm on it. )So I tested this on !THP too and the combined numbers are now:
| [ SPECjbb multi-4x8 ] | [ tx/sec ] v3.7 | numa/core-v16 [ higher is better ] ----- | ------------- | +THP: 639k | 655k +2.5% -THP: 510k | 517k +1.3%So it's not a regression anymore, regardless of whether THP is enabled or disabled.
The current updated table of performance results is:
-------------------------------------------------------------------------
[ seconds ] v3.7 AutoNUMA | numa/core-v16 [ vs. v3.7] [ lower is better ] ----- -------- | ------------- ----------- | numa01 340.3 192.3 | 139.4 +144.1% numa01_THREAD_ALLOC 425.1 135.1 | 121.1 +251.0% numa02 56.1 25.3 | 17.5 +220.5% | [ SPECjbb transactions/sec ] | [ higher is better ] | | SPECjbb 1x32 +THP 524k 507k | 638k +21.7% SPECjbb 1x32 !THP 395k | 512k +29.6% | ----------------------------------------------------------------------- | [ SPECjbb multi-4x8 ] | [ tx/sec ] v3.7 numa/core-v16 [ higher is better ] ----- | ------------- | +THP: 639k | 655k +2.5% -THP: 510k | 517k +1.3%Quoted 8 lines
Quoted 3 lines
So I think I've addressed all regressions reported so far - if anyone can still see something odd, please let me know so I can reproduce and fix it ASAP.
I can confirm single JVM JBB is working well for me. I see a 30% improvement over autoNUMA. What I can't make sense of is some perf stats (taken at 80 warehouses on 4 x WST-EX, 512GB memory):
Quoted 4 lines
I'm curious about possible effects with profiling. Can you rerun just this test without any profiling and see if the gain is the same? My own tests are running monitors but they only fire every 10 seconds and are not running profiles.
After using the patch Hugh provided, I did make a 2nd run, this time with no profiling at all, and the run was 2% higher. Not sure if this is due to profiling gone, or just run to run variance, but nevertheless a pretty low difference.
-Andrew Theurer
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: numa/core regressions fixed - more testers wanted
In reply to Ingo MolnarQuoted 25 lines · Ingo Molnar
The current updated table of performance results is:
-------------------------------------------------------------------------
[ seconds ] v3.7 AutoNUMA | numa/core-v16 [ vs. v3.7] [ lower is better ] ----- -------- | ------------- ----------- | numa01 340.3 192.3 | 139.4 +144.1% numa01_THREAD_ALLOC 425.1 135.1 | 121.1 +251.0% numa02 56.1 25.3 | 17.5 +220.5% | [ SPECjbb transactions/sec ] | [ higher is better ] | | SPECjbb 1x32 +THP 524k 507k | 638k +21.7% SPECjbb 1x32 !THP 395k | 512k +29.6% | ----------------------------------------------------------------------- | [ SPECjbb multi-4x8 ] | [ tx/sec ] v3.7 | numa/core-v16 [ higher is better ] ----- | ------------- | +THP: 639k | 655k +2.5% -THP: 510k | 517k +1.3%
Quoted 4 lines
So I think I've addressed all regressions reported so far - if anyone can still see something odd, please let me know so I can reproduce and fix it ASAP.
I started profiling on a new machine that is an exact duplicate of the 16-way, 4 node, 32GB machine I was profiling with earlier to rule out any machine-specific problems. I pulled master and ran new comparisons with THP enabled at c418de93e398 ("Merge branch 'x86/mm'"):
CONFIG_NUMA_BALANCING disabled 136521.55 SPECjbb2005 bops
CONFIG_NUMA_BALANCING enabled 132476.07 SPECjbb2005 bops (-3.0%)
Aside: neither 4739578c3ab3 ("x86/mm: Don't flush the TLB on #WP pmd fixups") nor 01e9c2441eee ("x86/vsyscall: Add Kconfig option to use native vsyscalls and switch to it") significantly improved upon the throughput on this system.
Over the past 24 hours, however, throughput has significantly improved from a 6.3% regression to a 3.0% regression because of 246c0b9a1caf ("mm,
numa: Turn 4K pte NUMA faults into effective hugepage ones")!
One request: I noticed that the entire patchset doesn't add any fields to /proc/vmstat through count_vm_event() like thp did, which I found very useful when profiling that set when it was being reviewed. Would it be possible to add some vm events to the balancing code so we can capture data of how the NUMA balancing is behaving? Their usefulness would extend beyond just the review period.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: numa/core regressions fixed - more testers wanted
Quoted 6 lines · David Rientjes
I started profiling on a new machine that is an exact duplicate of the 16-way, 4 node, 32GB machine I was profiling with earlier to rule out any machine-specific problems. I pulled master and ran new comparisons with THP enabled at
c418de93e398("Merge branch 'x86/mm'"):
Quoted 7 lines
CONFIG_NUMA_BALANCINGdisabled 136521.55 SPECjbb2005 bops
CONFIG_NUMA_BALANCINGenabled 132476.07 SPECjbb2005 bops (-3.0%)Aside: neither
4739578c3ab3("x86/mm: Don't flush the TLB on #WP pmd fixups") nor01e9c2441eee("x86/vsyscall: AddKconfigoption to use native vsyscalls and switch to it") significantly improved upon the throughput on this system.
Could you please send an updated profile done with latest -tip? The last profile I got from you still had the vsyscall emulation page faults in it.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: numa/core regressions fixed - more testers wanted
In reply to David RientjesQuoted 2 lines · David Rientjes
Over the past 24 hours, however, throughput has significantly improved from a 6.3% regression to a 3.0% regression [...]
It's still a regression though, and I'd like to figure out the root cause of that. An updated full profile from tip:master [which has all the latest fixes applied] would be helpful as a starting point.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: numa/core regressions fixed - more testers wanted
In reply to Ingo MolnarQuoted 6 lines
Those of you who would like to test all the latest patches are welcome to pick up latest bits at tip:master:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitmaster
I am wondering if it is a problem, but it still exists on HEAD: c418de93e39891
http://article.gmane.org/gmane.linux.kernel.mm/90131/match=compiled+with+name+pl+and+start+it+on+my
like when just start 4 pl tasks, often 3 were running on node 0, and 1 was running on node 1.
The old balance will average assign tasks to different node, different core.
Regards
Alex
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: numa/core regressions fixed - more testers wanted
Quoted 7 lines · Alex Shi
Quoted 6 lines
Those of you who would like to test all the latest patches are welcome to pick up latest bits at tip:master:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitmaster
Quoted 6 lines
I am wondering if it is a problem, but it still exists on HEAD:
c418de93e39891
http://article.gmane.org/gmane.linux.kernel.mm/90131/match=compiled+with+name+pl+and+start+it+on+mylike when just start 4 pl tasks, often 3 were running on node 0, and 1 was running on node 1. The old balance will average assign tasks to different node, different core.
This is "normal" in the sense that the current mainline scheduler is (supposed to be) doing something similar: if the node is still within capacity, then there's no reason to move those threads.
OTOH, I think with NUMA balancing we indeed want to spread them better, if those tasks do not share memory with each other but use their own memory. If they share memory then they should remain on the same node if possible.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: numa/core regressions fixed - more testers wanted
Quoted 21 lines · Ingo Molnar
Quoted 7 lines · Alex Shi
Quoted 6 lines
Those of you who would like to test all the latest patches are welcome to pick up latest bits at tip:master:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitmasterQuoted 6 lines
I am wondering if it is a problem, but it still exists on HEAD:
c418de93e39891
http://article.gmane.org/gmane.linux.kernel.mm/90131/match=compiled+with+name+pl+and+start+it+on+mylike when just start 4 pl tasks, often 3 were running on node 0, and 1 was running on node 1. The old balance will average assign tasks to different node, different core.
This is "normal" in the sense that the current mainline scheduler is (supposed to be) doing something similar: if the node is still within capacity, then there's no reason to move those threads.
Quoted 4 lines
OTOH, I think with NUMA balancing we indeed want to spread them better, if those tasks do not share memory with each other but use their own memory. If they share memory then they should remain on the same node if possible.
Could you please check tip:master with -v17:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git master
?
It should place your workload better than v16 did.
Note, you might be able to find other combinations of tasks that are not scheduled NUMA-perfectly yet, as task group placement is not exhaustive yet.
You might want to check which combination looks the weirdest to you and report it, so I can fix any remaining placement inefficiencies in order of importance.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: numa/core regressions fixed - more testers wanted
Quoted 24 lines · Ingo Molnar
Quoted 21 lines · Ingo Molnar
Quoted 7 lines · Alex Shi
Quoted 6 lines
Those of you who would like to test all the latest patches are welcome to pick up latest bits at tip:master:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitmasterQuoted 6 lines
I am wondering if it is a problem, but it still exists on HEAD:
c418de93e39891
http://article.gmane.org/gmane.linux.kernel.mm/90131/match=compiled+with+name+pl+and+start+it+on+mylike when just start 4 pl tasks, often 3 were running on node 0, and 1 was running on node 1. The old balance will average assign tasks to different node, different core.
This is "normal" in the sense that the current mainline scheduler is (supposed to be) doing something similar: if the node is still within capacity, then there's no reason to move those threads.
Quoted 4 lines
Quoted 4 lines
OTOH, I think with NUMA balancing we indeed want to spread them better, if those tasks do not share memory with each other but use their own memory. If they share memory then they should remain on the same node if possible.
There is no share memory between them.
Quoted 8 lines
Could you please check tip:master with -v17:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitmaster?
It should place your workload better than v16 did.
OK. will try it on next Monday, if it is not late to you.
Quoted 4 lines
Note, you might be able to find other combinations of tasks that are not scheduled NUMA-perfectly yet, as task group placement is not exhaustive yet.
I am not familiar with task group. but anyway, will try it too.
Quoted 4 lines
You might want to check which combination looks the weirdest to you and report it, so I can fix any remaining placement inefficiencies in order of importance.
Any suggestions of combination?
Quoted 4 lines
Thanks,
Ingo
Signature · 8 lines
Thanks
Alex
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>Re: numa/core regressions fixed - more testers wanted
In reply to Ingo MolnarQuoted 24 lines · Ingo Molnar
Quoted 21 lines · Ingo Molnar
Quoted 7 lines · Alex Shi
Quoted 6 lines
Those of you who would like to test all the latest patches are welcome to pick up latest bits at tip:master:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitmasterQuoted 6 lines
I am wondering if it is a problem, but it still exists on HEAD:
c418de93e39891
http://article.gmane.org/gmane.linux.kernel.mm/90131/match=compiled+with+name+pl+and+start+it+on+mylike when just start 4 pl tasks, often 3 were running on node 0, and 1 was running on node 1. The old balance will average assign tasks to different node, different core.
This is "normal" in the sense that the current mainline scheduler is (supposed to be) doing something similar: if the node is still within capacity, then there's no reason to move those threads.
Quoted 4 lines
Quoted 4 lines
OTOH, I think with NUMA balancing we indeed want to spread them better, if those tasks do not share memory with each other but use their own memory. If they share memory then they should remain on the same node if possible.
I rewrite the little test case by assemble:
==
.text
.global _start
_start:
do_nop:
nop
nop
jmp do_nop
==
It reproduced the problem on latest tip/master, HEAD: 7cb989d0159a6f43104992f18 like for 4 above tasks running, 3 of them running on node 0, one running on node 1.
If kernel can detect the LLC of CPU is allowed for tasks aggregate, it's a nice feature. if not, the aggregate may cause more cache missing.
Quoted 20 lines
Could you please check tip:master with -v17:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitmaster?
It should place your workload better than v16 did.
Note, you might be able to find other combinations of tasks that are not scheduled NUMA-perfectly yet, as task group placement is not exhaustive yet.
You might want to check which combination looks the weirdest to you and report it, so I can fix any remaining placement inefficiencies in order of importance.
Thanks,
Ingo
Signature · 8 lines
Thanks
Alex
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>Re: numa/core regressions fixed - more testers wanted
Quoted 6 lines
Quoted 6 lines
Could you please check tip:master with -v17:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitmaster
Tested this version on our SNB EP 2 sockets box, 8 cores * HT with specjbb2005 on jrockit.
With single JVM setting it has 40% performance increase compare to 3.7-rc6. impressive!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
btw., mind sending me a fuller/longer profile than the one you've sent before? In particular does your system have any vsyscall emulation page fault overhead?
If yes, does the patch below change anything for you?
Thanks,
Ingo
---------------->
Subject: x86/vsyscall: Add Kconfig option to use native vsyscalls, switch to it
From: Ingo Molnar <[email protected]>
Apparently there's still plenty of systems out there triggering the vsyscall emulation page faults - causing hard to track down performance regressions on page fault intense workloads...
Some people seem to have run into that with threading-intense Java workloads.
So until there's a better solution to this, add a Kconfig switch to make the vsyscall mode configurable and turn native vsyscall support back on by default.
Distributions whose libraries and applications use the vDSO and never access the vsyscall page can change the config option to off.
Note, I don't think we want to expose the "none" option via a Kconfig switch - it breaks the ABI. So it's "native" versus "emulate", with "none" being available as a kernel boot option, for the super paranoid.
For more background, see these upstream commits:
3ae36655b97a x86-64: Rework vsyscall emulation and add vsyscall= parameter 5cec93c216db x86-64: Emulate legacy vsyscalls
--- arch/x86/Kconfig | 21 +++++++++++++++++++++ arch/x86/kernel/vsyscall_64.c | 8 +++++++- 2 files changed, 28 insertions(+), 1 deletion(-)
arch/x86/Kconfig +21 -0
@@ -2234,6 +2234,27 @@ config X86_X32 elf32_x86_64 support enabled to compile a kernel with this option set. +config X86_VSYSCALL_COMPAT + bool "vsyscall compatibility" + default y + help + vsyscalls, as global executable pages, can be a security hole + escallation helper by exposing an easy shell code target with + a predictable address. + + Many versions of glibc rely on the vsyscall page though, so it + cannot be eliminated unconditionally. If you disable this + option these systems will still work but might incur the overhead + of vsyscall emulation page faults. + + The vsyscall=none, vsyscall=emulate, vsyscall=native kernel boot + option can be used to override this mode as well. + + Keeping this option enabled leaves the vsyscall page enabled, + i.e. vsyscall=native. Disabling this option means vsyscall=emulate. + + If unsure, say Y. + config COMPAT def_bool y depends on IA32_EMULATION || X86_X32
arch/x86/kernel/vsyscall_64.c +7 -1
@@ -56,7 +56,13 @@DEFINE_VVAR (int, vgetcpu_mode); DEFINE_VVAR(struct vsyscall_gtod_data, vsyscall_gtod_data); -static enum {EMULATE ,NATIVE ,NONE } vsyscall_mode =EMULATE ; +static enum { EMULATE, NATIVE, NONE } vsyscall_mode = +#ifdef CONFIG_X86_VSYSCALL_COMPAT +NATIVE +#else + EMULATE +#endif +;static int __init vsyscall_setup (char *str) {
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 5 lines · Ingo Molnar
btw., mind sending me a fuller/longer profile than the one you've sent before? In particular does your system have any vsyscall emulation page fault overhead?
I can't, the results for specjbb got trashed after I moved to 3.7-rc6 and the old run was incomplete :( I'll have to re-run for profiles. I generally do not run with profiling enabled because it can distort results very badly.
I went back and re-examined anything else that could distort the results. I have monitoring other than profiles enabled and the heaviest monitor by far is reading smaps every 10 seconds to see how processes are currently distributed between nodes and what CPU they are running on. The intention was to be able to examine after the fact if the individual java processes were ending up on the same nodes and scheduled properly.
However, this would impact the peak performance and affect contention on mmap_sem every 10 seconds as it's possible for the PTE scanner and smaps reader to contend. Care is taken to only read smaps once per process. So for each JVM instance, it would read smaps once even though it would examine where every thread is running. This minimises the distortion. top is also updating every 10 seconds which will also contend on locks. The other monitors are relatively harmless and are reading files like /proc/vmstat, each numastat file every 10 seconds. As before, all tests had these monitors enabled so they would all have suffered evenly.
Normally when I'm looking at a series I run both with and without monitors. I report without monitors and use monitors to help debug and problems that are identified. Unfortunately, time pressure is not allowing me to do that this time.
Quoted 2 lines
If yes, does the patch below change anything for you?
I'll check it.
Thanks.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 11 lines · Ingo Molnar
Quoted 6 lines
I was not able to run a full sets of tests today as I was distracted so all I have is a multi JVM comparison. I'll keep it shorter than average
3.7.0 3.7.0 rc5-stats-v4r2 rc5-schednuma-v16r1Thanks for the testing - I'll wait for your full results to see whether the other regressions you reported before are fixed/improved.
Here are the latest figures I have available. It includes figures from "Automatic NUMA Balancing V4" which I just released. Very short summary is as follows
Even without a proper placement policy, balancenuma does fairly well in a number of tests, shows a number of improvements in places and for the most part it does not regress against mainline. It does this without a decent placement policy on top and I expect a placement policy would only make it better. Its System CPU usage is still of concern but with proper feedback from a placement policy it could reduce the PTE scan rate and keep it down.
schednuma has improved a lot, particularly in terms in system CPU usage. However, even with THP enabled it is showing regressions for specjbb and a noticable regression when just building kernels. There have been follow-on patches since testing started and maybe they'll make a difference.
Now, the long report... the very long report. The full sets tests are still not complete but it should be enough to go with for now. A number of kernels are compared. All are using 3.7-rc6 are the base
stats-v4r12 This is patches 10 from "Automatic NUMA Balancing V4" and
is just the TLB fixes and a few minor stats patches for migration
schednuma-v16r2 tip/sched/core + the original series "Latest numa/core
release, v16". None of the follow up patches have been applied because testing started after these were posted.
autonuma-v28fastr3 is the autonuma-v28fast branch from Andrea's tree rebased
to v3.7-rc6
moron-v4r38 is patches 1-19 from "Automatic NUMA Balancing V4" and is
the most basic available policy
twostage-v4r38 is patches 1-36 from "Automatic NUMA Balancing V4" and includes
PMD fault handling, migration backoff if there is too much migration, the most rudimentary of scan rate adapation and a two-stage filter to mitigate ping-pong effects
thpmigrate-v4r38 is patches 1-37 from "Autonumatic NUMA Balancing". Patch 37
adds native THP migration so its effect can be observed
In all cases, tests were run via mmtests. Monitors were enabled but not profiling as profiling can distort results a lot. The monitors fire every 10 seconds and the heaviest reads numa_maps. THP is generally enabled but the vmstats from each test is usually an obvious indicator.
There is a very important point to note about specjbb. specjbb itself reports a single throughput figure and it bases this on a number of warehouses around the expected peak. It ignores warehouses outside this window which can be misleading. I'm reporting on all warehouses so if you find that my figures do not match what specjbb tells you, it could be because I'm reporting on low warehouse counts or counts outside the window when the peak performance as reported by specjbb was great.
First, the autonumabenchmark.
AUTONUMA BENCH
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12 rc6-schednuma-v16r2rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38 rc6-thpmigrate-v4r38
User NUMA01 75014.43 ( 0.00%) 22510.33 ( 69.99%) 32944.96 ( 56.08%) 25431.57 ( 66.10%) 69422.58 ( 7.45%) 47343.91 ( 36.89%)
User NUMA01_THEADLOCAL 55479.76 ( 0.00%) 18960.30 ( 65.82%) 16960.54 ( 69.43%) 20381.77 ( 63.26%) 23673.65 ( 57.33%) 16862.01 ( 69.61%)
User NUMA02 6801.32 ( 0.00%) 2208.32 ( 67.53%) 1921.66 ( 71.75%) 2979.36 ( 56.19%) 2213.03 ( 67.46%) 2053.89 ( 69.80%)
User NUMA02_SMT 2973.96 ( 0.00%) 1011.45 ( 65.99%) 1018.84 ( 65.74%) 1135.76 ( 61.81%) 912.61 ( 69.31%) 989.03 ( 66.74%)
System NUMA01 47.87 ( 0.00%) 140.01 (-192.48%) 286.39 (-498.27%) 743.09 (-1452.31%) 896.21 (-1772.17%) 489.09 (-921.70%)
System NUMA01_THEADLOCAL 43.52 ( 0.00%) 1014.35 (-2230.77%) 172.10 (-295.45%) 475.68 (-993.01%) 593.89 (-1264.64%) 144.30 (-231.57%)
System NUMA02 1.94 ( 0.00%) 36.90 (-1802.06%) 20.06 (-934.02%) 22.86 (-1078.35%) 43.01 (-2117.01%) 9.28 (-378.35%)
System NUMA02_SMT 0.93 ( 0.00%) 11.42 (-1127.96%) 11.68 (-1155.91%) 11.87 (-1176.34%) 31.31 (-3266.67%) 3.61 (-288.17%)
Elapsed NUMA01 1668.03 ( 0.00%) 486.04 ( 70.86%) 794.10 ( 52.39%) 601.19 ( 63.96%) 1575.52 ( 5.55%) 1066.67 ( 36.05%)
Elapsed NUMA01_THEADLOCAL 1266.49 ( 0.00%) 433.14 ( 65.80%) 412.50 ( 67.43%) 514.30 ( 59.39%) 542.26 ( 57.18%) 369.38 ( 70.83%)
Elapsed NUMA02 175.75 ( 0.00%) 53.15 ( 69.76%) 63.25 ( 64.01%) 84.51 ( 51.91%) 68.64 ( 60.94%) 49.42 ( 71.88%)
Elapsed NUMA02_SMT 163.55 ( 0.00%) 50.54 ( 69.10%) 56.75 ( 65.30%) 68.85 ( 57.90%) 59.85 ( 63.41%) 46.21 ( 71.75%)
CPU NUMA01 4500.00 ( 0.00%) 4660.00 ( -3.56%) 4184.00 ( 7.02%) 4353.00 ( 3.27%) 4463.00 ( 0.82%) 4484.00 ( 0.36%)
CPU NUMA01_THEADLOCAL 4384.00 ( 0.00%) 4611.00 ( -5.18%) 4153.00 ( 5.27%) 4055.00 ( 7.50%) 4475.00 ( -2.08%) 4603.00 ( -5.00%)
CPU NUMA02 3870.00 ( 0.00%) 4224.00 ( -9.15%) 3069.00 ( 20.70%) 3552.00 ( 8.22%) 3286.00 ( 15.09%) 4174.00 ( -7.86%)
CPU NUMA02_SMT 1818.00 ( 0.00%) 2023.00 (-11.28%) 1815.00 ( 0.17%) 1666.00 ( 8.36%) 1577.00 ( 13.26%) 2147.00 (-18.10%)In all cases, the baseline kernel is beaten in terms of elapsed time.
NUMA01 schednuma bestNUMA01_THREADLOCAL balancenuma best (required THP migration)
NUMA02 balancenuma best (required THP migration)NUMA02_SMT balancenuma best (required THP migration)
Note that even without a placement policy, balancenuma was still quite good but that it required native THP migration to do that. Not depending on THP to avoid regressions is important but it reinforces my point that THP migration was introduced too early in schednuma and potentially hid problems in the underlying mechanics.
System CPU usage -- schednuma has improved *dramatically* in this regard for this test.
NUMA01 schednuma lowest overheadNUMA01_THREADLOCAL balancenuma lowest overhead (THP again)
NUMA02 balancenuma lowest overhead (THP again)NUMA02_SMT balancenuma lowest overhead (THP again)
Again, balancenuma had the lowest overhead. Note that much of this was due to native THP migration. That patch was implemented in a hurry so it will need close scrutiny to make sure I'm not cheating in there somewhere.
MMTests Statistics: duration
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38rc6-thpmigrate-v4r38
User 140276.51 44697.54 52853.24 49933.77 96228.41 67256.37
System 94.93 1203.53 490.84 1254.00 1565.05 646.94
Elapsed 3284.21 1033.02 1336.01 1276.08 2255.08 1542.05schednuma completed the fast overall because it completely kicked ass at numa01. It's system CPU usage was apparently high but much of that was incurred in just NUMA01_THREADLOCAL.
MMTests Statistics: vmstat
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38rc6-thpmigrate-v4r38
Page Ins 43580 43444 43416 39176 43604 44184
Page Outs 30944 11504 14012 13332 20576 15944
Swap Ins 0 0 0 0 0 0
Swap Outs 0 0 0 0 0 0
Direct pages scanned 0 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0 0Kswapd efficiency 100% 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0 0 Page writes file 0 0 0 0 0 0 Page writes anon 0 0 0 0 0 0 Page reclaim immediate 0 0 0 0 0 0 Page rescued immediate 0 0 0 0 0 0 Slabs scanned 0 0 0 0 0 0 Direct inode steals 0 0 0 0 0 0 Kswapd inode steals 0 0 0 0 0 0 Kswapd skipped wait 0 0 0 0 0 0 THP fault alloc 17076 13240 19254 17165 16207 17298 THP collapse alloc 7 0 8950 534 1020 8 THP splits 3 2 9486 7585 7426 2 THP fault fallback 0 0 0 0 0 0 THP collapse fail 0 0 0 0 0 0 Compaction stalls 0 0 0 0 0 0 Compaction success 0 0 0 0 0 0 Compaction failures 0 0 0 0 0 0 Page migrate success 0 0 0 2988728 8265970 14679 Page migrate failure 0 0 0 0 0 0 Compaction pages isolated 0 0 0 0 0 0 Compaction migrate scanned 0 0 0 0 0 0 Compaction free scanned 0 0 0 0 0 0 Compaction cost 0 0 0 3102 8580 15 NUMA PTE updates 0 0 0 712623229 221698496 458517124 NUMA hint faults 0 0 0 604878754 105489260 1431976 NUMA hint local faults 0 0 0 163366888 48972507 621116 NUMA pages migrated 0 0 0 2988728 8265970 14679 AutoNUMA cost 0 0 0 3029438 529155 10369
So I don't have detailed stats for schednuma or autonuma so I don't know how many PTE updates it's doing. However, look at the "THP collapse alloc" and "THP splits". You can see the effect of native THP migration. schednuma and thpmigrate both have few collapses and splits due to the native migration.
Also note what thpmigrate does to "Page migrate success" as each THP migration only counts as 1. I don't have the same stats for schednuma but one would expect they would be similar if they existed.
SPECJBB BOPS Multiple JVMs, THP is DISABLED
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12 rc6-schednuma-v16r2rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38 rc6-thpmigrate-v4r38
Mean 1 25426.00 ( 0.00%) 17734.25 (-30.25%) 25828.25 ( 1.58%) 24972.75 ( -1.78%) 24944.25 ( -1.89%) 24557.25 ( -3.42%)
Mean 2 53316.50 ( 0.00%) 39883.50 (-25.19%) 56303.00 ( 5.60%) 51994.00 ( -2.48%) 51962.75 ( -2.54%) 49828.50 ( -6.54%)
Mean 3 77182.75 ( 0.00%) 58082.50 (-24.75%) 82874.00 ( 7.37%) 76428.50 ( -0.98%) 74272.75 ( -3.77%) 73934.50 ( -4.21%)
Mean 4 100698.25 ( 0.00%) 75740.25 (-24.78%) 107776.00 ( 7.03%) 98963.75 ( -1.72%) 96681.00 ( -3.99%) 95749.75 ( -4.91%)
Mean 5 120235.50 ( 0.00%) 87472.25 (-27.25%) 131299.75 ( 9.20%) 118226.50 ( -1.67%) 115981.25 ( -3.54%) 115904.50 ( -3.60%)
Mean 6 135085.00 ( 0.00%) 100947.25 (-25.27%) 152928.75 ( 13.21%) 133681.50 ( -1.04%) 134297.00 ( -0.58%) 133065.50 ( -1.49%)
Mean 7 135916.25 ( 0.00%) 112033.50 (-17.57%) 158917.50 ( 16.92%) 135273.25 ( -0.47%) 135100.50 ( -0.60%) 135286.00 ( -0.46%)
Mean 8 131696.25 ( 0.00%) 114805.25 (-12.83%) 160972.00 ( 22.23%) 126948.50 ( -3.61%) 135756.00 ( 3.08%) 135097.25 ( 2.58%)
Mean 9 129359.00 ( 0.00%) 113961.25 (-11.90%) 161584.00 ( 24.91%) 129655.75 ( 0.23%) 133621.50 ( 3.30%) 133027.00 ( 2.84%)
Mean 10 121682.75 ( 0.00%) 114095.25 ( -6.24%) 159302.75 ( 30.92%) 119806.00 ( -1.54%) 127338.50 ( 4.65%) 128388.50 ( 5.51%)
Mean 11 114355.25 ( 0.00%) 112794.25 ( -1.37%) 154468.75 ( 35.08%) 114229.75 ( -0.11%) 121907.00 ( 6.60%) 125957.00 ( 10.15%)
Mean 12 109110.00 ( 0.00%) 110618.00 ( 1.38%) 149917.50 ( 37.40%) 106851.00 ( -2.07%) 121331.50 ( 11.20%) 122557.25 ( 12.32%)
Mean 13 106055.00 ( 0.00%) 109073.25 ( 2.85%) 146731.75 ( 38.35%) 105273.75 ( -0.74%) 118965.25 ( 12.17%) 121129.25 ( 14.21%)
Mean 14 105102.25 ( 0.00%) 107065.00 ( 1.87%) 143996.50 ( 37.01%) 103972.00 ( -1.08%) 118018.50 ( 12.29%) 120379.50 ( 14.54%)
Mean 15 105070.00 ( 0.00%) 104714.50 ( -0.34%) 142079.50 ( 35.22%) 102753.50 ( -2.20%) 115214.50 ( 9.65%) 114074.25 ( 8.57%)
Mean 16 101610.50 ( 0.00%) 103741.25 ( 2.10%) 140463.75 ( 38.24%) 103084.75 ( 1.45%) 115000.25 ( 13.18%) 112132.75 ( 10.36%)
Mean 17 99653.00 ( 0.00%) 101577.25 ( 1.93%) 137886.50 ( 38.37%) 101658.00 ( 2.01%) 116072.25 ( 16.48%) 114797.75 ( 15.20%)
Mean 18 99804.25 ( 0.00%) 99625.75 ( -0.18%) 136973.00 ( 37.24%) 101557.25 ( 1.76%) 113653.75 ( 13.88%) 112361.00 ( 12.58%)
Stddev 1 956.30 ( 0.00%) 696.13 ( 27.21%) 729.45 ( 23.72%) 692.14 ( 27.62%) 344.73 ( 63.95%) 620.60 ( 35.10%)
Stddev 2 1105.71 ( 0.00%) 1219.79 (-10.32%) 819.00 ( 25.93%) 497.85 ( 54.97%) 1571.77 (-42.15%) 1584.30 (-43.28%)
Stddev 3 782.85 ( 0.00%) 1293.42 (-65.22%) 1016.53 (-29.85%) 777.41 ( 0.69%) 559.90 ( 28.48%) 1451.35 (-85.39%)
Stddev 4 1583.94 ( 0.00%) 1266.70 ( 20.03%) 1418.75 ( 10.43%) 1117.71 ( 29.43%) 879.59 ( 44.47%) 3081.68 (-94.56%)
Stddev 5 1361.30 ( 0.00%) 2958.17 (-117.31%) 1254.51 ( 7.84%) 1085.07 ( 20.29%) 821.75 ( 39.63%) 1971.46 (-44.82%)
Stddev 6 980.46 ( 0.00%) 2401.48 (-144.93%) 1693.67 (-72.74%) 865.73 ( 11.70%) 995.95 ( -1.58%) 1484.04 (-51.36%)
Stddev 7 1596.69 ( 0.00%) 1152.52 ( 27.82%) 1278.42 ( 19.93%) 2125.55 (-33.12%) 780.03 ( 51.15%) 7738.34 (-384.65%)
Stddev 8 5335.38 ( 0.00%) 2228.09 ( 58.24%) 720.44 ( 86.50%) 1425.78 ( 73.28%) 4981.34 ( 6.64%) 3015.77 ( 43.48%)
Stddev 9 2644.97 ( 0.00%) 2559.52 ( 3.23%) 1676.05 ( 36.63%) 6018.44 (-127.54%) 4856.12 (-83.60%) 2224.33 ( 15.90%)
Stddev 10 2887.45 ( 0.00%) 2237.65 ( 22.50%) 2592.28 ( 10.22%) 4871.48 (-68.71%) 3211.83 (-11.23%) 2934.03 ( -1.61%)
Stddev 11 4397.53 ( 0.00%) 1507.18 ( 65.73%) 5111.36 (-16.23%) 2741.08 ( 37.67%) 2954.59 ( 32.81%) 2812.71 ( 36.04%)
Stddev 12 4591.96 ( 0.00%) 313.48 ( 93.17%) 9008.19 (-96.17%) 3077.80 ( 32.97%) 888.55 ( 80.65%) 1665.82 ( 63.72%)
Stddev 13 3949.88 ( 0.00%) 743.20 ( 81.18%) 9978.16 (-152.62%) 2622.11 ( 33.62%) 1869.85 ( 52.66%) 1048.64 ( 73.45%)
Stddev 14 3727.46 ( 0.00%) 462.24 ( 87.60%) 9933.35 (-166.49%) 2702.25 ( 27.50%) 1596.33 ( 57.17%) 1276.03 ( 65.77%)
Stddev 15 2034.89 ( 0.00%) 490.28 ( 75.91%) 8688.84 (-326.99%) 2309.97 (-13.52%) 1212.53 ( 40.41%) 2088.72 ( -2.65%)
Stddev 16 3979.74 ( 0.00%) 648.50 ( 83.70%) 9606.85 (-141.39%) 2284.15 ( 42.61%) 1769.97 ( 55.53%) 2083.18 ( 47.66%)
Stddev 17 3619.30 ( 0.00%) 415.80 ( 88.51%) 9636.97 (-166.27%) 2838.78 ( 21.57%) 1034.92 ( 71.41%) 760.91 ( 78.98%)
Stddev 18 3276.41 ( 0.00%) 238.77 ( 92.71%) 11295.37 (-244.75%) 1061.62 ( 67.60%) 589.37 ( 82.01%) 881.04 ( 73.11%)
TPut 1 101704.00 ( 0.00%) 70937.00 (-30.25%) 103313.00 ( 1.58%) 99891.00 ( -1.78%) 99777.00 ( -1.89%) 98229.00 ( -3.42%)
TPut 2 213266.00 ( 0.00%) 159534.00 (-25.19%) 225212.00 ( 5.60%) 207976.00 ( -2.48%) 207851.00 ( -2.54%) 199314.00 ( -6.54%)
TPut 3 308731.00 ( 0.00%) 232330.00 (-24.75%) 331496.00 ( 7.37%) 305714.00 ( -0.98%) 297091.00 ( -3.77%) 295738.00 ( -4.21%)
TPut 4 402793.00 ( 0.00%) 302961.00 (-24.78%) 431104.00 ( 7.03%) 395855.00 ( -1.72%) 386724.00 ( -3.99%) 382999.00 ( -4.91%)
TPut 5 480942.00 ( 0.00%) 349889.00 (-27.25%) 525199.00 ( 9.20%) 472906.00 ( -1.67%) 463925.00 ( -3.54%) 463618.00 ( -3.60%)
TPut 6 540340.00 ( 0.00%) 403789.00 (-25.27%) 611715.00 ( 13.21%) 534726.00 ( -1.04%) 537188.00 ( -0.58%) 532262.00 ( -1.49%)
TPut 7 543665.00 ( 0.00%) 448134.00 (-17.57%) 635670.00 ( 16.92%) 541093.00 ( -0.47%) 540402.00 ( -0.60%) 541144.00 ( -0.46%)
TPut 8 526785.00 ( 0.00%) 459221.00 (-12.83%) 643888.00 ( 22.23%) 507794.00 ( -3.61%) 543024.00 ( 3.08%) 540389.00 ( 2.58%)
TPut 9 517436.00 ( 0.00%) 455845.00 (-11.90%) 646336.00 ( 24.91%) 518623.00 ( 0.23%) 534486.00 ( 3.30%) 532108.00 ( 2.84%)
TPut 10 486731.00 ( 0.00%) 456381.00 ( -6.24%) 637211.00 ( 30.92%) 479224.00 ( -1.54%) 509354.00 ( 4.65%) 513554.00 ( 5.51%)
TPut 11 457421.00 ( 0.00%) 451177.00 ( -1.37%) 617875.00 ( 35.08%) 456919.00 ( -0.11%) 487628.00 ( 6.60%) 503828.00 ( 10.15%)
TPut 12 436440.00 ( 0.00%) 442472.00 ( 1.38%) 599670.00 ( 37.40%) 427404.00 ( -2.07%) 485326.00 ( 11.20%) 490229.00 ( 12.32%)
TPut 13 424220.00 ( 0.00%) 436293.00 ( 2.85%) 586927.00 ( 38.35%) 421095.00 ( -0.74%) 475861.00 ( 12.17%) 484517.00 ( 14.21%)
TPut 14 420409.00 ( 0.00%) 428260.00 ( 1.87%) 575986.00 ( 37.01%) 415888.00 ( -1.08%) 472074.00 ( 12.29%) 481518.00 ( 14.54%)
TPut 15 420280.00 ( 0.00%) 418858.00 ( -0.34%) 568318.00 ( 35.22%) 411014.00 ( -2.20%) 460858.00 ( 9.65%) 456297.00 ( 8.57%)
TPut 16 406442.00 ( 0.00%) 414965.00 ( 2.10%) 561855.00 ( 38.24%) 412339.00 ( 1.45%) 460001.00 ( 13.18%) 448531.00 ( 10.36%)
TPut 17 398612.00 ( 0.00%) 406309.00 ( 1.93%) 551546.00 ( 38.37%) 406632.00 ( 2.01%) 464289.00 ( 16.48%) 459191.00 ( 15.20%)
TPut 18 399217.00 ( 0.00%) 398503.00 ( -0.18%) 547892.00 ( 37.24%) 406229.00 ( 1.76%) 454615.00 ( 13.88%) 449444.00 ( 12.58%)In case you missed it at the header, THP is disabled in this test.
Overall, autonuma is the best showing gains no matter how many warehouses are used.
schednuma starts badly with a 30% regression but improves as the number of warehouses increases until it is comparable with a baseline kernel. Remember what I said about specjbb itself using the peak range of warehouses? I checked and in this case it used warehouses 12-18 for its throughput figure which would have missed all the regressions for low numbers. Watch for this in your own testing.
moron-v4r38 does nothing but it's not expected to, it lacks proper handling of PMDs.
twostage-v4r38 does better. It also regresses for low number of workloads but from 8 warehouses on it has a decent improvement over the baseline kernel.
thpmigrate-v4r38 makes no real difference here. There are some changed but it's likely just testing jitter as THP was disabled.
SPECJBB PEAKS
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12 rc6-schednuma-v16r2 rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38 rc6-thpmigrate-v4r38
Expctd Warehouse 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) Expctd Peak Bops 436440.00 ( 0.00%) 442472.00 ( 1.38%) 599670.00 ( 37.40%) 427404.00 ( -2.07%) 485326.00 ( 11.20%) 490229.00 ( 12.32%) Actual Warehouse 7.00 ( 0.00%) 8.00 ( 14.29%) 9.00 ( 28.57%) 7.00 ( 0.00%) 8.00 ( 14.29%) 7.00 ( 0.00%) Actual Peak Bops 543665.00 ( 0.00%) 459221.00 (-15.53%) 646336.00 ( 18.88%) 541093.00 ( -0.47%) 543024.00 ( -0.12%) 541144.00 ( -0.46%)
schednumas actual peak throughput regressed 15% from the baseline kernel
autonuma did best with an 18% improveent on the peak.
balancenuma does no worse at the peak. Note the peak warehouses of 7
was around when it started showing improvements.
MMTests Statistics: duration
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38rc6-thpmigrate-v4r38
User 101947.42 88113.29 101723.29 100931.37 99788.91 99783.34
System 66.48 12389.75 174.59 906.21 1575.66 1576.91
Elapsed 2457.45 2459.94 2461.46 2451.58 2457.17 2452.21schednumas system CPU usage is through the roof.
autonumas looks great but could be hiding it in threads.
balancenumas is pretty poor but a lot less than schednumas.
MMTests Statistics: vmstat
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38rc6-thpmigrate-v4r38
Page Ins 38540 38240 38524 38224 38104 38284
Page Outs 33276 34448 31808 31928 32380 30676
Swap Ins 0 0 0 0 0 0
Swap Outs 0 0 0 0 0 0
Direct pages scanned 0 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0 0Kswapd efficiency 100% 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0 0 Page writes file 0 0 0 0 0 0 Page writes anon 0 0 0 0 0 0 Page reclaim immediate 0 0 0 0 0 0 Page rescued immediate 0 0 0 0 0 0 Slabs scanned 0 0 0 0 0 0 Direct inode steals 0 0 0 0 0 0 Kswapd inode steals 0 0 0 0 0 0 Kswapd skipped wait 0 0 0 0 0 0 THP fault alloc 2 1 2 2 2 2 THP collapse alloc 0 0 0 0 0 0 THP splits 0 0 8 1 2 0 THP fault fallback 0 0 0 0 0 0 THP collapse fail 0 0 0 0 0 0 Compaction stalls 0 0 0 0 0 0 Compaction success 0 0 0 0 0 0 Compaction failures 0 0 0 0 0 0 Page migrate success 0 0 0 520232 44930994 44969103 Page migrate failure 0 0 0 0 0 0 Compaction pages isolated 0 0 0 0 0 0 Compaction migrate scanned 0 0 0 0 0 0 Compaction free scanned 0 0 0 0 0 0 Compaction cost 0 0 0 540 46638 46677 NUMA PTE updates 0 0 0 2985879895 386687008 386289592 NUMA hint faults 0 0 0 2762800008 360149388 359807642 NUMA hint local faults 0 0 0 700107356 97822934 97064458 NUMA pages migrated 0 0 0 520232 44930994 44969103 AutoNUMA cost 0 0 0 13834911 1804307 1802596
You can see the possible source of balancenumas overhead here. It updated an extremely large number of PTEs and incurred a very large number of faults. It needs better scan rate adaption but it needs a placement policy to drive that to detect if it's converging or not.
Note the THP figures -- there is almost no activity because THP is disabled.
SPECJBB BOPS Multiple JVMs, THP is enabled
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12 rc6-schednuma-v16r2rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38 rc6-thpmigrate-v4r38
Mean 1 31245.50 ( 0.00%) 26282.75 (-15.88%) 29527.75 ( -5.50%) 28873.50 ( -7.59%) 29596.25 ( -5.28%) 31146.00 ( -0.32%)
Mean 2 61735.75 ( 0.00%) 57095.50 ( -7.52%) 62362.50 ( 1.02%) 51322.50 (-16.87%) 55991.50 ( -9.30%) 61055.00 ( -1.10%)
Mean 3 90068.00 ( 0.00%) 87035.50 ( -3.37%) 94382.50 ( 4.79%) 78299.25 (-13.07%) 77209.25 (-14.28%) 91018.25 ( 1.06%)
Mean 4 116542.75 ( 0.00%) 113082.00 ( -2.97%) 123228.75 ( 5.74%) 97686.50 (-16.18%) 100294.75 (-13.94%) 116657.50 ( 0.10%)
Mean 5 136686.50 ( 0.00%) 119901.75 (-12.28%) 150850.25 ( 10.36%) 104357.25 (-23.65%) 121599.50 (-11.04%) 139162.25 ( 1.81%)
Mean 6 154764.00 ( 0.00%) 148642.25 ( -3.96%) 175157.25 ( 13.18%) 115533.25 (-25.35%) 140291.75 ( -9.35%) 158279.25 ( 2.27%)
Mean 7 152353.50 ( 0.00%) 154544.50 ( 1.44%) 180972.50 ( 18.78%) 131652.75 (-13.59%) 142895.00 ( -6.21%) 162127.00 ( 6.42%)
Mean 8 153510.50 ( 0.00%) 156682.00 ( 2.07%) 184412.00 ( 20.13%) 134736.75 (-12.23%) 141980.00 ( -7.51%) 161740.00 ( 5.36%)
Mean 9 141531.25 ( 0.00%) 151687.00 ( 7.18%) 184020.50 ( 30.02%) 133901.75 ( -5.39%) 137555.50 ( -2.81%) 157858.25 ( 11.54%)
Mean 10 141536.00 ( 0.00%) 144682.75 ( 2.22%) 179991.50 ( 27.17%) 131299.75 ( -7.23%) 132871.00 ( -6.12%) 151339.75 ( 6.93%)
Mean 11 139880.50 ( 0.00%) 140449.25 ( 0.41%) 174480.75 ( 24.74%) 122725.75 (-12.26%) 126864.00 ( -9.31%) 145256.50 ( 3.84%)
Mean 12 122948.25 ( 0.00%) 136247.50 ( 10.82%) 169831.25 ( 38.13%) 116190.25 ( -5.50%) 124048.00 ( 0.89%) 137139.25 ( 11.54%)
Mean 13 123131.75 ( 0.00%) 133700.75 ( 8.58%) 166204.50 ( 34.98%) 113206.25 ( -8.06%) 119934.00 ( -2.60%) 138639.25 ( 12.59%)
Mean 14 124271.25 ( 0.00%) 131856.75 ( 6.10%) 163368.25 ( 31.46%) 112379.75 ( -9.57%) 122836.75 ( -1.15%) 131143.50 ( 5.53%)
Mean 15 120426.75 ( 0.00%) 128455.25 ( 6.67%) 162290.00 ( 34.76%) 110448.50 ( -8.29%) 121109.25 ( 0.57%) 135818.25 ( 12.78%)
Mean 16 120899.00 ( 0.00%) 124334.00 ( 2.84%) 160002.00 ( 32.34%) 108771.25 (-10.03%) 113568.75 ( -6.06%) 127873.50 ( 5.77%)
Mean 17 120508.25 ( 0.00%) 124564.50 ( 3.37%) 158369.25 ( 31.42%) 106233.50 (-11.85%) 116768.50 ( -3.10%) 129826.50 ( 7.73%)
Mean 18 113974.00 ( 0.00%) 121539.25 ( 6.64%) 156437.50 ( 37.26%) 108424.50 ( -4.87%) 114648.50 ( 0.59%) 129318.50 ( 13.46%)
Stddev 1 1030.82 ( 0.00%) 781.13 ( 24.22%) 276.53 ( 73.17%) 1216.87 (-18.05%) 1666.25 (-61.64%) 949.68 ( 7.87%)
Stddev 2 837.50 ( 0.00%) 1449.41 (-73.06%) 937.19 (-11.90%) 1758.28 (-109.94%) 2300.84 (-174.73%) 1191.02 (-42.21%)
Stddev 3 629.40 ( 0.00%) 1314.87 (-108.91%) 1606.92 (-155.31%) 1682.12 (-167.26%) 2028.25 (-222.25%) 788.05 (-25.21%)
Stddev 4 1234.97 ( 0.00%) 525.14 ( 57.48%) 617.46 ( 50.00%) 2162.57 (-75.11%) 522.03 ( 57.73%) 1389.65 (-12.52%)
Stddev 5 997.81 ( 0.00%) 4516.97 (-352.69%) 2366.16 (-137.14%) 5545.91 (-455.81%) 2477.82 (-148.33%) 396.92 ( 60.22%)
Stddev 6 1196.81 ( 0.00%) 2759.43 (-130.56%) 1680.54 (-40.42%) 3188.65 (-166.43%) 2534.28 (-111.75%) 1648.18 (-37.71%)
Stddev 7 2808.10 ( 0.00%) 6114.11 (-117.73%) 2004.86 ( 28.60%) 6714.17 (-139.10%) 3538.72 (-26.02%) 3334.99 (-18.76%)
Stddev 8 3059.06 ( 0.00%) 8582.09 (-180.55%) 3534.51 (-15.54%) 5823.74 (-90.38%) 4425.50 (-44.67%) 3089.27 ( -0.99%)
Stddev 9 2244.91 ( 0.00%) 4927.67 (-119.50%) 5014.87 (-123.39%) 3233.41 (-44.03%) 3622.19 (-61.35%) 2718.62 (-21.10%)
Stddev 10 4662.71 ( 0.00%) 905.03 ( 80.59%) 6637.16 (-42.35%) 3183.20 ( 31.73%) 6056.20 (-29.89%) 3339.35 ( 28.38%)
Stddev 11 3671.80 ( 0.00%) 1863.28 ( 49.25%) 12270.82 (-234.19%) 2186.10 ( 40.46%) 3335.54 ( 9.16%) 1388.36 ( 62.19%)
Stddev 12 6802.60 ( 0.00%) 1897.86 ( 72.10%) 16818.87 (-147.24%) 2461.95 ( 63.81%) 1908.58 ( 71.94%) 5683.00 ( 16.46%)
Stddev 13 4798.34 ( 0.00%) 225.34 ( 95.30%) 16911.42 (-252.44%) 2282.32 ( 52.44%) 1952.91 ( 59.30%) 3572.80 ( 25.54%)
Stddev 14 4266.81 ( 0.00%) 1311.71 ( 69.26%) 16842.35 (-294.73%) 1898.80 ( 55.50%) 1738.97 ( 59.24%) 5058.54 (-18.56%)
Stddev 15 2361.19 ( 0.00%) 926.70 ( 60.75%) 17701.84 (-649.70%) 1907.33 ( 19.22%) 1599.64 ( 32.25%) 2199.69 ( 6.84%)
Stddev 16 1927.00 ( 0.00%) 521.78 ( 72.92%) 19107.14 (-891.55%) 2704.74 (-40.36%) 2354.42 (-22.18%) 3355.74 (-74.14%)
Stddev 17 3098.03 ( 0.00%) 910.17 ( 70.62%) 18920.22 (-510.72%) 2214.42 ( 28.52%) 2290.00 ( 26.08%) 1939.87 ( 37.38%)
Stddev 18 4045.82 ( 0.00%) 798.22 ( 80.27%) 17789.94 (-339.71%) 1287.48 ( 68.18%) 2189.19 ( 45.89%) 2531.60 ( 37.43%)
TPut 1 124982.00 ( 0.00%) 105131.00 (-15.88%) 118111.00 ( -5.50%) 115494.00 ( -7.59%) 118385.00 ( -5.28%) 124584.00 ( -0.32%)
TPut 2 246943.00 ( 0.00%) 228382.00 ( -7.52%) 249450.00 ( 1.02%) 205290.00 (-16.87%) 223966.00 ( -9.30%) 244220.00 ( -1.10%)
TPut 3 360272.00 ( 0.00%) 348142.00 ( -3.37%) 377530.00 ( 4.79%) 313197.00 (-13.07%) 308837.00 (-14.28%) 364073.00 ( 1.06%)
TPut 4 466171.00 ( 0.00%) 452328.00 ( -2.97%) 492915.00 ( 5.74%) 390746.00 (-16.18%) 401179.00 (-13.94%) 466630.00 ( 0.10%)
TPut 5 546746.00 ( 0.00%) 479607.00 (-12.28%) 603401.00 ( 10.36%) 417429.00 (-23.65%) 486398.00 (-11.04%) 556649.00 ( 1.81%)
TPut 6 619056.00 ( 0.00%) 594569.00 ( -3.96%) 700629.00 ( 13.18%) 462133.00 (-25.35%) 561167.00 ( -9.35%) 633117.00 ( 2.27%)
TPut 7 609414.00 ( 0.00%) 618178.00 ( 1.44%) 723890.00 ( 18.78%) 526611.00 (-13.59%) 571580.00 ( -6.21%) 648508.00 ( 6.42%)
TPut 8 614042.00 ( 0.00%) 626728.00 ( 2.07%) 737648.00 ( 20.13%) 538947.00 (-12.23%) 567920.00 ( -7.51%) 646960.00 ( 5.36%)
TPut 9 566125.00 ( 0.00%) 606748.00 ( 7.18%) 736082.00 ( 30.02%) 535607.00 ( -5.39%) 550222.00 ( -2.81%) 631433.00 ( 11.54%)
TPut 10 566144.00 ( 0.00%) 578731.00 ( 2.22%) 719966.00 ( 27.17%) 525199.00 ( -7.23%) 531484.00 ( -6.12%) 605359.00 ( 6.93%)
TPut 11 559522.00 ( 0.00%) 561797.00 ( 0.41%) 697923.00 ( 24.74%) 490903.00 (-12.26%) 507456.00 ( -9.31%) 581026.00 ( 3.84%)
TPut 12 491793.00 ( 0.00%) 544990.00 ( 10.82%) 679325.00 ( 38.13%) 464761.00 ( -5.50%) 496192.00 ( 0.89%) 548557.00 ( 11.54%)
TPut 13 492527.00 ( 0.00%) 534803.00 ( 8.58%) 664818.00 ( 34.98%) 452825.00 ( -8.06%) 479736.00 ( -2.60%) 554557.00 ( 12.59%)
TPut 14 497085.00 ( 0.00%) 527427.00 ( 6.10%) 653473.00 ( 31.46%) 449519.00 ( -9.57%) 491347.00 ( -1.15%) 524574.00 ( 5.53%)
TPut 15 481707.00 ( 0.00%) 513821.00 ( 6.67%) 649160.00 ( 34.76%) 441794.00 ( -8.29%) 484437.00 ( 0.57%) 543273.00 ( 12.78%)
TPut 16 483596.00 ( 0.00%) 497336.00 ( 2.84%) 640008.00 ( 32.34%) 435085.00 (-10.03%) 454275.00 ( -6.06%) 511494.00 ( 5.77%)
TPut 17 482033.00 ( 0.00%) 498258.00 ( 3.37%) 633477.00 ( 31.42%) 424934.00 (-11.85%) 467074.00 ( -3.10%) 519306.00 ( 7.73%)
TPut 18 455896.00 ( 0.00%) 486157.00 ( 6.64%) 625750.00 ( 37.26%) 433698.00 ( -4.87%) 458594.00 ( 0.59%) 517274.00 ( 13.46%)In case you missed it in the header, THP is enabled this time.
Again, autonuma is the best.
schednuma does much better here. It regresses for small number of warehouses and note that the specjbb reporting will have missed this because it focuses on the peak. For higher number of warehouses it sees a nice improvement of very roughly 2-8% performance gain. Again, it is worth double checking if the positive specjbb reports were based on peak warehouses and looking at what the other warehouse figures looked like.
twostage-v4r38 from balancenuma suffers here which initially surprised me but then I looked at the THP figures below. It's splitting its huge pages and trying to migrate them.
thpmigrate-v4r38 natively migrates pages. It marginally regresses for 1-2 warehouses but shows decent performance gains after that.
SPECJBB PEAKS
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12 rc6-schednuma-v16r2 rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38 rc6-thpmigrate-v4r38
Expctd Warehouse 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) Expctd Peak Bops 491793.00 ( 0.00%) 544990.00 ( 10.82%) 679325.00 ( 38.13%) 464761.00 ( -5.50%) 496192.00 ( 0.89%) 548557.00 ( 11.54%) Actual Warehouse 6.00 ( 0.00%) 8.00 ( 33.33%) 8.00 ( 33.33%) 8.00 ( 33.33%) 7.00 ( 16.67%) 7.00 ( 16.67%) Actual Peak Bops 619056.00 ( 0.00%) 626728.00 ( 1.24%) 737648.00 ( 19.16%) 538947.00 (-12.94%) 571580.00 ( -7.67%) 648508.00 ( 4.76%)
schednuma reports a 1.24% gain at the peak
autonuma reports 19.16%
balancenuma reports 4.76% but note it needed native THP migration to do that.
MMTests Statistics: duration
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38rc6-thpmigrate-v4r38
User 102073.40 101389.03 101952.32 100475.04 99905.11 101627.79
System 145.14 586.45 157.47 1257.01 1582.86 546.22
Elapsed 2457.98 2461.43 2450.75 2459.24 2459.39 2456.16schednumas system CPU usage is much more acceptable here. As it can deal with THPs a possible conclusion is that schednuma suffers when it has to deal with the individual PTE updates and faults.
autonuma had the lowest overhead for system CPU. Usual disclaimers apply about the kernel threads.
balancenuma had similar system CPU overhead to schednuma. Note how much a different native THP migration made to the system CPU usage.
MMTests Statistics: vmstat
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38rc6-thpmigrate-v4r38
Page Ins 38416 38260 38272 38076 38384 38104
Page Outs 33340 34696 31912 31736 31980 31360
Swap Ins 0 0 0 0 0 0
Swap Outs 0 0 0 0 0 0
Direct pages scanned 0 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0 0Kswapd efficiency 100% 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0 0 Page writes file 0 0 0 0 0 0 Page writes anon 0 0 0 0 0 0 Page reclaim immediate 0 0 0 0 0 0 Page rescued immediate 0 0 0 0 0 0 Slabs scanned 0 0 0 0 0 0 Direct inode steals 0 0 0 0 0 0 Kswapd inode steals 0 0 0 0 0 0 Kswapd skipped wait 0 0 0 0 0 0 THP fault alloc 64863 53973 48980 61397 61028 62441 THP collapse alloc 60 53 2254 1667 1575 56 THP splits 342 175 2194 12729 11544 329 THP fault fallback 0 0 0 0 0 0 THP collapse fail 0 0 0 0 0 0 Compaction stalls 0 0 0 0 0 0 Compaction success 0 0 0 0 0 0 Compaction failures 0 0 0 0 0 0 Page migrate success 0 0 0 5087468 41144914 340035 Page migrate failure 0 0 0 0 0 0 Compaction pages isolated 0 0 0 0 0 0 Compaction migrate scanned 0 0 0 0 0 0 Compaction free scanned 0 0 0 0 0 0 Compaction cost 0 0 0 5280 42708 352 NUMA PTE updates 0 0 0 2997404728 393796213 521840907 NUMA hint faults 0 0 0 2739639942 328788995 3461566 NUMA hint local faults 0 0 0 709168519 83931322 815569 NUMA pages migrated 0 0 0 5087468 41144914 340035 AutoNUMA cost 0 0 0 13719278 1647483 20967
There are a lot of PTE updates and faults here but it's not completely crazy.
The main point to note is the THP figures. THP migration heavily reduces the number of collapses and splits. Note however that all kernels showed some THP activity reflecting the fact it's actually enabled this time.
I do not have data yet on running specjbb on single JVM instances. I probably will not have for a long time either as I'm going to have to rerun more schednuma tests with additional patches on top.
The remainder of this covers some more basic performance tests. Unfortunately I do not have figures for the thpmigrate kernel as it's still running. However I would expect it to make very little difference to these results. If I'm wrong, then whoops.
KERNBENCH
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12 rc6-schednuma-v16r2rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38
User min 1296.75 ( 0.00%) 1299.23 ( -0.19%) 1290.49 ( 0.48%) 1297.40 ( -0.05%) 1297.74 ( -0.08%)
User mean 1299.08 ( 0.00%) 1309.99 ( -0.84%) 1293.82 ( 0.41%) 1300.66 ( -0.12%) 1299.70 ( -0.05%)
User stddev 1.78 ( 0.00%) 7.65 (-329.49%) 3.62 (-103.18%) 1.90 ( -6.92%) 1.17 ( 34.25%)
User max 1301.82 ( 0.00%) 1319.59 ( -1.37%) 1300.12 ( 0.13%) 1303.27 ( -0.11%) 1301.23 ( 0.05%)
System min 121.16 ( 0.00%) 139.16 (-14.86%) 123.79 ( -2.17%) 124.58 ( -2.82%) 124.06 ( -2.39%)
System mean 121.26 ( 0.00%) 146.11 (-20.49%) 124.42 ( -2.60%) 124.97 ( -3.05%) 124.32 ( -2.52%)
System stddev 0.07 ( 0.00%) 3.59 (-4725.82%) 0.45 (-506.41%) 0.29 (-294.47%) 0.22 (-195.02%)
System max 121.37 ( 0.00%) 148.94 (-22.72%) 125.04 ( -3.02%) 125.48 ( -3.39%) 124.65 ( -2.70%)
Elapsed min 41.90 ( 0.00%) 44.92 ( -7.21%) 40.10 ( 4.30%) 40.85 ( 2.51%) 41.56 ( 0.81%)
Elapsed mean 42.47 ( 0.00%) 45.74 ( -7.69%) 41.23 ( 2.93%) 42.49 ( -0.05%) 42.42 ( 0.13%)
Elapsed stddev 0.44 ( 0.00%) 0.52 (-17.51%) 0.93 (-110.57%) 1.01 (-129.42%) 0.74 (-68.20%)
Elapsed max 43.06 ( 0.00%) 46.51 ( -8.01%) 42.19 ( 2.02%) 43.56 ( -1.16%) 43.70 ( -1.49%)
CPU min 3300.00 ( 0.00%) 3133.00 ( 5.06%) 3354.00 ( -1.64%) 3277.00 ( 0.70%) 3257.00 ( 1.30%)
CPU mean 3343.80 ( 0.00%) 3183.20 ( 4.80%) 3441.00 ( -2.91%) 3356.20 ( -0.37%) 3357.20 ( -0.40%)
CPU stddev 36.31 ( 0.00%) 39.99 (-10.14%) 82.80 (-128.06%) 81.41 (-124.23%) 59.23 (-63.13%)
CPU max 3395.00 ( 0.00%) 3242.00 ( 4.51%) 3552.00 ( -4.62%) 3489.00 ( -2.77%) 3428.00 ( -0.97%)schednuma has improved a lot here. It used to be a 50% regression, now it's just a 7.69% regression.
autonuma showed a small gain but it's within 2*stddev so I would not get too excited.
balancenuma is comparable to the baseline kernel which is what you'd expect.
MMTests Statistics: duration
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38
User 7809.47 8426.10 7798.15 7834.32 7831.34
System 748.23 967.97 767.00 771.10 767.15
Elapsed 303.48 340.40 297.36 304.79 303.16schednuma is showing a lot higher system CPU usage. autonuma and balancenuma are showing some too.
MMTests Statistics: vmstat
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38
Page Ins 336 96 0 84 60
Page Outs 1606596 1565384 1470956 1477020 1682808
Swap Ins 0 0 0 0 0
Swap Outs 0 0 0 0 0
Direct pages scanned 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0Kswapd efficiency 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0 Page writes file 0 0 0 0 0 Page writes anon 0 0 0 0 0 Page reclaim immediate 0 0 0 0 0 Page rescued immediate 0 0 0 0 0 Slabs scanned 0 0 0 0 0 Direct inode steals 0 0 0 0 0 Kswapd inode steals 0 0 0 0 0 Kswapd skipped wait 0 0 0 0 0 THP fault alloc 373 331 392 334 338 THP collapse alloc 7 1 9913 57 69 THP splits 2 2 340 45 18 THP fault fallback 0 0 0 0 0 THP collapse fail 0 0 0 0 0 Compaction stalls 0 0 0 0 0 Compaction success 0 0 0 0 0 Compaction failures 0 0 0 0 0 Page migrate success 0 0 0 20870 567171 Page migrate failure 0 0 0 0 0 Compaction pages isolated 0 0 0 0 0 Compaction migrate scanned 0 0 0 0 0 Compaction free scanned 0 0 0 0 0 Compaction cost 0 0 0 21 588 NUMA PTE updates 0 0 0 104807469 108314529 NUMA hint faults 0 0 0 67587495 67487394 NUMA hint local faults 0 0 0 53813675 64082455 NUMA pages migrated 0 0 0 20870 567171 AutoNUMA cost 0 0 0 338671 338205
Ok... wow. So, schednuma does not report how many updates it made but look at balancenuma. It's updating PTEs and migrating pages for short-lived processes from a kernel build. Some of these updates will be against the monitors themselves but it's too high to be only the monitors. This is a big surprise to me but indicates that the delay start is still too fast or that there needs to be better identification of processes that do not care about NUMA.
===BEGIN aim9
AIM9
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12 rc6-schednuma-v16r2rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38
Min page_test 387600.00 ( 0.00%) 268486.67 (-30.73%) 356875.42 ( -7.93%) 342718.19 (-11.58%) 361405.73 ( -6.76%)
Min brk_test 2350099.93 ( 0.00%) 1996933.33 (-15.03%) 2198334.44 ( -6.46%) 2360733.33 ( 0.45%) 1856295.80 (-21.01%)
Min exec_test 255.99 ( 0.00%) 261.98 ( 2.34%) 273.15 ( 6.70%) 254.50 ( -0.58%) 257.33 ( 0.52%)
Min fork_test 1416.22 ( 0.00%) 1422.87 ( 0.47%) 1678.88 ( 18.55%) 1364.85 ( -3.63%) 1404.79 ( -0.81%)
Mean page_test 393893.69 ( 0.00%) 299688.63 (-23.92%) 374714.36 ( -4.87%) 377638.64 ( -4.13%) 373460.48 ( -5.19%)
Mean brk_test 2372673.79 ( 0.00%) 2221715.20 ( -6.36%) 2348968.24 ( -1.00%) 2394503.04 ( 0.92%) 2073987.04 (-12.59%)
Mean exec_test 258.91 ( 0.00%) 264.89 ( 2.31%) 280.17 ( 8.21%) 259.41 ( 0.19%) 260.94 ( 0.78%)
Mean fork_test 1428.88 ( 0.00%) 1447.96 ( 1.34%) 1812.08 ( 26.82%) 1398.49 ( -2.13%) 1430.22 ( 0.09%)
Stddev page_test 2689.70 ( 0.00%) 19221.87 (614.65%) 12994.24 (383.11%) 15871.82 (490.10%) 11104.15 (312.84%)
Stddev brk_test 11440.58 ( 0.00%) 174875.02 (1428.55%) 59011.99 (415.81%) 20870.31 ( 82.42%) 92043.46 (704.54%)
Stddev exec_test 1.42 ( 0.00%) 2.08 ( 46.59%) 6.06 (325.92%) 3.60 (152.88%) 1.80 ( 26.77%)
Stddev fork_test 8.30 ( 0.00%) 14.34 ( 72.70%) 48.64 (485.78%) 25.26 (204.22%) 17.05 (105.39%)
Max page_test 397800.00 ( 0.00%) 342833.33 (-13.82%) 396326.67 ( -0.37%) 393117.92 ( -1.18%) 391645.57 ( -1.55%)
Max brk_test 2386800.00 ( 0.00%) 2381133.33 ( -0.24%) 2416266.67 ( 1.23%) 2428733.33 ( 1.76%) 2245902.73 ( -5.90%)
Max exec_test 261.65 ( 0.00%) 267.82 ( 2.36%) 294.80 ( 12.67%) 266.00 ( 1.66%) 264.98 ( 1.27%)
Max fork_test 1446.58 ( 0.00%) 1468.44 ( 1.51%) 1869.59 ( 29.24%) 1454.18 ( 0.53%) 1475.08 ( 1.97%)Straight up, I find aim9 to be generally unreliable and can show regressions and gains for all sorts of unrelated nonsense. I keep running it because over long enough periods of time it can still identify trends.
schednuma is regressing 23% in the page fault microbenchmark. autonuma and balancenuma are also showing regressions. Not as bad, but not great by any means. brktest is also showing regressions and here balancenuma is showing quite a bit of hurt.
MMTests Statistics: duration
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38
User 2.77 2.81 2.88 2.76 2.76
System 0.76 0.72 0.74 0.74 0.74Elapsed 724.78 724.58 724.40 724.61 724.53
Not reflected in system CPU usage though. Cost is somewhere else
MMTests Statistics: vmstat
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38
Page Ins 7124 7096 6964 7388 7032
Page Outs 74380 73996 74324 73800 74576
Swap Ins 0 0 0 0 0
Swap Outs 0 0 0 0 0
Direct pages scanned 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0Kswapd efficiency 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0 Page writes file 0 0 0 0 0 Page writes anon 0 0 0 0 0 Page reclaim immediate 0 0 0 0 0 Page rescued immediate 0 0 0 0 0 Slabs scanned 0 0 0 0 0 Direct inode steals 0 0 0 0 0 Kswapd inode steals 0 0 0 0 0 Kswapd skipped wait 0 0 0 0 0 THP fault alloc 36 2 23 0 1 THP collapse alloc 0 0 8 8 1 THP splits 0 0 8 8 1 THP fault fallback 0 0 0 0 0 THP collapse fail 0 0 0 0 0 Compaction stalls 0 0 0 0 0 Compaction success 0 0 0 0 0 Compaction failures 0 0 0 0 0 Page migrate success 0 0 0 236 475 Page migrate failure 0 0 0 0 0 Compaction pages isolated 0 0 0 0 0 Compaction migrate scanned 0 0 0 0 0 Compaction free scanned 0 0 0 0 0 Compaction cost 0 0 0 0 0 NUMA PTE updates 0 0 0 21404376 40316461 NUMA hint faults 0 0 0 76711 10144 NUMA hint local faults 0 0 0 21258 9628 NUMA pages migrated 0 0 0 236 475 AutoNUMA cost 0 0 0 533 332
In balancenuma, you can see that it's taking NUMA faults and migrating. Maybe schednuma is doing the same.
HACKBENCH PIPES
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12 rc6-schednuma-v16r2rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38
Procs 1 0.0320 ( 0.00%) 0.0354 (-10.53%) 0.0410 (-28.28%) 0.0310 ( 3.00%) 0.0296 ( 7.55%)
Procs 4 0.0560 ( 0.00%) 0.0699 (-24.87%) 0.0641 (-14.47%) 0.0556 ( 0.79%) 0.0562 ( -0.36%)
Procs 8 0.0850 ( 0.00%) 0.1084 (-27.51%) 0.1397 (-64.30%) 0.0833 ( 1.96%) 0.0953 (-12.07%)
Procs 12 0.1047 ( 0.00%) 0.1084 ( -3.54%) 0.1789 (-70.91%) 0.0990 ( 5.44%) 0.1127 ( -7.72%)
Procs 16 0.1276 ( 0.00%) 0.1323 ( -3.67%) 0.1395 ( -9.34%) 0.1236 ( 3.16%) 0.1240 ( 2.83%)
Procs 20 0.1405 ( 0.00%) 0.1578 (-12.29%) 0.2452 (-74.52%) 0.1471 ( -4.73%) 0.1454 ( -3.50%)
Procs 24 0.1823 ( 0.00%) 0.1800 ( 1.24%) 0.3030 (-66.22%) 0.1776 ( 2.58%) 0.1574 ( 13.63%)
Procs 28 0.2019 ( 0.00%) 0.2143 ( -6.13%) 0.3403 (-68.52%) 0.2000 ( 0.94%) 0.1983 ( 1.78%)
Procs 32 0.2162 ( 0.00%) 0.2329 ( -7.71%) 0.6526 (-201.85%) 0.2235 ( -3.36%) 0.2158 ( 0.20%)
Procs 36 0.2354 ( 0.00%) 0.2577 ( -9.47%) 0.4468 (-89.77%) 0.2619 (-11.24%) 0.2451 ( -4.11%)
Procs 40 0.2600 ( 0.00%) 0.2850 ( -9.62%) 0.5247 (-101.79%) 0.2724 ( -4.77%) 0.2646 ( -1.75%)The number of procs hackbench is running is too low here for a 48-core machine. It should have been reconfigured but this is better than nothing.
schednuma and autonuma both show large regressions in the performance here. I do not investigate why but as there are a number of scheduler changes it could be anything.
balancenuma is showing some impact on the figures but it's gains and losses.
MMTests Statistics: duration
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38User 65.98 75.68 68.61 61.40 62.96
System 1934.87 2129.32 2104.72 1958.01 1902.99
Elapsed 100.52 106.29 153.66 102.06 99.96
Nothing major there. schednumas system CPu usage is higher which might be it.
MMTests Statistics: vmstat
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38
Page Ins 24 24 24 24 24
Page Outs 2092 1840 2636 1948 1912
Swap Ins 0 0 0 0 0
Swap Outs 0 0 0 0 0
Direct pages scanned 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0Kswapd efficiency 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0 Page writes file 0 0 0 0 0 Page writes anon 0 0 0 0 0 Page reclaim immediate 0 0 0 0 0 Page rescued immediate 0 0 0 0 0 Slabs scanned 0 0 0 0 0 Direct inode steals 0 0 0 0 0 Kswapd inode steals 0 0 0 0 0 Kswapd skipped wait 0 0 0 0 0 THP fault alloc 6 0 0 0 0 THP collapse alloc 0 0 0 3 0 THP splits 0 0 0 0 0 THP fault fallback 0 0 0 0 0 THP collapse fail 0 0 0 0 0 Compaction stalls 0 0 0 0 0 Compaction success 0 0 0 0 0 Compaction failures 0 0 0 0 0 Page migrate success 0 0 0 84 0 Page migrate failure 0 0 0 0 0 Compaction pages isolated 0 0 0 0 0 Compaction migrate scanned 0 0 0 0 0 Compaction free scanned 0 0 0 0 0 Compaction cost 0 0 0 0 0 NUMA PTE updates 0 0 0 152332 0 NUMA hint faults 0 0 0 21271 3 NUMA hint local faults 0 0 0 6778 0 NUMA pages migrated 0 0 0 84 0 AutoNUMA cost 0 0 0 107 0
Big surprise, moron-v4r38 was updating PTEs so some process was living long enough. Could have been the monitors though.
HACKBENCH SOCKETS
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12 rc6-schednuma-v16r2rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38
Procs 1 0.0260 ( 0.00%) 0.0320 (-23.08%) 0.0259 ( 0.55%) 0.0285 ( -9.62%) 0.0274 ( -5.57%)
Procs 4 0.0512 ( 0.00%) 0.0471 ( 7.99%) 0.0864 (-68.81%) 0.0481 ( 5.97%) 0.0469 ( 8.37%)
Procs 8 0.0739 ( 0.00%) 0.0782 ( -5.84%) 0.0823 (-11.41%) 0.0699 ( 5.38%) 0.0762 ( -3.12%)
Procs 12 0.0999 ( 0.00%) 0.1011 ( -1.18%) 0.1130 (-13.09%) 0.0961 ( 3.86%) 0.0977 ( 2.27%)
Procs 16 0.1270 ( 0.00%) 0.1311 ( -3.24%) 0.3777 (-197.40%) 0.1252 ( 1.38%) 0.1286 ( -1.29%)
Procs 20 0.1568 ( 0.00%) 0.1624 ( -3.56%) 0.3955 (-152.14%) 0.1568 ( -0.00%) 0.1566 ( 0.13%)
Procs 24 0.1845 ( 0.00%) 0.1914 ( -3.75%) 0.4127 (-123.73%) 0.1853 ( -0.47%) 0.1844 ( 0.06%)
Procs 28 0.2172 ( 0.00%) 0.2247 ( -3.48%) 0.5268 (-142.60%) 0.2163 ( 0.40%) 0.2230 ( -2.71%)
Procs 32 0.2505 ( 0.00%) 0.2553 ( -1.93%) 0.5334 (-112.96%) 0.2489 ( 0.63%) 0.2487 ( 0.72%)
Procs 36 0.2830 ( 0.00%) 0.2872 ( -1.47%) 0.7256 (-156.39%) 0.2787 ( 1.53%) 0.2751 ( 2.79%)
Procs 40 0.3041 ( 0.00%) 0.3200 ( -5.22%) 0.9365 (-207.91%) 0.3100 ( -1.93%) 0.3134 ( -3.04%)schednuma showing small regressions here.
autonuma showed massive regressions here.
balancenuma is ok because scheduler decisions are mostly left alone. It's the PTE numa updates where it kicks in.
MMTests Statistics: duration
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38User 43.39 48.16 46.27 39.19 38.39
System 2305.48 2339.98 2461.69 2271.80 2265.79
Elapsed 109.65 111.15 173.41 108.75 108.52
Nothing major there.
MMTests Statistics: vmstat
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38
Page Ins 4 4 4 4 4
Page Outs 1848 1840 2672 1788 1896
Swap Ins 0 0 0 0 0
Swap Outs 0 0 0 0 0
Direct pages scanned 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0Kswapd efficiency 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0 Page writes file 0 0 0 0 0 Page writes anon 0 0 0 0 0 Page reclaim immediate 0 0 0 0 0 Page rescued immediate 0 0 0 0 0 Slabs scanned 0 0 0 0 0 Direct inode steals 0 0 0 0 0 Kswapd inode steals 0 0 0 0 0 Kswapd skipped wait 0 0 0 0 0 THP fault alloc 6 0 0 0 0 THP collapse alloc 1 0 3 0 0 THP splits 0 0 3 3 0 THP fault fallback 0 0 0 0 0 THP collapse fail 0 0 0 0 0 Compaction stalls 0 0 0 0 0 Compaction success 0 0 0 0 0 Compaction failures 0 0 0 0 0 Page migrate success 0 0 0 96 0 Page migrate failure 0 0 0 0 0 Compaction pages isolated 0 0 0 0 0 Compaction migrate scanned 0 0 0 0 0 Compaction free scanned 0 0 0 0 0 Compaction cost 0 0 0 0 0 NUMA PTE updates 0 0 0 117626 0 NUMA hint faults 0 0 0 11781 0 NUMA hint local faults 0 0 0 2785 0 NUMA pages migrated 0 0 0 96 0 AutoNUMA cost 0 0 0 59 0
Some PTE updates from moron-v4r8 again. Again could be the monitors.
I ran the STREAM benchmark but it's long and there was nothing interesting to report. performance was flat and there was some migration activity which is bad but as STREAM is long-lived for larger amounts of memory it was not too suprising. It deserves better investigation but is realtively low priority when it showed no regressions.
PAGE FAULT TEST
This is a microbenchmark for page faults. The number of clients are badly ordered which again, I really should fix but anyway.
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12 rc6-schednuma-v16r2rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38
System 1 8.0710 ( 0.00%) 8.1085 ( -0.46%) 8.0925 ( -0.27%) 8.0170 ( 0.67%) 37.3075 (-362.24%
System 10 9.4975 ( 0.00%) 9.5690 ( -0.75%) 12.0055 (-26.41%) 9.5915 ( -0.99%) 9.5835 ( -0.91%)
System 11 9.7740 ( 0.00%) 9.7915 ( -0.18%) 13.4890 (-38.01%) 9.7275 ( 0.48%) 9.6810 ( 0.95%)
System 12 9.6300 ( 0.00%) 9.7065 ( -0.79%) 13.6075 (-41.30%) 9.8320 ( -2.10%) 9.7365 ( -1.11%)
System 13 10.3300 ( 0.00%) 10.2560 ( 0.72%) 17.2815 (-67.29%) 10.2435 ( 0.84%) 10.2480 ( 0.79%)
System 14 10.7300 ( 0.00%) 10.6860 ( 0.41%) 13.5335 (-26.13%) 10.5975 ( 1.23%) 10.6490 ( 0.75%)
System 15 10.7860 ( 0.00%) 10.8695 ( -0.77%) 18.8370 (-74.64%) 10.7860 ( 0.00%) 10.7685 ( 0.16%)
System 16 11.2070 ( 0.00%) 11.3730 ( -1.48%) 17.6445 (-57.44%) 11.1970 ( 0.09%) 11.2270 ( -0.18%)
System 17 11.8695 ( 0.00%) 11.9420 ( -0.61%) 15.7420 (-32.63%) 11.8660 ( 0.03%) 11.8465 ( 0.19%)
System 18 12.3110 ( 0.00%) 12.3800 ( -0.56%) 18.7010 (-51.90%) 12.4065 ( -0.78%) 12.3975 ( -0.70%)
System 19 12.8610 ( 0.00%) 13.0375 ( -1.37%) 17.5450 (-36.42%) 12.9510 ( -0.70%) 13.0045 ( -1.12%)
System 2 8.0750 ( 0.00%) 8.1405 ( -0.81%) 8.2075 ( -1.64%) 8.0670 ( 0.10%) 11.5805 (-43.41%)
System 20 13.5975 ( 0.00%) 13.4650 ( 0.97%) 17.6630 (-29.90%) 13.4485 ( 1.10%) 13.2655 ( 2.44%)
System 21 13.9945 ( 0.00%) 14.1510 ( -1.12%) 16.6380 (-18.89%) 13.9305 ( 0.46%) 13.9215 ( 0.52%)
System 22 14.5055 ( 0.00%) 14.6145 ( -0.75%) 19.8770 (-37.03%) 14.5555 ( -0.34%) 14.6435 ( -0.95%)
System 23 15.0345 ( 0.00%) 15.2365 ( -1.34%) 19.6190 (-30.49%) 15.0930 ( -0.39%) 15.2005 ( -1.10%)
System 24 15.5565 ( 0.00%) 15.7380 ( -1.17%) 20.5575 (-32.15%) 15.5965 ( -0.26%) 15.6015 ( -0.29%)
System 25 16.1795 ( 0.00%) 16.3190 ( -0.86%) 21.6805 (-34.00%) 16.1595 ( 0.12%) 16.2315 ( -0.32%)
System 26 17.0595 ( 0.00%) 16.9270 ( 0.78%) 19.8575 (-16.40%) 16.9075 ( 0.89%) 16.7940 ( 1.56%)
System 27 17.3200 ( 0.00%) 17.4150 ( -0.55%) 19.2015 (-10.86%) 17.5160 ( -1.13%) 17.3045 ( 0.09%)
System 28 17.9900 ( 0.00%) 18.0230 ( -0.18%) 20.3495 (-13.12%) 18.0700 ( -0.44%) 17.8465 ( 0.80%)
System 29 18.5160 ( 0.00%) 18.6785 ( -0.88%) 21.1070 (-13.99%) 18.5375 ( -0.12%) 18.5735 ( -0.31%)
System 3 8.1575 ( 0.00%) 8.2200 ( -0.77%) 8.3190 ( -1.98%) 8.2200 ( -0.77%) 9.5105 (-16.59%)
System 30 19.2095 ( 0.00%) 19.4355 ( -1.18%) 22.2920 (-16.05%) 19.1850 ( 0.13%) 19.1160 ( 0.49%)
System 31 19.7165 ( 0.00%) 19.7785 ( -0.31%) 21.5625 ( -9.36%) 19.7635 ( -0.24%) 20.0735 ( -1.81%)
System 32 20.5370 ( 0.00%) 20.5395 ( -0.01%) 22.7315 (-10.69%) 20.2400 ( 1.45%) 20.2930 ( 1.19%)
System 33 20.9265 ( 0.00%) 21.3055 ( -1.81%) 22.2900 ( -6.52%) 20.9520 ( -0.12%) 21.0705 ( -0.69%)
System 34 21.9625 ( 0.00%) 21.7200 ( 1.10%) 24.1665 (-10.04%) 21.5605 ( 1.83%) 21.6485 ( 1.43%)
System 35 22.3010 ( 0.00%) 22.4145 ( -0.51%) 23.5105 ( -5.42%) 22.3475 ( -0.21%) 22.4405 ( -0.63%)
System 36 23.0040 ( 0.00%) 23.0160 ( -0.05%) 23.8965 ( -3.88%) 23.2190 ( -0.93%) 22.9625 ( 0.18%)
System 37 23.6785 ( 0.00%) 23.7325 ( -0.23%) 24.8125 ( -4.79%) 23.7495 ( -0.30%) 23.6925 ( -0.06%)
System 38 24.7495 ( 0.00%) 24.8330 ( -0.34%) 25.0045 ( -1.03%) 24.2465 ( 2.03%) 24.3775 ( 1.50%)
System 39 25.0975 ( 0.00%) 25.1845 ( -0.35%) 25.8640 ( -3.05%) 25.0515 ( 0.18%) 25.0655 ( 0.13%)
System 4 8.2660 ( 0.00%) 8.3770 ( -1.34%) 9.0370 ( -9.33%) 8.3380 ( -0.87%) 8.6195 ( -4.28%)
System 40 25.9170 ( 0.00%) 26.1390 ( -0.86%) 25.7945 ( 0.47%) 25.8330 ( 0.32%) 25.7755 ( 0.55%)
System 41 26.4745 ( 0.00%) 26.6030 ( -0.49%) 26.0005 ( 1.79%) 26.4665 ( 0.03%) 26.6990 ( -0.85%)
System 42 27.4050 ( 0.00%) 27.4030 ( 0.01%) 27.1415 ( 0.96%) 27.4045 ( 0.00%) 27.1995 ( 0.75%)
System 43 27.9820 ( 0.00%) 28.3140 ( -1.19%) 27.2640 ( 2.57%) 28.1045 ( -0.44%) 28.0070 ( -0.09%)
System 44 28.7245 ( 0.00%) 28.9940 ( -0.94%) 27.4990 ( 4.27%) 28.6740 ( 0.18%) 28.6515 ( 0.25%)
System 45 29.5315 ( 0.00%) 29.8435 ( -1.06%) 28.3015 ( 4.17%) 29.5350 ( -0.01%) 29.3825 ( 0.50%)
System 46 30.2260 ( 0.00%) 30.5220 ( -0.98%) 28.3505 ( 6.20%) 30.2100 ( 0.05%) 30.2865 ( -0.20%)
System 47 31.0865 ( 0.00%) 31.3480 ( -0.84%) 28.6695 ( 7.78%) 30.9940 ( 0.30%) 30.9930 ( 0.30%)
System 48 31.5745 ( 0.00%) 31.9750 ( -1.27%) 28.8480 ( 8.64%) 31.6925 ( -0.37%) 31.6355 ( -0.19%)
System 5 8.5895 ( 0.00%) 8.6365 ( -0.55%) 10.7745 (-25.44%) 8.6905 ( -1.18%) 8.7105 ( -1.41%)
System 6 8.8350 ( 0.00%) 8.8820 ( -0.53%) 10.7165 (-21.30%) 8.8105 ( 0.28%) 8.8090 ( 0.29%)
System 7 8.9120 ( 0.00%) 8.9095 ( 0.03%) 10.0140 (-12.37%) 8.9440 ( -0.36%) 9.0585 ( -1.64%)
System 8 8.8235 ( 0.00%) 8.9295 ( -1.20%) 10.3175 (-16.93%) 8.9185 ( -1.08%) 8.8695 ( -0.52%)
System 9 9.4775 ( 0.00%) 9.5080 ( -0.32%) 10.9855 (-15.91%) 9.4815 ( -0.04%) 9.4435 ( 0.36%)autonuma shows high system CPU usage overhead here.
schednuma and balancenuma show some but it's not crazy. Processes are likely too short-lived
Elapsed 1 8.7755 ( 0.00%) 8.8080 ( -0.37%) 8.7870 ( -0.13%) 8.7060 ( 0.79%) 38.0820 (-333.96%) Elapsed 10 1.0985 ( 0.00%) 1.0965 ( 0.18%) 1.3965 (-27.13%) 1.1120 ( -1.23%) 1.1070 ( -0.77%) Elapsed 11 1.0280 ( 0.00%) 1.0340 ( -0.58%) 1.4540 (-41.44%) 1.0220 ( 0.58%) 1.0160 ( 1.17%) Elapsed 12 0.9155 ( 0.00%) 0.9250 ( -1.04%) 1.3995 (-52.87%) 0.9430 ( -3.00%) 0.9455 ( -3.28%) Elapsed 13 0.9500 ( 0.00%) 0.9325 ( 1.84%) 1.6625 (-75.00%) 0.9345 ( 1.63%) 0.9470 ( 0.32%) Elapsed 14 0.8910 ( 0.00%) 0.9000 ( -1.01%) 1.2435 (-39.56%) 0.8835 ( 0.84%) 0.9005 ( -1.07%) Elapsed 15 0.8245 ( 0.00%) 0.8290 ( -0.55%) 1.7575 (-113.16%) 0.8250 ( -0.06%) 0.8205 ( 0.49%) Elapsed 16 0.8050 ( 0.00%) 0.8040 ( 0.12%) 1.5650 (-94.41%) 0.7980 ( 0.87%) 0.8140 ( -1.12%) Elapsed 17 0.8365 ( 0.00%) 0.8440 ( -0.90%) 1.3350 (-59.59%) 0.8355 ( 0.12%) 0.8305 ( 0.72%) Elapsed 18 0.8015 ( 0.00%) 0.8030 ( -0.19%) 1.5420 (-92.39%) 0.8040 ( -0.31%) 0.8000 ( 0.19%) Elapsed 19 0.7700 ( 0.00%) 0.7720 ( -0.26%) 1.4410 (-87.14%) 0.7770 ( -0.91%) 0.7805 ( -1.36%) Elapsed 2 4.4485 ( 0.00%) 4.4850 ( -0.82%) 4.5230 ( -1.67%) 4.4145 ( 0.76%) 6.2950 (-41.51%) Elapsed 20 0.7725 ( 0.00%) 0.7565 ( 2.07%) 1.4245 (-84.40%) 0.7580 ( 1.88%) 0.7485 ( 3.11%) Elapsed 21 0.7965 ( 0.00%) 0.8135 ( -2.13%) 1.2630 (-58.57%) 0.7995 ( -0.38%) 0.8055 ( -1.13%) Elapsed 22 0.7785 ( 0.00%) 0.7785 ( 0.00%) 1.5505 (-99.17%) 0.7940 ( -1.99%) 0.7905 ( -1.54%) Elapsed 23 0.7665 ( 0.00%) 0.7700 ( -0.46%) 1.5335 (-100.07%) 0.7605 ( 0.78%) 0.7905 ( -3.13%) Elapsed 24 0.7655 ( 0.00%) 0.7630 ( 0.33%) 1.5210 (-98.69%) 0.7455 ( 2.61%) 0.7660 ( -0.07%) Elapsed 25 0.8430 ( 0.00%) 0.8580 ( -1.78%) 1.6220 (-92.41%) 0.8565 ( -1.60%) 0.8640 ( -2.49%) Elapsed 26 0.8585 ( 0.00%) 0.8385 ( 2.33%) 1.3195 (-53.70%) 0.8240 ( 4.02%) 0.8480 ( 1.22%) Elapsed 27 0.8195 ( 0.00%) 0.8115 ( 0.98%) 1.2000 (-46.43%) 0.8165 ( 0.37%) 0.8060 ( 1.65%) Elapsed 28 0.7985 ( 0.00%) 0.7845 ( 1.75%) 1.2925 (-61.87%) 0.8085 ( -1.25%) 0.8020 ( -0.44%) Elapsed 29 0.7995 ( 0.00%) 0.7995 ( 0.00%) 1.3140 (-64.35%) 0.8135 ( -1.75%) 0.8050 ( -0.69%) Elapsed 3 3.0140 ( 0.00%) 3.0110 ( 0.10%) 3.0735 ( -1.97%) 3.0230 ( -0.30%) 3.4670 (-15.03%) Elapsed 30 0.8075 ( 0.00%) 0.7935 ( 1.73%) 1.3905 (-72.20%) 0.8045 ( 0.37%) 0.8000 ( 0.93%) Elapsed 31 0.7895 ( 0.00%) 0.7735 ( 2.03%) 1.2075 (-52.94%) 0.8015 ( -1.52%) 0.8135 ( -3.04%) Elapsed 32 0.8055 ( 0.00%) 0.7745 ( 3.85%) 1.3090 (-62.51%) 0.7705 ( 4.35%) 0.7815 ( 2.98%) Elapsed 33 0.7860 ( 0.00%) 0.7710 ( 1.91%) 1.1485 (-46.12%) 0.7850 ( 0.13%) 0.7985 ( -1.59%) Elapsed 34 0.7950 ( 0.00%) 0.7750 ( 2.52%) 1.4080 (-77.11%) 0.7800 ( 1.89%) 0.7870 ( 1.01%) Elapsed 35 0.7900 ( 0.00%) 0.7720 ( 2.28%) 1.1245 (-42.34%) 0.7965 ( -0.82%) 0.8230 ( -4.18%) Elapsed 36 0.7930 ( 0.00%) 0.7600 ( 4.16%) 1.1240 (-41.74%) 0.8150 ( -2.77%) 0.7875 ( 0.69%) Elapsed 37 0.7830 ( 0.00%) 0.7565 ( 3.38%) 1.2870 (-64.37%) 0.7860 ( -0.38%) 0.7795 ( 0.45%) Elapsed 38 0.8035 ( 0.00%) 0.7960 ( 0.93%) 1.1955 (-48.79%) 0.7700 ( 4.17%) 0.7695 ( 4.23%) Elapsed 39 0.7760 ( 0.00%) 0.7680 ( 1.03%) 1.3305 (-71.46%) 0.7700 ( 0.77%) 0.7820 ( -0.77%) Elapsed 4 2.2845 ( 0.00%) 2.3185 ( -1.49%) 2.4895 ( -8.97%) 2.3010 ( -0.72%) 2.4175 ( -5.82%) Elapsed 40 0.7710 ( 0.00%) 0.7720 ( -0.13%) 1.0095 (-30.93%) 0.7655 ( 0.71%) 0.7670 ( 0.52%) Elapsed 41 0.7880 ( 0.00%) 0.7510 ( 4.70%) 1.1440 (-45.18%) 0.7590 ( 3.68%) 0.7985 ( -1.33%) Elapsed 42 0.7780 ( 0.00%) 0.7690 ( 1.16%) 1.2405 (-59.45%) 0.7845 ( -0.84%) 0.7815 ( -0.45%) Elapsed 43 0.7650 ( 0.00%) 0.7760 ( -1.44%) 1.0820 (-41.44%) 0.7795 ( -1.90%) 0.7600 ( 0.65%) Elapsed 44 0.7595 ( 0.00%) 0.7590 ( 0.07%) 1.1615 (-52.93%) 0.7590 ( 0.07%) 0.7540 ( 0.72%) Elapsed 45 0.7730 ( 0.00%) 0.7535 ( 2.52%) 0.9845 (-27.36%) 0.7735 ( -0.06%) 0.7705 ( 0.32%) Elapsed 46 0.7735 ( 0.00%) 0.7650 ( 1.10%) 0.9610 (-24.24%) 0.7625 ( 1.42%) 0.7660 ( 0.97%) Elapsed 47 0.7645 ( 0.00%) 0.7670 ( -0.33%) 1.1040 (-44.41%) 0.7650 ( -0.07%) 0.7675 ( -0.39%) Elapsed 48 0.7655 ( 0.00%) 0.7675 ( -0.26%) 1.2085 (-57.87%) 0.7590 ( 0.85%) 0.7700 ( -0.59%) Elapsed 5 1.9355 ( 0.00%) 1.9425 ( -0.36%) 2.3495 (-21.39%) 1.9710 ( -1.83%) 1.9675 ( -1.65%) Elapsed 6 1.6640 ( 0.00%) 1.6760 ( -0.72%) 1.9865 (-19.38%) 1.6430 ( 1.26%) 1.6405 ( 1.41%) Elapsed 7 1.4405 ( 0.00%) 1.4295 ( 0.76%) 1.6215 (-12.57%) 1.4370 ( 0.24%) 1.4550 ( -1.01%) Elapsed 8 1.2320 ( 0.00%) 1.2545 ( -1.83%) 1.4595 (-18.47%) 1.2465 ( -1.18%) 1.2440 ( -0.97%) Elapsed 9 1.2260 ( 0.00%) 1.2270 ( -0.08%) 1.3955 (-13.83%) 1.2285 ( -0.20%) 1.2180 ( 0.65%)
Same story. autonuma takes a hit. schednuma and balancenuma are ok.
There are also faults/sec and faults/cpu/sec stats but they all tell more or less the same story. autonuma took a hit. schednuma and balancenuma are ok.
MMTests Statistics: duration
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38User 1097.70 963.35 1275.69 1095.71 1104.06
System 18926.22 18947.86 22664.44 18895.61 19587.47
Elapsed 1374.39 1360.35 1888.67 1369.07 2008.11
autonuma has higher system CPU usage so that might account for its loss. Again balancenuma and schednuma are ok.
MMTests Statistics: vmstat
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0
rc6-stats-v4r12rc6-schednuma-v16r2rc6-autonuma-v28fastr3rc6-moron-v4r38rc6-twostage-v4r38
Page Ins 364 364 364 364 364
Page Outs 14756 15188 20036 15152 19152
Swap Ins 0 0 0 0 0
Swap Outs 0 0 0 0 0
Direct pages scanned 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0Kswapd efficiency 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0 Page writes file 0 0 0 0 0 Page writes anon 0 0 0 0 0 Page reclaim immediate 0 0 0 0 0 Page rescued immediate 0 0 0 0 0 Slabs scanned 0 0 0 0 0 Direct inode steals 0 0 0 0 0 Kswapd inode steals 0 0 0 0 0 Kswapd skipped wait 0 0 0 0 0 THP fault alloc 0 0 0 0 0 THP collapse alloc 0 0 0 0 0 THP splits 0 0 5 1 0 THP fault fallback 0 0 0 0 0 THP collapse fail 0 0 0 0 0 Compaction stalls 0 0 0 0 0 Compaction success 0 0 0 0 0 Compaction failures 0 0 0 0 0 Page migrate success 0 0 0 938 2892 Page migrate failure 0 0 0 0 0 Compaction pages isolated 0 0 0 0 0 Compaction migrate scanned 0 0 0 0 0 Compaction free scanned 0 0 0 0 0 Compaction cost 0 0 0 0 3 NUMA PTE updates 0 0 0 297476912 497772489 NUMA hint faults 0 0 0 290139 2456411 NUMA hint local faults 0 0 0 115544 2449766 NUMA pages migrated 0 0 0 938 2892 AutoNUMA cost 0 0 0 3533 15766
Some NUMA update activity here. Again, might be the monitors. As these stats are collected before and after the test they are collected even if monitors are disabled so that would indicate if monitors are making a difference. It could be some other long-lived process on the system too.
So there you have it. balancenumas foundation has many things in common with schednuma but does a lot more in just the basic mechanics to keep the overhead under control and to avoid falling apart when the placement policy makes wrong decisions. Even without a placment policy it can beat schednuma in a number of cases and while I do not expect this to be universal to all machines, it's encouraging.
Can the schednuma people please reconsider rebasing on top of this? It should be able to show in all cases that it improves performance over no placement policy and it'll be a bit more obvious how it does it. I would also hope that the concepts of autonuma would be reimplemented on top of this foundation so we can do a meaningful comparison between different placement policies.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi,
Quoted 15 lines · Mel Gorman
HACKBENCH PIPES
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 rc6-stats-v4r12 rc6-schednuma-v16r2rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38 Procs 1 0.0320 ( 0.00%) 0.0354 (-10.53%) 0.0410 (-28.28%) 0.0310 ( 3.00%) 0.0296 ( 7.55%) Procs 4 0.0560 ( 0.00%) 0.0699 (-24.87%) 0.0641 (-14.47%) 0.0556 ( 0.79%) 0.0562 ( -0.36%) Procs 8 0.0850 ( 0.00%) 0.1084 (-27.51%) 0.1397 (-64.30%) 0.0833 ( 1.96%) 0.0953 (-12.07%) Procs 12 0.1047 ( 0.00%) 0.1084 ( -3.54%) 0.1789 (-70.91%) 0.0990 ( 5.44%) 0.1127 ( -7.72%) Procs 16 0.1276 ( 0.00%) 0.1323 ( -3.67%) 0.1395 ( -9.34%) 0.1236 ( 3.16%) 0.1240 ( 2.83%) Procs 20 0.1405 ( 0.00%) 0.1578 (-12.29%) 0.2452 (-74.52%) 0.1471 ( -4.73%) 0.1454 ( -3.50%) Procs 24 0.1823 ( 0.00%) 0.1800 ( 1.24%) 0.3030 (-66.22%) 0.1776 ( 2.58%) 0.1574 ( 13.63%) Procs 28 0.2019 ( 0.00%) 0.2143 ( -6.13%) 0.3403 (-68.52%) 0.2000 ( 0.94%) 0.1983 ( 1.78%) Procs 32 0.2162 ( 0.00%) 0.2329 ( -7.71%) 0.6526 (-201.85%) 0.2235 ( -3.36%) 0.2158 ( 0.20%) Procs 36 0.2354 ( 0.00%) 0.2577 ( -9.47%) 0.4468 (-89.77%) 0.2619 (-11.24%) 0.2451 ( -4.11%) Procs 40 0.2600 ( 0.00%) 0.2850 ( -9.62%) 0.5247 (-101.79%) 0.2724 ( -4.77%) 0.2646 ( -1.75%)
Quoted 6 lines
The number of procs hackbench is running is too low here for a 48-core machine. It should have been reconfigured but this is better than nothing.
schednuma and autonuma both show large regressions in the performance here. I do not investigate why but as there are a number of scheduler changes it could be anything.
Strange, last time I tested hackbench it was perfectly ok, I even had this test shown in some of the pdf.
Lately (post my last hackbench run) I disabled the affine wakeups cross-node and pipes use sd_affine wakeups. That could matter for these heavy scheduling tests as it practically disables the _sync in wake_up_interruptible_sync_poll used by the pipe code, if the waker CPU is in a different node than the wakee prev_cpu. I discussed this with Mike and he liked this change IIRC but it's the first thing that should be checked at the light of above regression.
Quoted 10 lines
PAGE FAULT TEST
This is a microbenchmark for page faults. The number of clients are badly ordered which again, I really should fix but anyway.
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 rc6-stats-v4r12 rc6-schednuma-v16r2rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38 System 1 8.0710 ( 0.00%) 8.1085 ( -0.46%) 8.0925 ( -0.27%) 8.0170 ( 0.67%) 37.3075 (-362.24% System 10 9.4975 ( 0.00%) 9.5690 ( -0.75%) 12.0055 (-26.41%) 9.5915 ( -0.99%) 9.5835 ( -0.91%) System 11 9.7740 ( 0.00%) 9.7915 ( -0.18%) 13.4890 (-38.01%) 9.7275 ( 0.48%) 9.6810 ( 0.95%)
No real clue on this one as I should look in what the test does. It might be related to THP splits though. I can't imagine anything else because there's nothing at all in autonuma that alters the page faults (except from arming NUMA hinting faults which should be lighter in autonuma than in the other implementation using task work).
Chances are the faults are tested by touching bytes at different 4k offsets in the same 2m naturally aligned virtual range.
Hugh THP native migration patch will clarify things on the above.
Quoted 3 lines
also hope that the concepts of autonuma would be reimplemented on top of this foundation so we can do a meaningful comparison between different placement policies.
I'll try to help with this to see what could be added from autonuma on top to improve on top your balancenuma foundation. Your current foundation looks ideal for inclusion to me.
I noticed you haven't run any single instance specjbb workload, that should be added to the battery of tests. But hey take your time, the amount of data you provided is already very comprehensive and you were so fast.
The thing is: single instance and multi instance are totally different beasts.
multi instance is all about avoiding NUMA false sharing in the first place (the anti false sharing algorithm becomes a noop), and it has a trivial perfect solution with all cross node traffic guaranteed to stop after converence has been reached for the whole duration of the workload.
single instance is all about NUMA false sharing detection and it has no perfect solution and there's no way to fully converge and to stop all cross node traffic. So it's a tradeoff between doing too many CPU/memory spurious migrations (harmful, causes regressions) and doing too few (i.e. not improving at all compared to upstream but not regressing either).
autonuma27/28/28fast will perform identical on multi instance loads (i.e. optimal, a few percent away from hard bindings).
I was just starting to improve the anti false sharing algorithm in autonuma28/28fast to improve single instance specjbb too (this is why you really need autonuma28 or autonuma28fast to test single instance specjbb and not autonuma27).
About THP, normally when I was running benchmarks I was testing these 4 configs:
1) THP on PMD scan on
2) THP on PMD scan off
3) THP off PMD scan on
4) THP off PMD scan off
(1 and 2 are practically the same for the autonuma benchmark, because all memory is backed by THP rendering the PMD level hinting faults for 4k pages very unlikely, but I was testing it anyway just in case)
THP off is going to hit KVM guests the most and much less host workloads. But even for KVM it's good practice to test with THP off too, to verify the cost of the numa hinting page faults remains very low (the cost is much higher for guests than host because of the vmexists).
The KVM benchmark run by IBM was also done in all 4 combinations: THP on/off KSM on/off and showed improvement even for the "No THP" case (btw, things should run much better these days than the old autonuma13).
http://dl.dropbox.com/u/82832537/kvm-numa-comparison-0.png
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 21 lines · Andrea Arcangeli
Hi,
Quoted 18 lines · Mel Gorman
HACKBENCH PIPES
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 rc6-stats-v4r12 rc6-schednuma-v16r2rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38 Procs 1 0.0320 ( 0.00%) 0.0354 (-10.53%) 0.0410 (-28.28%) 0.0310 ( 3.00%) 0.0296 ( 7.55%) Procs 4 0.0560 ( 0.00%) 0.0699 (-24.87%) 0.0641 (-14.47%) 0.0556 ( 0.79%) 0.0562 ( -0.36%) Procs 8 0.0850 ( 0.00%) 0.1084 (-27.51%) 0.1397 (-64.30%) 0.0833 ( 1.96%) 0.0953 (-12.07%) Procs 12 0.1047 ( 0.00%) 0.1084 ( -3.54%) 0.1789 (-70.91%) 0.0990 ( 5.44%) 0.1127 ( -7.72%) Procs 16 0.1276 ( 0.00%) 0.1323 ( -3.67%) 0.1395 ( -9.34%) 0.1236 ( 3.16%) 0.1240 ( 2.83%) Procs 20 0.1405 ( 0.00%) 0.1578 (-12.29%) 0.2452 (-74.52%) 0.1471 ( -4.73%) 0.1454 ( -3.50%) Procs 24 0.1823 ( 0.00%) 0.1800 ( 1.24%) 0.3030 (-66.22%) 0.1776 ( 2.58%) 0.1574 ( 13.63%) Procs 28 0.2019 ( 0.00%) 0.2143 ( -6.13%) 0.3403 (-68.52%) 0.2000 ( 0.94%) 0.1983 ( 1.78%) Procs 32 0.2162 ( 0.00%) 0.2329 ( -7.71%) 0.6526 (-201.85%) 0.2235 ( -3.36%) 0.2158 ( 0.20%) Procs 36 0.2354 ( 0.00%) 0.2577 ( -9.47%) 0.4468 (-89.77%) 0.2619 (-11.24%) 0.2451 ( -4.11%) Procs 40 0.2600 ( 0.00%) 0.2850 ( -9.62%) 0.5247 (-101.79%) 0.2724 ( -4.77%) 0.2646 ( -1.75%)The number of procs hackbench is running is too low here for a 48-core machine. It should have been reconfigured but this is better than nothing.
Quoted 7 lines
Quoted 3 lines
schednuma and autonuma both show large regressions in the performance here. I do not investigate why but as there are a number of scheduler changes it could be anything.
Strange, last time I tested hackbench it was perfectly ok, I even had this test shown in some of the pdf.
It's been rebased to 3.7-rc6 since so there may be an incompatible scheduler change somewhere.
Quoted 8 lines
Lately (post my last hackbench run) I disabled the affine wakeups cross-node and pipes use
sd_affinewakeups. That could matter for these heavy scheduling tests as it practically disables the_syncinwake_up_interruptible_sync_pollused by the pipe code, if the waker CPU is in a different node than the wakeeprev_cpu. I discussed this with Mike and he liked this change IIRC but it's the first thing that should be checked at the light of above regression.
Understood. I found in early profiles that the mutex_spin_on_owner logic was also relevant but did not pin down why. I expected it was contention on mmap_sem due to the PTE scanner but have not had the chance to verify.
Quoted 5 lines
Quoted 5 lines
PAGE FAULT TEST
This is a microbenchmark for page faults. The number of clients are badly ordered which again, I really should fix but anyway.
Quoted 7 lines
Quoted 5 lines
3.7.0 3.7.0 3.7.0 3.7.0 3.7.0 rc6-stats-v4r12 rc6-schednuma-v16r2rc6-autonuma-v28fastr3 rc6-moron-v4r38 rc6-twostage-v4r38 System 1 8.0710 ( 0.00%) 8.1085 ( -0.46%) 8.0925 ( -0.27%) 8.0170 ( 0.67%) 37.3075 (-362.24% System 10 9.4975 ( 0.00%) 9.5690 ( -0.75%) 12.0055 (-26.41%) 9.5915 ( -0.99%) 9.5835 ( -0.91%) System 11 9.7740 ( 0.00%) 9.7915 ( -0.18%) 13.4890 (-38.01%) 9.7275 ( 0.48%) 9.6810 ( 0.95%)No real clue on this one as I should look in what the test does.
It's the PFT test in MMTests and it should run it by default out of the box. Running it will fetch the relevant source and it'll be in work/testsdisk/sources
Quoted 11 lines
It
might be related to THP splits though. I can't imagine anything else because there's nothing at all in autonuma that alters the page faults (except from arming NUMA hinting faults which should be lighter in autonuma than in the other implementation using task work).Chances are the faults are tested by touching bytes at different 4k offsets in the same 2m naturally aligned virtual range.
Hugh THP native migration patch will clarify things on the above.
The current sets of tests been run has Hugh's THP native migration patch on top. There was a trivial conflict but otherwise it applied.
Quoted 8 lines
Quoted 3 lines
also hope that the concepts of autonuma would be reimplemented on top of this foundation so we can do a meaningful comparison between different placement policies.
I'll try to help with this to see what could be added from autonuma on top to improve on top your balancenuma foundation. Your current foundation looks ideal for inclusion to me.
That would be really great. If this happened then potentially numacore and autonuma can be directly compared in terms of placement and scheduler policies if both depended on the same underlying infrastrcture. If there was a better implementation of the PTE scanner for example then it should be usable by both.
Quoted 5 lines
I noticed you haven't run any single instance specjbb workload, that should be added to the battery of tests. But hey take your time, the amount of data you provided is already very comprehensive and you were so fast.
No, I haven't. Each time they got cancelled due to patch updates before they had a chance to run. They're still not queued because I want profiles for the other tests first. When they'll complete I'll fire them up.
Quoted 9 lines
The thing is: single instance and multi instance are totally different beasts.
multi instance is all about avoiding NUMA false sharing in the first place (the anti false sharing algorithm becomes a noop), and it has a trivial perfect solution with all cross node traffic guaranteed to stop after converence has been reached for the whole duration of the workload.
Quoted 7 lines
single instance is all about NUMA false sharing detection and it has no perfect solution and there's no way to fully converge and to stop all cross node traffic. So it's a tradeoff between doing too many CPU/memory spurious migrations (harmful, causes regressions) and doing too few (i.e. not improving at all compared to upstream but not regressing either).
Thanks for that explanation. It does mean that for any specjbb results that it'll have to be declared if it's single or multi configurations as they cannot be directly compared in a meaningful manner. If the majority of JVM tests are single-configuration then I'll prioritise those over the multi-JVM configurations just to have compatability in comparisons.
Quoted 11 lines
autonuma27/28/28fast will perform identical on multi instance loads (i.e. optimal, a few percent away from hard bindings).
I was just starting to improve the anti false sharing algorithm in autonuma28/28fast to improve single instance specjbb too (this is why you really need autonuma28 or autonuma28fast to test single instance specjbb and not autonuma27).
About THP, normally when I was running benchmarks I was testing these 4 configs:
Quoted 9 lines
1) THP on PMD scan on
2) THP on PMD scan off
3) THP off PMD scan on
4) THP off PMD scan off(1 and 2 are practically the same for the autonuma benchmark, because all memory is backed by THP rendering the PMD level hinting faults for 4k pages very unlikely, but I was testing it anyway just in case)
Testing the PMD and !PMD cases was important as I expect the results are different depending on whether the workload converges on a 2M boundary or not. A similar tunable is not available in my current tree but perhaps it should be added to allow the same comparison to happen.
Quoted 6 lines
THP off is going to hit KVM guests the most and much less host workloads. But even for KVM it's good practice to test with THP off too, to verify the cost of the numa hinting page faults remains very low (the cost is much higher for guests than host because of the vmexists).
Agreed.
Quoted 6 lines
The KVM benchmark run by IBM was also done in all 4 combinations: THP on/off KSM on/off and showed improvement even for the "No THP" case (btw, things should run much better these days than the old autonuma13).
Thanks for that.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Mel Gorman
Quoted 6 lines
[ SPECjbb transactions/sec ] | [ higher is better ] | | SPECjbb single-1x32 524k 507k | 638k +21.7%-----------------------------------------------------------------------
I was not able to run a full sets of tests today as I was distracted so all I have is a multi JVM comparison. I'll keep it shorter than average
Quoted 12 lines
3.7.0 3.7.0
rc5-stats-v4r2 rc5-schednuma-v16r1
TPut 1 101903.00 ( 0.00%) 77651.00 (-23.80%)
TPut 2 213825.00 ( 0.00%) 160285.00 (-25.04%)
TPut 3 307905.00 ( 0.00%) 237472.00 (-22.87%)
TPut 4 397046.00 ( 0.00%) 302814.00 (-23.73%)
TPut 5 477557.00 ( 0.00%) 364281.00 (-23.72%)
TPut 6 542973.00 ( 0.00%) 420810.00 (-22.50%)
TPut 7 540466.00 ( 0.00%) 448976.00 (-16.93%)
TPut 8 543226.00 ( 0.00%) 463568.00 (-14.66%)
TPut 9 513351.00 ( 0.00%) 468238.00 ( -8.79%)
TPut 10 484126.00 ( 0.00%) 457018.00 ( -5.60%)These figures are IMO way too low for a 64-way system. I have a 32-way system with midrange server CPUs and get 650k+/sec easily.
Have you tried to analyze the root cause, what does 'perf top' show during the run and how much idle time is there?
Trying to reproduce your findings I have done 4x JVM tests myself, using 4x 8-warehouse setups, with a sizing of -Xms8192m -Xmx8192m -Xss256k, and here are the results:
v3.7 v3.7 SPECjbb single-1x32 524k 638k +21.7% SPECjbb multi-4x8 633k 655k +3.4%
So while here we are only marginally better than the single-instance numbers (I will try to improve that in numa/core v17), they are still better than mainline - and they are definitely not slower as your numbers suggest ...
So we need to go back to the basics to figure this out: please outline exactly which commit ID of the numa/core tree you have booted. Also, how does 'perf top' look like on your box?
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 27 lines · Ingo Molnar
Quoted 11 lines · Mel Gorman
Quoted 6 lines
[ SPECjbb transactions/sec ] | [ higher is better ] | | SPECjbb single-1x32 524k 507k | 638k +21.7%-----------------------------------------------------------------------
I was not able to run a full sets of tests today as I was distracted so all I have is a multi JVM comparison. I'll keep it shorter than average
Quoted 12 lines
3.7.0 3.7.0 rc5-stats-v4r2 rc5-schednuma-v16r1 TPut 1 101903.00 ( 0.00%) 77651.00 (-23.80%) TPut 2 213825.00 ( 0.00%) 160285.00 (-25.04%) TPut 3 307905.00 ( 0.00%) 237472.00 (-22.87%) TPut 4 397046.00 ( 0.00%) 302814.00 (-23.73%) TPut 5 477557.00 ( 0.00%) 364281.00 (-23.72%) TPut 6 542973.00 ( 0.00%) 420810.00 (-22.50%) TPut 7 540466.00 ( 0.00%) 448976.00 (-16.93%) TPut 8 543226.00 ( 0.00%) 463568.00 (-14.66%) TPut 9 513351.00 ( 0.00%) 468238.00 ( -8.79%) TPut 10 484126.00 ( 0.00%) 457018.00 ( -5.60%)
Quoted 4 lines
These figures are IMO way too low for a 64-way system. I have a 32-way system with midrange server CPUs and get 650k+/sec easily.
48-way as I said here https://lkml.org/lkml/2012/11/3/109. If I said 64-way somewhere else, it was a mistake. The lack of THP would account for some of the difference. As I was looking for potential locking-related issues, I also had CONFIG_DEBUG_VMA nd CONFIG_DEBUG_MUTEXES set which would account for more overhead. Any options set are set for all tests that make up a group.
Quoted 3 lines
Have you tried to analyze the root cause, what does 'perf top' show during the run and how much idle time is there?
No, I haven't and the machine is currently occupied. However, a second profile run was run as part of the test above. The figures I reported are based on a run without profiling. With profiling, oprofile reported
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 6000
samples % image name app name symbol name 176552 42.9662 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 intel_idle 22790 5.5462 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 find_busiest_group 10533 2.5633 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 update_blocked_averages 10489 2.5526 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 rb_get_reader_page 9514 2.3154 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 native_write_msr_safe 8511 2.0713 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 ring_buffer_consume 7406 1.8023 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 idle_cpu 6549 1.5938 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 update_cfs_rq_blocked_load 6482 1.5775 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 rebalance_domains 5212 1.2684 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 run_rebalance_domains 5037 1.2258 perl perl /usr/bin/perl 4167 1.0141 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 page_fault 3885 0.9455 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 cpumask_next_and 3704 0.9014 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 find_next_bit 3498 0.8513 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 getnstimeofday 3345 0.8140 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 __update_cpu_load 3175 0.7727 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 load_balance 3018 0.7345 vmlinux-3.7.0-rc5-schednuma-v16r1 vmlinux-3.7.0-rc5-schednuma-v16r1 menu_select
Quoted 8 lines
Trying to reproduce your findings I have done 4x JVM tests myself, using 4x 8-warehouse setups, with a sizing of -Xms8192m -Xmx8192m -Xss256k, and here are the results:
v3.7 v3.7 SPECjbb single-1x32 524k 638k +21.7% SPECjbb multi-4x8 633k 655k +3.4%
I'll re-run with THP enabled the next time and see what I find.
Quoted 9 lines
So while here we are only marginally better than the single-instance numbers (I will try to improve that in numa/core v17), they are still better than mainline - and they are definitely not slower as your numbers suggest ...
So we need to go back to the basics to figure this out: please outline exactly which commit ID of the numa/core tree you have booted. Also, how does 'perf top' look like on your box?
I'll find out what perf top looks like ASAP.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 24 lines · Mel Gorman
I was not able to run a full sets of tests today as I was distracted so all I have is a multi JVM comparison. I'll keep it shorter than average
3.7.0 3.7.0 rc5-stats-v4r2 rc5-schednuma-v16r1 TPut 1 101903.00 ( 0.00%) 77651.00 (-23.80%) TPut 2 213825.00 ( 0.00%) 160285.00 (-25.04%) TPut 3 307905.00 ( 0.00%) 237472.00 (-22.87%) TPut 4 397046.00 ( 0.00%) 302814.00 (-23.73%) TPut 5 477557.00 ( 0.00%) 364281.00 (-23.72%) TPut 6 542973.00 ( 0.00%) 420810.00 (-22.50%) TPut 7 540466.00 ( 0.00%) 448976.00 (-16.93%) TPut 8 543226.00 ( 0.00%) 463568.00 (-14.66%) TPut 9 513351.00 ( 0.00%) 468238.00 ( -8.79%) TPut 10 484126.00 ( 0.00%) 457018.00 ( -5.60%) TPut 11 467440.00 ( 0.00%) 457999.00 ( -2.02%) TPut 12 430423.00 ( 0.00%) 447928.00 ( 4.07%) TPut 13 445803.00 ( 0.00%) 434823.00 ( -2.46%) TPut 14 427388.00 ( 0.00%) 430667.00 ( 0.77%) TPut 15 437183.00 ( 0.00%) 423746.00 ( -3.07%) TPut 16 423245.00 ( 0.00%) 416259.00 ( -1.65%) TPut 17 417666.00 ( 0.00%) 407186.00 ( -2.51%) TPut 18 413046.00 ( 0.00%) 398197.00 ( -3.59%)
Quoted 5 lines
This version of the patches manages to cripple performance entirely. I do not have a single JVM comparison available as the machine has been in use during the day. I accept that it is very possible that the single JVM figures are better.
I confirm that SPECjbb2005 1.07 -Xmx4g regresses in terms of throughput on my 16-way, 4 node system with 32GB of memory using 16 warehouses and 240 measurement seconds. I averaged the throughput for five runs on each kernel.
Java(TM) SE Runtime Environment (build 1.6.0_06-b02) Java HotSpot(TM) 64-Bit Server VM (build 10.0-b22, mixed mode)
Both kernels haveCONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
numa/core at 01aa90068b12 ("sched: Use the best-buddy 'ideal cpu' in balancing decisions") with
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=yCONFIG_ARCH_WANTS_NUMA_GENERIC_PGPROT=yCONFIG_NUMA_BALANCING=yCONFIG_NUMA_BALANCING_HUGEPAGE=yCONFIG_ARCH_USES_NUMA_GENERIC_PGPROT=yCONFIG_ARCH_USES_NUMA_GENERIC_PGPROT_HUGEPAGE=y
had a throughput of 128315.19 SPECjbb2005 bops.
numa/core at ec05a2311c35 ("Merge branch 'sched/urgent' into sched/core") had an average throughput of 136918.34 SPECjbb2005 bops, which is a 6.3% regression.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 23 lines · David Rientjes
I confirm that SPECjbb2005 1.07 -Xmx4g regresses in terms of throughput on my 16-way, 4 node system with 32GB of memory using 16 warehouses and 240 measurement seconds. I averaged the throughput for five runs on each kernel.
Java(TM) SE Runtime Environment (build 1.6.0_06-b02) Java HotSpot(TM) 64-Bit Server VM (build 10.0-b22, mixed mode)
Both kernels have
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=ynuma/core at
01aa90068b12("sched: Use the best-buddy 'ideal cpu' in balancing decisions") with
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=yCONFIG_ARCH_WANTS_NUMA_GENERIC_PGPROT=yCONFIG_NUMA_BALANCING=yCONFIG_NUMA_BALANCING_HUGEPAGE=yCONFIG_ARCH_USES_NUMA_GENERIC_PGPROT=yCONFIG_ARCH_USES_NUMA_GENERIC_PGPROT_HUGEPAGE=y
Quoted 6 lines
had a throughput of 128315.19 SPECjbb2005 bops.
numa/core at
ec05a2311c35("Merge branch 'sched/urgent' into sched/core") had an average throughput of 136918.34 SPECjbb2005 bops, which is a 6.3% regression.
perftop during the run on numa/core at 01aa90068b12 ("sched: Use the best-buddy 'ideal cpu' in balancing decisions"):
15.99% [kernel] [k] page_fault
4.05% [kernel] [k] getnstimeofday
3.96% [kernel] [k] _raw_spin_lock
3.20% [kernel] [k] rcu_check_callbacks
2.93% [kernel] [k] generic_smp_call_function_interrupt
2.90% [kernel] [k] __do_page_fault
2.82% [kernel] [k] ktime_get
2.62% [kernel] [k] read_tsc
2.41% [kernel] [k] handle_mm_fault
2.01% [kernel] [k] flush_tlb_func
1.99% [kernel] [k] retint_swapgs
1.83% [kernel] [k] emulate_vsyscall
1.71% [kernel] [k] handle_pte_fault
1.63% [kernel] [k] task_tick_fair
1.57% [kernel] [k] clear_page_c
1.55% [kernel] [k] down_read_trylock
1.54% [kernel] [k] copy_user_generic_string
1.48% [kernel] [k] ktime_get_update_offsets
1.37% [kernel] [k] find_vma
1.23% [kernel] [k] mpol_misplaced
1.14% [kernel] [k] task_numa_fault
1.10% [kernel] [k] run_timer_softirq
1.06% [kernel] [k] up_read
0.87% [kernel] [k] __bad_area_nosemaphore
0.82% [kernel] [k] write_ok_or_segv
0.77% [kernel] [k] update_cfs_shares
0.76% [kernel] [k] update_curr
0.75% [kernel] [k] error_sti
0.75% [kernel] [k] get_vma_policy
0.73% [kernel] [k] smp_call_function_many
0.66% [kernel] [k] do_wp_page
0.60% [kernel] [k] error_entry
0.60% [kernel] [k] call_function_interrupt
0.59% [kernel] [k] error_exit
0.58% [kernel] [k] _raw_spin_lock_irqsave
0.58% [kernel] [k] tick_sched_timer
0.57% [kernel] [k] __do_softirq
0.57% [kernel] [k] mem_cgroup_count_vm_event
0.56% [kernel] [k] account_user_time
0.56% [kernel] [k] spurious_fault
0.54% [kernel] [k] acct_update_integrals
0.54% [kernel] [k] bad_area_nosemaphore [ Both kernels for this test were booted with cgroup_disable=memory on
the command line, why mem_cgroup_count_vm_event shows up at all here is strange... ]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · David Rientjes
Quoted 3 lines
numa/core at
ec05a2311c35("Merge branch 'sched/urgent' into sched/core") had an average throughput of 136918.34 SPECjbb2005 bops, which is a 6.3% regression.perftop during the run on numa/core at
01aa90068b12("sched:
Use the best-buddy 'ideal cpu' in balancing decisions"):
Quoted 7 lines
15.99% [kernel] [k] page_fault
4.05% [kernel] [k] getnstimeofday
3.96% [kernel] [k] _raw_spin_lock
3.20% [kernel] [k] rcu_check_callbacks
2.93% [kernel] [k] generic_smp_call_function_interrupt
2.90% [kernel] [k] __do_page_fault
2.82% [kernel] [k] ktime_get Thanks for testing, that's very interesting - could you tell me more about exactly what kind of hardware this is? I'll try to find a similar system and reproduce the performance regression.
(A wild guess would be an older 4x Opteron system, 83xx series or so?)
Also, the profile looks weird to me. Here is how perf top looks like on my system during a similarly configured, "healthy" SPECjbb run:
91.29% perf-6687.map [.] 0x00007fffed1e8f21 4.81% libjvm.so [.] 0x00000000007004a0 0.93% [vdso] [.] 0x00007ffff7ffe60c 0.72% [kernel] [k] do_raw_spin_lock 0.36% [kernel] [k] generic_smp_call_function_interrupt 0.10% [kernel] [k] format_decode 0.07% [kernel] [k] rcu_check_callbacks 0.07% [kernel] [k] apic_timer_interrupt 0.07% [kernel] [k] call_function_interrupt 0.06% libc-2.15.so [.] __strcmp_sse42 0.06% [kernel] [k] irqtime_account_irq 0.06% perf [.] 0x000000000004bb7c 0.05% [kernel] [k] x86_pmu_disable_all 0.04% libc-2.15.so [.] __memcpy_ssse3 0.04% [kernel] [k] ktime_get 0.04% [kernel] [k] account_group_user_time 0.03% [kernel] [k] vbin_printf
and that is what SPECjbb does: it spends 97% of its time in Java code - yet there's no Java overhead visible in your profile - how is that possible? Could you try a newer perf on that box:
cd tools/perf/
make -j install
to make sure perf picks up the Java symbols as well (or at least includes them as a summary, as in the profile above). Note that no page fault overhead is visible in my profile.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 15 lines · Ingo Molnar
Quoted 7 lines
Quoted 3 lines
numa/core at
ec05a2311c35("Merge branch 'sched/urgent' into sched/core") had an average throughput of 136918.34 SPECjbb2005 bops, which is a 6.3% regression.perftop during the run on numa/core at
01aa90068b12("sched:
Use the best-buddy 'ideal cpu' in balancing decisions"):Quoted 7 lines
15.99% [kernel] [k] page_fault 4.05% [kernel] [k] getnstimeofday 3.96% [kernel] [k] _raw_spin_lock 3.20% [kernel] [k] rcu_check_callbacks 2.93% [kernel] [k] generic_smp_call_function_interrupt 2.90% [kernel] [k] __do_page_fault 2.82% [kernel] [k] ktime_get
Quoted 4 lines
Thanks for testing, that's very interesting - could you tell me more about exactly what kind of hardware this is? I'll try to find a similar system and reproduce the performance regression.
This happened to be an Opteron (but not 83xx series), 2.4Ghz. Your benchmarks were different in the number of cores but also in the amount of memory, do you think numa/core would regress because this is 32GB and not 64GB?
Quoted 3 lines
(A wild guess would be an older 4x Opteron system, 83xx series or so?)
Just curious, how you would guess that? Is there something about Opteron 83xx that make numa/core regress?
Quoted 22 lines
Also, the profile looks weird to me. Here is how perf top looks like on my system during a similarly configured, "healthy" SPECjbb run:
91.29% perf-6687.map [.] 0x00007fffed1e8f21 4.81% libjvm.so [.] 0x00000000007004a0 0.93% [vdso] [.] 0x00007ffff7ffe60c 0.72% [kernel] [k] do_raw_spin_lock 0.36% [kernel] [k] generic_smp_call_function_interrupt 0.10% [kernel] [k] format_decode 0.07% [kernel] [k] rcu_check_callbacks 0.07% [kernel] [k] apic_timer_interrupt 0.07% [kernel] [k] call_function_interrupt 0.06% libc-2.15.so [.] __strcmp_sse42 0.06% [kernel] [k] irqtime_account_irq 0.06% perf [.] 0x000000000004bb7c 0.05% [kernel] [k] x86_pmu_disable_all 0.04% libc-2.15.so [.] __memcpy_ssse3 0.04% [kernel] [k] ktime_get 0.04% [kernel] [k] account_group_user_time 0.03% [kernel] [k] vbin_printf
Quoted 4 lines
and that is what SPECjbb does: it spends 97% of its time in Java code - yet there's no Java overhead visible in your profile - how is that possible? Could you try a newer perf on that box:
It's perf top -U, the benchmark itself was unchanged so I didn't think it was interesting to gather the user symbols. If that would be helpful, let me know!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 17 lines · David Rientjes
Quoted 15 lines · Ingo Molnar
Quoted 14 lines
Quoted 3 lines
numa/core at
ec05a2311c35("Merge branch 'sched/urgent' into sched/core") had an average throughput of 136918.34 SPECjbb2005 bops, which is a 6.3% regression.perftop during the run on numa/core at
01aa90068b12("sched:
Use the best-buddy 'ideal cpu' in balancing decisions"):15.99% [kernel] [k] page_fault 4.05% [kernel] [k] getnstimeofday 3.96% [kernel] [k] _raw_spin_lock 3.20% [kernel] [k] rcu_check_callbacks 2.93% [kernel] [k] generic_smp_call_function_interrupt 2.90% [kernel] [k] __do_page_fault 2.82% [kernel] [k] ktime_get
Quoted 6 lines
Quoted 4 lines
Thanks for testing, that's very interesting - could you tell me more about exactly what kind of hardware this is? I'll try to find a similar system and reproduce the performance regression.
This happened to be an Opteron (but not 83xx series), 2.4Ghz.
Ok - roughly which family/model from /proc/cpuinfo?
Quoted 3 lines
Your benchmarks were different in the number of cores but also in the amount of memory, do you think numa/core would regress because this is 32GB and not 64GB?
I'd not expect much sensitivity to RAM size.
Quoted 4 lines
Quoted 2 lines
(A wild guess would be an older 4x Opteron system, 83xx series or so?)
Just curious, how you would guess that? [...]
I'm testing numa/core on many systems and the performance figures seemed to roughly map to that range.
Quoted 2 lines
[...] Is there something about Opteron 83xx that make numa/core regress?
Not that I knew of - but apparently there is! I'll try to find a system that matches yours as closely as possible and have a look.
Quoted 22 lines
Quoted 22 lines
Also, the profile looks weird to me. Here is how perf top looks like on my system during a similarly configured, "healthy" SPECjbb run:
91.29% perf-6687.map [.] 0x00007fffed1e8f21 4.81% libjvm.so [.] 0x00000000007004a0 0.93% [vdso] [.] 0x00007ffff7ffe60c 0.72% [kernel] [k] do_raw_spin_lock 0.36% [kernel] [k] generic_smp_call_function_interrupt 0.10% [kernel] [k] format_decode 0.07% [kernel] [k] rcu_check_callbacks 0.07% [kernel] [k] apic_timer_interrupt 0.07% [kernel] [k] call_function_interrupt 0.06% libc-2.15.so [.] __strcmp_sse42 0.06% [kernel] [k] irqtime_account_irq 0.06% perf [.] 0x000000000004bb7c 0.05% [kernel] [k] x86_pmu_disable_all 0.04% libc-2.15.so [.] __memcpy_ssse3 0.04% [kernel] [k] ktime_get 0.04% [kernel] [k] account_group_user_time 0.03% [kernel] [k] vbin_printf
Quoted 8 lines
Quoted 4 lines
and that is what SPECjbb does: it spends 97% of its time in Java code - yet there's no Java overhead visible in your profile - how is that possible? Could you try a newer perf on that box:
It's perf top -U, the benchmark itself was unchanged so I didn't think it was interesting to gather the user symbols. If that would be helpful, let me know!
Yeah, regular perf top output would be very helpful to get a general sense of proportion. Thanks!
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 43 lines · Ingo Molnar
Quoted 17 lines · David Rientjes
Quoted 15 lines · Ingo Molnar
Quoted 14 lines
Quoted 3 lines
numa/core at
ec05a2311c35("Merge branch 'sched/urgent' into sched/core") had an average throughput of 136918.34 SPECjbb2005 bops, which is a 6.3% regression.perftop during the run on numa/core at
01aa90068b12("sched:
Use the best-buddy 'ideal cpu' in balancing decisions"):15.99% [kernel] [k] page_fault 4.05% [kernel] [k] getnstimeofday 3.96% [kernel] [k] _raw_spin_lock 3.20% [kernel] [k] rcu_check_callbacks 2.93% [kernel] [k] generic_smp_call_function_interrupt 2.90% [kernel] [k] __do_page_fault 2.82% [kernel] [k] ktime_getQuoted 6 lines
Quoted 4 lines
Thanks for testing, that's very interesting - could you tell me more about exactly what kind of hardware this is? I'll try to find a similar system and reproduce the performance regression.
This happened to be an Opteron (but not 83xx series), 2.4Ghz.
Ok - roughly which family/model from
/proc/cpuinfo?Quoted 3 lines
Your benchmarks were different in the number of cores but also in the amount of memory, do you think numa/core would regress because this is 32GB and not 64GB?
I'd not expect much sensitivity to RAM size.
Quoted 4 lines
Quoted 2 lines
(A wild guess would be an older 4x Opteron system, 83xx series or so?)
Just curious, how you would guess that? [...]
I'm testing numa/core on many systems and the performance figures seemed to roughly map to that range.
Quoted 6 lines
Quoted 2 lines
[...] Is there something about Opteron 83xx that make numa/core regress?
Not that I knew of - but apparently there is! I'll try to find a system that matches yours as closely as possible and have a look.
Here I'd note the node-distances that David included above. This system is not fully connected, having an (asymmetric) kite topology. Only nodes nodes 1 and 2 are fully connected.
This is sufficiently whacky that it seems a likely candidate :-).
- Paul
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Also, the profile looks weird to me. Here is how perf top looks like on my system during a similarly configured, "healthy" SPECjbb run:
91.29% perf-6687.map [.] 0x00007fffed1e8f21 4.81% libjvm.so [.] 0x00000000007004a0 0.93% [vdso] [.] 0x00007ffff7ffe60c 0.72% [kernel] [k] do_raw_spin_lock 0.36% [kernel] [k] generic_smp_call_function_interrupt 0.10% [kernel] [k] format_decode 0.07% [kernel] [k] rcu_check_callbacks 0.07% [kernel] [k] apic_timer_interrupt 0.07% [kernel] [k] call_function_interrupt 0.06% libc-2.15.so [.] __strcmp_sse42 0.06% [kernel] [k] irqtime_account_irq 0.06% perf [.] 0x000000000004bb7c 0.05% [kernel] [k] x86_pmu_disable_all 0.04% libc-2.15.so [.] __memcpy_ssse3 0.04% [kernel] [k] ktime_get 0.04% [kernel] [k] account_group_user_time 0.03% [kernel] [k] vbin_printfand that is what SPECjbb does: it spends 97% of its time in Java code - yet there's no Java overhead visible in your profile - how is that possible? Could you try a newer perf on that box:
Quoted 8 lines
Quoted 3 lines
It's perf top -U, the benchmark itself was unchanged so I didn't think it was interesting to gather the user symbols. If that would be helpful, let me know!
Yeah, regular perf top output would be very helpful to get a general sense of proportion. Thanks!
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines · Ingo Molnar
Quoted 1 line
This happened to be an Opteron (but not 83xx series), 2.4Ghz.
Ok - roughly which family/model from
/proc/cpuinfo?
It's close enough, it's 23xx.
Quoted 7 lines
Quoted 3 lines
It's perf top -U, the benchmark itself was unchanged so I didn't think it was interesting to gather the user symbols. If that would be helpful, let me know!
Yeah, regular perf top output would be very helpful to get a general sense of proportion. Thanks!
Ok, here it is:
91.24% perf-10971.map [.] 0x00007f116a6c6fb8
1.19% libjvm.so [.] instanceKlass::oop_push_contents(PSPromotionMa
1.04% libjvm.so [.] PSPromotionManager::drain_stacks_depth(bool)
0.79% libjvm.so [.] PSPromotionManager::copy_to_survivor_space(oop
0.60% libjvm.so [.] PSPromotionManager::claim_or_forward_internal_
0.58% [kernel] [k] page_fault
0.28% libc-2.3.6.so [.] __gettimeofday
0.26% libjvm.so [.] Copy::pd_disjoint_words(HeapWord*, HeapWord*, unsigned
0.22% [kernel] [k] getnstimeofday
0.18% libjvm.so [.] CardTableExtension::scavenge_contents_parallel(ObjectS
0.15% [kernel] [k] _raw_spin_lock
0.12% [kernel] [k] ktime_get_update_offsets
0.11% [kernel] [k] ktime_get
0.11% [kernel] [k] rcu_check_callbacks
0.10% [kernel] [k] generic_smp_call_function_interrupt
0.10% [kernel] [k] read_tsc
0.10% [kernel] [k] clear_page_c
0.10% [kernel] [k] __do_page_fault
0.08% [kernel] [k] handle_mm_fault
0.08% libjvm.so [.] os::javaTimeMillis()
0.08% [kernel] [k] emulate_vsyscall
0.08% [kernel] [k] flush_tlb_func
0.07% [kernel] [k] task_tick_fair
0.07% [kernel] [k] retint_swapgs
0.06% libjvm.so [.] oopDesc::size_given_klass(Klass*)
0.05% [kernel] [k] handle_pte_fault
0.05% perf [.] 0x0000000000033190
0.05% sanctuaryd [.] 0x00000000006f0ad3
0.05% [kernel] [k] copy_user_generic_string
0.05% libjvm.so [.] objArrayKlass::oop_push_contents(PSPromotionManager*,
0.04% [kernel] [k] find_vma
0.04% [kernel] [k] mpol_misplaced
0.04% [kernel] [k] __bad_area_nosemaphore
0.04% [kernel] [k] get_vma_policy
0.04% [kernel] [k] task_numa_fault
0.04% [kernel] [k] error_sti
0.03% [kernel] [k] write_ok_or_segv
0.03% [kernel] [k] do_gettimeofday
0.03% [kernel] [k] down_read_trylock
0.03% [kernel] [k] update_cfs_shares
0.03% [kernel] [k] error_entry
0.03% [kernel] [k] run_timer_softirq--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · David Rientjes
Quoted 3 lines · Ingo Molnar
Quoted 1 line
This happened to be an Opteron (but not 83xx series), 2.4Ghz.
Ok - roughly which family/model from
/proc/cpuinfo?It's close enough, it's 23xx.
Ok - which family/model number in /proc/cpuinfo?
I'm asking because that will matter most to page fault micro-characteristics and the 23xx series existed in Barcelona form as well (family/model of 16/2) and it still exists in its current Shanghai form as well.
My guess is Barcelona 16/2?
If that is correct then the closest I can get to your topology is a 4-socket 32-way Opteron system with 32 GB of RAM - which seems close enough for testing purposes.
But checking numa/core on such a system still keeps me absolutely puzzled, as I get the following with a similar 16-warehouses SPECjbb 2005 test, using java -Xms8192m -Xmx8192m -Xss256k sizing, THP enabled, 2x 240 seconds runs (I tried to configure it all very close to yours), using -tip-a07005cbd847:
kernel warehouses transactions/sec
--------- ----------
v3.7-rc6: 16 197802
16 197997
numa/core: 16 203086
16 203967So sadly numa/core is about 2%-3% faster on this 4x4 system too! :-/
But I have to say, your SPECjbb score is uncharacteristically low even for an oddball-topology Barcelona system - which is the oldest/slowest system I can think of. So there might be more to this.
To further characterise a "good" SPECjbb run, there's no page_fault overhead visible in perf top:
Mainline profile:
94.99% perf-1244.map [.] 0x00007f04cd1aa523
2.52% libjvm.so [.] 0x00000000007004a1
0.62% [vdso] [.] 0x0000000000000972
0.31% [kernel] [k] clear_page_c
0.17% [kernel] [k] timekeeping_get_ns.constprop.7
0.11% [kernel] [k] rep_nop
0.09% [kernel] [k] ktime_get
0.08% [kernel] [k] get_cycles
0.06% [kernel] [k] read_tsc
0.05% libc-2.15.so [.] __strcmp_sse2 numa/core profile:
95.66% perf-1201.map [.] 0x00007fe4ad1c8fc7
1.70% libjvm.so [.] 0x0000000000381581
0.59% [vdso] [.] 0x0000000000000607
0.19% [kernel] [k] do_raw_spin_lock
0.11% [kernel] [k] generic_smp_call_function_interrupt
0.11% [kernel] [k] timekeeping_get_ns.constprop.7
0.08% [kernel] [k] ktime_get
0.06% [kernel] [k] get_cycles
0.05% [kernel] [k] __native_flush_tlb
0.05% [kernel] [k] rep_nop
0.04% perf [.] add_hist_entry.isra.9
0.04% [kernel] [k] rcu_check_callbacks
0.04% [kernel] [k] ktime_get_update_offsets
0.04% libc-2.15.so [.] __strcmp_sse2 No page fault overhead (see the page fault rate further below) - the NUMA scanning overhead shows up only through some mild TLB flush activity (which I'll fix btw).
[ Stupid question: cpufreq is configured to always-2.4GHz,
right? If you could send me your kernel config (you can do that privately as well) then I can try to boot it and see. ]
Quoted 22 lines
Quoted 6 lines
Quoted 3 lines
It's perf top -U, the benchmark itself was unchanged so I didn't think it was interesting to gather the user symbols. If that would be helpful, let me know!
Yeah, regular perf top output would be very helpful to get a general sense of proportion. Thanks!
Ok, here it is:
91.24% perf-10971.map [.] 0x00007f116a6c6fb8 1.19% libjvm.so [.] instanceKlass::oop_push_contents(PSPromotionMa 1.04% libjvm.so [.] PSPromotionManager::drain_stacks_depth(bool) 0.79% libjvm.so [.] PSPromotionManager::copy_to_survivor_space(oop 0.60% libjvm.so [.] PSPromotionManager::claim_or_forward_internal_ 0.58% [kernel] [k] page_fault 0.28% libc-2.3.6.so [.] __gettimeofday 0.26% libjvm.so [.] Copy::pd_disjoint_words(HeapWord*, HeapWord*, unsigned 0.22% [kernel] [k] getnstimeofday 0.18% libjvm.so [.] CardTableExtension::scavenge_contents_parallel(ObjectS 0.15% [kernel] [k] _raw_spin_lock 0.12% [kernel] [k] ktime_get_update_offsets 0.11% [kernel] [k] ktime_get
Quoted 8 lines
0.11% [kernel] [k] rcu_check_callbacks
0.10% [kernel] [k] generic_smp_call_function_interrupt
0.10% [kernel] [k] read_tsc
0.10% [kernel] [k] clear_page_c
0.10% [kernel] [k] __do_page_fault
0.08% [kernel] [k] handle_mm_fault
0.08% libjvm.so [.] os::javaTimeMillis()
0.08% [kernel] [k] emulate_vsyscall Oh, finally a clue: you seem to have vsyscall emulation overhead!
Vsyscall emulation is fundamentally page fault driven - which might explain why you are seeing page fault overhead. It might also interact with other sources of faults - such as numa/core's working set probing ...
Many JVMs try to be smart with the vsyscall. As a test, does the vsyscall=native boot option change the results/behavior in any way?
Stupid question, if you apply the patch attached below and if you do page fault profiling while the run is in steady state:
perf record -e faults -g -a sleep 10
do you see it often coming from the vsyscall page?
Also, this:
perf stat -e faults -a --repeat 10 sleep 1
should normally report something like this during SPECjbb steady state, numa/core:
warmup: 3,895 faults/sec ( +- 12.11% )
steady state: 3,910 faults/sec ( +- 6.72% )Which is about 250 faults/sec/CPU - i.e. it should be barely recognizable in profiles - let alone be prominent as in yours.
Thanks,
Ingo
--- arch/x86/mm/fault.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/mm/fault.c +3 -2
@@ -1030,6 +1030,9 @@ __do_page_fault(struct pt_regs *regs, un/* Get the faulting address: */ address =read_cr2 (); +/* Instrument as early as possible: */ +perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); +/* * Detect and handle instructions that would cause a page fault for * both a tracked kernel page and a userspace page.
@@ -1107,8 +1110,6 @@ __do_page_fault(struct pt_regs *regs, un } } -perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); -/* * If we're in an interrupt, have no user context or are running * in an atomic region then we must not take the fault:
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/1 x86/vsyscall: Add Kconfig option to use native vsyscalls, switch to it
Quoted 8 lines · Ingo Molnar
Quoted 4 lines
0.10% [kernel] [k] __do_page_fault 0.08% [kernel] [k] handle_mm_fault 0.08% libjvm.so [.] os::javaTimeMillis() 0.08% [kernel] [k] emulate_vsyscallOh, finally a clue: you seem to have vsyscall emulation overhead!
Quoted 8 lines
Vsyscall emulation is fundamentally page fault driven - which might explain why you are seeing page fault overhead. It might also interact with other sources of faults - such as numa/core's working set probing ...
Many JVMs try to be smart with the vsyscall. As a test, does the vsyscall=native boot option change the results/behavior in any way?
As a blind shot into the dark, does the attached patch help?
If that's the root cause then it should measurably help mainline SPECjbb performance as well. It could turn numa/core from a regression into a win on your system.
Thanks,
Ingo
----------------->
Subject: x86/vsyscall: Add Kconfig option to use native vsyscalls, switch to it
From: Ingo Molnar <[email protected]>
Apparently there's still plenty of systems out there triggering the vsyscall emulation page faults - causing hard to track down performance regressions on page fault intense workloads...
Some people seem to have run into that with threading-intense Java workloads.
So until there's a better solution to this, add a Kconfig switch to make the vsyscall mode configurable and turn native vsyscall support back on by default.
Distributions whose libraries and applications use the vDSO and never access the vsyscall page can change the config option to off.
Note, I don't think we want to expose the "none" option via a Kconfig switch - it breaks the ABI. So it's "native" versus "emulate", with "none" being available as a kernel boot option, for the super paranoid.
For more background, see these upstream commits:
3ae36655b97a x86-64: Rework vsyscall emulation and add vsyscall= parameter 5cec93c216db x86-64: Emulate legacy vsyscalls
--- arch/x86/Kconfig | 21 +++++++++++++++++++++ arch/x86/kernel/vsyscall_64.c | 8 +++++++- 2 files changed, 28 insertions(+), 1 deletion(-)
arch/x86/Kconfig +21 -0
@@ -2234,6 +2234,27 @@ config X86_X32 elf32_x86_64 support enabled to compile a kernel with this option set. +config X86_VSYSCALL_COMPAT + bool "vsyscall compatibility" + default y + help + vsyscalls, as global executable pages, can be a security hole + escallation helper by exposing an easy shell code target with + a predictable address. + + Many versions of glibc rely on the vsyscall page though, so it + cannot be eliminated unconditionally. If you disable this + option these systems will still work but might incur the overhead + of vsyscall emulation page faults. + + The vsyscall=none, vsyscall=emulate, vsyscall=native kernel boot + option can be used to override this mode as well. + + Keeping this option enabled leaves the vsyscall page enabled, + i.e. vsyscall=native. Disabling this option means vsyscall=emulate. + + If unsure, say Y. + config COMPAT def_bool y depends on IA32_EMULATION || X86_X32
arch/x86/kernel/vsyscall_64.c +7 -1
@@ -56,7 +56,13 @@DEFINE_VVAR (int, vgetcpu_mode); DEFINE_VVAR(struct vsyscall_gtod_data, vsyscall_gtod_data); -static enum {EMULATE ,NATIVE ,NONE } vsyscall_mode =EMULATE ; +static enum { EMULATE, NATIVE, NONE } vsyscall_mode = +#ifdef CONFIG_X86_VSYSCALL_COMPAT +NATIVE +#else + EMULATE +#endif +;static int __init vsyscall_setup (char *str) {
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [patch] x86/vsyscall: Add Kconfig option to use native vsyscalls, switch to it
Quoted 38 lines · Ingo Molnar
Quoted 8 lines · Ingo Molnar
Quoted 4 lines
0.10% [kernel] [k] __do_page_fault 0.08% [kernel] [k] handle_mm_fault 0.08% libjvm.so [.] os::javaTimeMillis() 0.08% [kernel] [k] emulate_vsyscallOh, finally a clue: you seem to have vsyscall emulation overhead!
Quoted 8 lines
Vsyscall emulation is fundamentally page fault driven - which might explain why you are seeing page fault overhead. It might also interact with other sources of faults - such as numa/core's working set probing ...
Many JVMs try to be smart with the vsyscall. As a test, does the vsyscall=native boot option change the results/behavior in any way?
As a blind shot into the dark, does the attached patch help?
If that's the root cause then it should measurably help mainline SPECjbb performance as well. It could turn numa/core from a regression into a win on your system.
Thanks,
Ingo
----------------->
Subject: x86/vsyscall: AddKconfigoption to use native vsyscalls, switch to it
From: Ingo Molnar <[email protected]>Apparently there's still plenty of systems out there triggering the vsyscall emulation page faults - causing hard to track down performance regressions on page fault intense workloads...
Quoted 7 lines
Some people seem to have run into that with threading-intense Java workloads.
So until there's a better solution to this, add a
Kconfigswitch to make the vsyscall mode configurable and turn native vsyscall support back on by default.
I'm not sure the default should be changed. Presumably only a smallish minority of users are affected, and all of their code still *works* -- it's just a little bit slower.
arch/x86/Kconfig config X86_X32 · line 2240
Quoted 5 lines
arch/x86/Kconfig +4
-0+config X86_VSYSCALL_COMPAT + bool "vsyscall compatibility" + default y + help
This is IMO misleading. Perhaps the option should be X86_VSYSCALL_EMULATION. A description like "compatibility" makes turning it on sound like a no-brainer.
Perhaps the vsyscall emulation code should be tweaked to warn if it's getting called more than, say, 1k times per second. The kernel could log something like "Detected large numbers of emulated vsyscalls. Consider upgading, setting vsyscall=native, or adjusting CONFIG_X86_WHATEVER."
arch/x86/Kconfig config X86_X32 · line 2242
Quoted 2 lines
arch/x86/Kconfig +2
-0+ vsyscalls, as global executable pages, can be a security hole + escallation helper by exposing an easy shell code target with
escalation?
Quoted 5 lines
arch/x86/Kconfig +5
-0+ a predictable address. + + Many versions of glibc rely on the vsyscall page though, so it + cannot be eliminated unconditionally. If you disable this + option these systems will still work but might incur the overhead
arch/x86/Kconfig config X86_X32 · line 2257
Quoted 10 lines
arch/x86/Kconfig +10
-0+ of vsyscall emulation page faults. + + The vsyscall=none, vsyscall=emulate, vsyscall=native kernel boot + option can be used to override this mode as well. + + Keeping this option enabled leaves the vsyscall page enabled, + i.e. vsyscall=native. Disabling this option means vsyscall=emulate. + + If unsure, say Y. +
--Andy
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [patch] x86/vsyscall: Add Kconfig option to use native vsyscalls, switch to it
In reply to Ingo MolnarQuoted 21 lines · Ingo Molnar
Subject: x86/vsyscall: Add
Kconfigoption to use native vsyscalls, switch to it
From: Ingo Molnar <[email protected]>Apparently there's still plenty of systems out there triggering the vsyscall emulation page faults - causing hard to track down performance regressions on page fault intense workloads...
Some people seem to have run into that with threading-intense Java workloads.
So until there's a better solution to this, add a
Kconfigswitch to make the vsyscall mode configurable and turn native vsyscall support back on by default.Distributions whose libraries and applications use the vDSO and never access the vsyscall page can change the config option to off.
Note, I don't think we want to expose the "none" option via a
Kconfigswitch - it breaks the ABI. So it's "native" versus "emulate", with "none" being available as a kernel boot option, for the super paranoid.
Quoted 8 lines
For more background, see these upstream commits:
3ae36655b97ax86-64: Rework vsyscall emulation and add vsyscall= parameter5cec93c216dbx86-64: Emulate legacy vsyscallsCc Andy Lutomirski <[email protected]>Cc Linus Torvalds <[email protected]>Signed-off-by Ingo Molnar <[email protected]>
It's slightly better, not sure if it's worth changing everybody's default for the small speedup here, though.
THP enabled:
numa/core at ec05a2311c35: 136918.34 SPECjbb2005 bops
numa/core at 01aa90068b12: 128315.19 SPECjbb2005 bops (-6.3%)
numa/core at 01aa90068b12 + patch: 128589.34 SPECjbb2005 bops (-6.1%)
THP disabled:
numa/core at ec05a2311c35: 122246.90 SPECjbb2005 bops
numa/core at 01aa90068b12: 99389.49 SPECjbb2005 bops (-18.7%)
numa/core at 01aa90068b12 + patch: 100726.34 SPECjbb2005 bops (-17.6%)
perf top w/ THP enabled:
92.56% perf-13343.map [.] 0x00007fd513d2ba7a
1.24% libjvm.so [.] instanceKlass::oop_push_contents(PSPromoti
1.01% libjvm.so [.] PSPromotionManager::drain_stacks_depth(boo
0.77% libjvm.so [.] PSPromotionManager::copy_to_survivor_space
0.57% libjvm.so [.] PSPromotionManager::claim_or_forward_inter
0.26% libjvm.so [.] Copy::pd_disjoint_words(HeapWord*, HeapWord*, unsi
0.22% libjvm.so [.] CardTableExtension::scavenge_contents_parallel(Obj
0.20% [kernel] [k] read_tsc
0.15% [kernel] [k] _raw_spin_lock
0.13% [kernel] [k] getnstimeofday
0.12% [kernel] [k] page_fault
0.11% [kernel] [k] generic_smp_call_function_interrupt
0.10% [kernel] [k] ktime_get
0.10% [kernel] [k] rcu_check_callbacks
0.09% [kernel] [k] ktime_get_update_offsets
0.09% libjvm.so [.] objArrayKlass::oop_push_contents(PSPromotionManage
0.08% [kernel] [k] flush_tlb_func
0.07% [kernel] [k] system_call
0.07% libjvm.so [.] oopDesc::size_given_klass(Klass*)
0.06% [kernel] [k] handle_mm_fault
0.06% [kernel] [k] task_tick_fair
0.05% libc-2.3.6.so [.] __gettimeofday
0.05% libjvm.so [.] os::javaTimeMillis()
0.05% [kernel] [k] handle_pte_fault
0.05% [kernel] [k] system_call_after_swapgs
0.04% [kernel] [k] mpol_misplaced
0.04% [kernel] [k] __do_page_fault
0.04% perf [.] 0x0000000000035903
0.04% [kernel] [k] copy_user_generic_string
0.04% [kernel] [k] task_numa_fault
0.04% [acpi_cpufreq] [.] 0x000000005f51a009
0.04% [kernel] [k] find_vma
0.04% [kernel] [k] run_timer_softirq
0.03% [kernel] [k] sysret_check
0.03% [kernel] [k] smp_call_function_many
0.03% [kernel] [k] update_cfs_shares
0.02% [kernel] [k] do_wp_page
0.02% [kernel] [k] get_vma_policy
0.02% [kernel] [k] update_curr
0.02% [kernel] [k] down_read_trylock
0.02% [kernel] [k] apic_timer_interrupt--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/1 x86/mm: Don't flush the TLB on #WP pmd fixups
In reply to Ingo MolnarQuoted 17 lines · Ingo Molnar
numa/core profile:
95.66% perf-1201.map [.] 0x00007fe4ad1c8fc7 1.70% libjvm.so [.] 0x0000000000381581 0.59% [vdso] [.] 0x0000000000000607 0.19% [kernel] [k] do_raw_spin_lock 0.11% [kernel] [k] generic_smp_call_function_interrupt 0.11% [kernel] [k] timekeeping_get_ns.constprop.7 0.08% [kernel] [k] ktime_get 0.06% [kernel] [k] get_cycles 0.05% [kernel] [k] __native_flush_tlb 0.05% [kernel] [k] rep_nop 0.04% perf [.] add_hist_entry.isra.9 0.04% [kernel] [k] rcu_check_callbacks 0.04% [kernel] [k] ktime_get_update_offsets 0.04% libc-2.15.so [.] __strcmp_sse2
Quoted 3 lines
No page fault overhead (see the page fault rate further below)
- the NUMA scanning overhead shows up only through some mild TLB flush activity (which I'll fix btw).
The patch attached below should get rid of that mild TLB flushing activity as well.
Thanks,
Ingo
--------------------------->
Subject: x86/mm: Don't flush the TLB on #WP pmd fixups
From: Ingo Molnar <[email protected]>
Date: Tue Nov 20 14:46:34 CET 2012
If we have a write protection #PF and fix up the pmd then the hugetlb code [the only user of pmdp_set_access_flags], in its do_huge_pmd_wp_page() page fault resolution function calls pmdp_set_access_flags() to mark the pmd permissive again, and flushes the TLB.
This TLB flush is unnecessary: a flush on #PF is guaranteed on most (all?) x86 CPUs, and even in the worst-case we'll generate a spurious fault.
So remove it.
--- arch/x86/mm/pgtable.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/x86/mm/pgtable.c +6 -1
@@ -334,7 +334,12 @@ int pmdp_set_access_flags(struct vm_areaif (changed && dirty) { *pmdp = entry;pmd_update_defer (vma->vm_mm, address, pmdp); -flush_tlb_range (vma, address, address +HPAGE_PMD_SIZE ); +/* +* We had a write-protection fault here and changed the pmd +* to to more permissive. No need to flush the TLB for that, +* #PF is architecturally guaranteed to do that and in the +* worst-case we'll generate a spurious fault. +*/ }return changed;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] x86/mm: Don't flush the TLB on #WP pmd fixups
Quoted 19 lines · Ingo Molnar
Quoted 17 lines · Ingo Molnar
numa/core profile:
95.66% perf-1201.map [.] 0x00007fe4ad1c8fc7 1.70% libjvm.so [.] 0x0000000000381581 0.59% [vdso] [.] 0x0000000000000607 0.19% [kernel] [k] do_raw_spin_lock 0.11% [kernel] [k] generic_smp_call_function_interrupt 0.11% [kernel] [k] timekeeping_get_ns.constprop.7 0.08% [kernel] [k] ktime_get 0.06% [kernel] [k] get_cycles 0.05% [kernel] [k] __native_flush_tlb 0.05% [kernel] [k] rep_nop 0.04% perf [.] add_hist_entry.isra.9 0.04% [kernel] [k] rcu_check_callbacks 0.04% [kernel] [k] ktime_get_update_offsets 0.04% libc-2.15.so [.] __strcmp_sse2
Quoted 6 lines
Quoted 3 lines
No page fault overhead (see the page fault rate further below)
- the NUMA scanning overhead shows up only through some mild TLB flush activity (which I'll fix btw).The patch attached below should get rid of that mild TLB flushing activity as well.
This has further increased SPECjbb from 203k/sec to 207k/sec, i.e. it's now 5% faster than mainline - THP enabled.
The profile is now totally flat even during a full 32-WH SPECjbb run, with the highest overhead entries left all related to timer IRQ processing or profiling. That is on a system that should be very close to yours.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: [PATCH] x86/mm: Don't flush the TLB on #WP pmd fixups
Quoted 29 lines · Ingo Molnar
Quoted 19 lines · Ingo Molnar
Quoted 17 lines · Ingo Molnar
numa/core profile:
95.66% perf-1201.map [.] 0x00007fe4ad1c8fc7 1.70% libjvm.so [.] 0x0000000000381581 0.59% [vdso] [.] 0x0000000000000607 0.19% [kernel] [k] do_raw_spin_lock 0.11% [kernel] [k] generic_smp_call_function_interrupt 0.11% [kernel] [k] timekeeping_get_ns.constprop.7 0.08% [kernel] [k] ktime_get 0.06% [kernel] [k] get_cycles 0.05% [kernel] [k] __native_flush_tlb 0.05% [kernel] [k] rep_nop 0.04% perf [.] add_hist_entry.isra.9 0.04% [kernel] [k] rcu_check_callbacks 0.04% [kernel] [k] ktime_get_update_offsets 0.04% libc-2.15.so [.] __strcmp_sse2Quoted 6 lines
Quoted 3 lines
No page fault overhead (see the page fault rate further below)
- the NUMA scanning overhead shows up only through some mild TLB flush activity (which I'll fix btw).The patch attached below should get rid of that mild TLB flushing activity as well.
Quoted 8 lines
This has further increased SPECjbb from 203k/sec to 207k/sec, i.e. it's now 5% faster than mainline - THP enabled.
The profile is now totally flat even during a full 32-WH SPECjbb run, with the highest overhead entries left all related to timer IRQ processing or profiling. That is on a system that should be very close to yours.
This is a stab in the dark but are you always running with profiling enabled?
I have not checked this with perf but a number of years ago I found that oprofile could distort results really badly (7-30% depending on the workload at the time) when I was evalating hugetlbfs and THP. In some cases I would find that profiling would show that a patch series improved performance when the same series showed regressions if profiling was disabled. The sampling rate had to be reduced quite a bit to avoid this effect.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] x86/mm: Don't flush the TLB on #WP pmd fixups
In reply to Ingo MolnarQuoted 10 lines · Ingo Molnar
Subject: x86/mm: Don't flush the TLB on #WP pmd fixups
From: Ingo Molnar <[email protected]>
Date: Tue Nov 20 14:46:34 CET 2012If we have a write protection #PF and fix up the pmd then the hugetlb code [the only user of
pmdp_set_access_flags], in itsdo_huge_pmd_wp_page()page fault resolution function callspmdp_set_access_flags()to mark the pmd permissive again, and flushes the TLB.
Quoted 6 lines
This TLB flush is unnecessary: a flush on #PF is guaranteed on most (all?) x86 CPUs, and even in the worst-case we'll generate a spurious fault.
So remove it.
This patch did not cause the 2% speedup that you reported with THP enabled for me:
numa/core at ec05a2311c35: 136918.34 SPECjbb2005 bops numa/core at 01aa90068b12: 128315.19 SPECjbb2005 bops (-6.3%) numa/core at 01aa90068b12 + patch: 128184.77 SPECjbb2005 bops (-6.4%)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
[tip:x86/mm] x86/mm: Don't flush the TLB on #WP pmd fixups
In reply to Ingo MolnarCommit-ID: 5e4bf1a55da976a5ed60901bb8801f1024ef9774 Gitweb: http://git.kernel.org/tip/5e4bf1a55da976a5ed60901bb8801f1024ef9774 Author: Ingo Molnar <[email protected]> AuthorDate: Tue, 20 Nov 2012 13:02:51 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Thu, 22 Nov 2012 21:52:06 +0100
x86/mm: Don't flush the TLB on #WP pmd fixups
If we have a write protection #PF and fix up the pmd then the hugetlb code [the only user of pmdp_set_access_flags], in its do_huge_pmd_wp_page() page fault resolution function calls pmdp_set_access_flags() to mark the pmd permissive again, and flushes the TLB.
This TLB flush is unnecessary: a flush on #PF is guaranteed on most (all?) x86 CPUs, and even in the worst-case we'll generate a spurious fault.
So remove it.
--- arch/x86/mm/pgtable.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/x86/mm/pgtable.c +6 -1
@@ -328,7 +328,12 @@ int pmdp_set_access_flags(struct vm_area_struct *vma,if (changed && dirty) { *pmdp = entry;pmd_update_defer (vma->vm_mm, address, pmdp); -flush_tlb_range (vma, address, address +HPAGE_PMD_SIZE ); +/* +* We had a write-protection fault here and changed the pmd +* to to more permissive. No need to flush the TLB for that, +* #PF is architecturally guaranteed to do that and in the +* worst-case we'll generate a spurious fault. +*/ }return changed;
Quoted 3 lines · Ingo Molnar
Oh, finally a clue: you seem to have vsyscall emulation overhead!
Ingo, stop it already!
This is *exactly* the kind of "blame everybody else than yourself" behavior that I was talking about earlier.
There have been an absolute *shitload* of patches to try to make up for the schednuma regressions THAT HAVE ABSOLUTELY NOTHING TO DO WITH SCHEDNUMA, and are all about trying to work around the fact that it regresses. The whole TLB optimization, and now this kind of crap.
Ingo, look your code in the mirror some day, and ask yourself: why do you think this fixes a "regression"?
The fact is, the *baseline* had the exact same vsyscall emulation too. So by trying to make up for vsyscalls only in your numbers, you are basically trying to lie about regressions, and try to cover up the schednuma regression by fixing something else.
See? That's bogus. When you now compare numbers, YOU ARE LYING. You have introduced a performance regression, and then trying to hide it by making something else go faster.
The same is true of all your arguments about Mel's numbers wrt THP etc. Your arguments are misleading - either intentionally, of because you yourself didn't think things through. For schednuma, it's not enough to be par with mainline with THP off - the competition (autonuma) has been beating mainline soundly in Mel's configuration. So the target to beat is not mainline, but the much *higher* performance that autonuma got.
The fact that Mel has a different configuration from yours IS IRRELEVANT. You should not blame his configuration for the regression, you should instead ask yourself "Why does schednuma regress in that configuration"? And not look at vsyscalls or anything, but look at what schednuma does wrong!
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Ingo Molnar
Oh, finally a clue: you seem to have vsyscall emulation overhead!
Ingo, stop it already!
This is *exactly* the kind of "blame everybody else than yourself" behavior that I was talking about earlier.
Quoted 8 lines
There have been an absolute *shitload* of patches to try to make up for the schednuma regressions THAT HAVE ABSOLUTELY NOTHING TO DO WITH SCHEDNUMA, and are all about trying to work around the fact that it regresses. The whole TLB optimization, and now this kind of crap.
Ingo, look your code in the mirror some day, and ask yourself:
why do you think this fixes a "regression"?
Because scalability slowdowns are often non-linear.
So with CONFIG_NUMA_BALANCING=y we are taking a higher page fault rate, in exchange for a speedup.
But if some other factor further increases the page fault rate (such as vsyscall emulation) then the speedup can be non-linearly slower than the cost of the technique - washing it out or even turning it into an outright regression.
So, for example:
- 10K page faults/sec from CONFIG_SCHED_BALANCING: 0.5% cost
- 10K page faults/sec from vsyscall emu: 0.5% cost
If the two are mixed together the slowdown is non-linear:
- 10K+10K page faults/sec overhead is not a linear 1%, but might be 3%
So because I did not have an old-glibc system like David's, I did not know the actual page fault rate. If it is high enough then nonlinear effects might cause such effects.
This is an entirely valid line of inquiry IMO.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 1 line · Ingo Molnar
This is an entirely valid line of inquiry IMO.
Btw., what I did was to simply look at David's profile on the regressing system and I compared it to the profile I got on a pretty similar (but unfortunately not identical and not regressing) system. I saw 3 differences:
- the numa emulation faults
- the higher TLB miss cost
- numa/core's failure to handle 4K pages properly
And addressed those, in the hope of one of them making a difference.
There's a fourth line of inquiry I'm pursuing as well: the node assymetry that David and Paul mentioned could have a performance effect as well - resulting from non-ideal placement under numa/core.
That is not easy to cure - I have written a patch to take the node assymetry into consideration, I'm still testing it with David's topology simulated on a testbox:
numa=fake=4:10,20,20,30,20,10,20,20,20,20,10,20,30,20,20,10
Will send the patch out later.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Ingo Molnar
Btw., what I did was to simply look at David's profile on the regressing system and I compared it to the profile I got on a pretty similar (but unfortunately not identical and not regressing) system. I saw 3 differences:
Quoted 7 lines
- the numa emulation faults
- the higher TLB miss cost
- numa/core's failure to handle 4K pages properlyAnd addressed those, in the hope of one of them making a difference.
I agree that it's worth a try and if it's helpful to your debugging then I'll always commit to trying things out. I've pulled tip#master at 9f7b91a96bb6 ("Merge branch 'x86/urgent'") and performance improves 0.3% with 16 warehouses with vsyscall=emulate, i.e. a revert of 01e9c2441eee ("x86/vsyscall: Add Kconfig option to use native vsyscalls and switch to it") so I'd recommend that gets dropped based on my results and Andy's feedback unless anybody can demonstrate it's better (which very well may be the case on some systems, but then again that's why its configurable from the command line).
You're also completely right about the old glibc, mine is seven years old; I can upgrade that since I need to install libnuma as well on this box since you asked for the autonuma topology information that I haven't forgotten about but will definitely get around to doing.
Quoted 11 lines
There's a fourth line of inquiry I'm pursuing as well: the node assymetry that David and Paul mentioned could have a performance effect as well - resulting from non-ideal placement under numa/core.
That is not easy to cure - I have written a patch to take the node assymetry into consideration, I'm still testing it with David's topology simulated on a testbox:
numa=fake=4:10,20,20,30,20,10,20,20,20,20,10,20,30,20,20,10
This very much may be the case and that characteristic of this box is why I picked it to test with first. Just curious what types of topologies you've benchmarked on for your results if you have that available? An early version of sched/numa used to panic on this machine and Peter was interested in its topology (see https://lkml.org/lkml/2012/5/25/89) so perhaps I'm the only one testing with such a thing thus far?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Ingo Molnar
So because I did not have an old-glibc system like David's, I did not know the actual page fault rate. If it is high enough then nonlinear effects might cause such effects.
This is an entirely valid line of inquiry IMO.
Btw., when comparing against 'mainline' I routinely use a vanilla kernel that has the same optimization applied. (first I make sure it's not a regression to vanilla.)
I do that to factor out the linear component of the independent
speedup: it would not be valid to compare vanilla against numa/core+optimization, but the comparison has to be:
vanilla + optimization
vs.
numa/core + optimization
I did that with last night's numbers as well.
So any of this can only address a regression if a non-linear factor is in play.
Since I have no direct access to a regressing system I have to work with the theories that I can think of: one had a larger effect, the other had a smaller effect, the third one had no effect on David's system.
How would you have done it instead?
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 2 lines · Ingo Molnar
Because scalability slowdowns are often non-linear.
Only if you hold locks or have other non-cpu-private activity.
Which the vsyscall code really shouldn't have.
That said, it might be worth removing the "prefetchw(&mm->mmap_sem)" from the VM fault path. Partly because software prefetches have never ever worked on any reasonable hardware, and partly because it could seriously screw up things like the vsyscall stuff.
I think we only turn prefetchw into an actual prefetch instruction on 3DNOW hardware. Which is the *old* AMD chips. I don't think even the Athlon does that.
Anyway, it might be interesting to see a instruction-level annotated profile of do_page_fault() or whatever
Quoted 2 lines
So with
CONFIG_NUMA_BALANCING=ywe are taking a higher page fault rate, in exchange for a speedup.
The thing is, so is autonuma.
And autonuma doesn't show any of these problems. Autonuma didn't need vsyscall hacks, autonuma didn't need TLB flushing optimizations, autonuma just *worked*, and in fact got big speedups when Mel did the exact same loads on that same machine, presumably with all the same issues..
Why are you ignoring that fact?
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Linus Torvalds
Quoted 2 lines · Ingo Molnar
Because scalability slowdowns are often non-linear.
Only if you hold locks or have other non-cpu-private activity.
Which the vsyscall code really shouldn't have.
Yeah, the faults accessing any sort of thread shared cache line was my main thinking - the vsyscall faults are so hidden, and David's transaction score was so low that I could not exclude some extremely high page fault rate (which would not get reported by anything other than a strange blip on the profile). I was thinking of a hundred thousand vsyscall page faults per second as a possibility - SPECjbb measures time for every transaction.
So this was just a "maybe-that-has-an-effect" blind theory of mine - and David's testing did not confirm it so we know it was a bad idea.
I basically wanted to see a profile from David that looked as flat as mine - that would have excluded a handful of unknown unknowns.
Quoted 5 lines
That said, it might be worth removing the "
prefetchw(&mm->mmap_sem)" from the VM fault path. Partly because software prefetches have never ever worked on any reasonable hardware, and partly because it could seriously screw up things like the vsyscall stuff.
Yeah, I was wondering about that one too ...
Quoted 6 lines
I think we only turn prefetchw into an actual prefetch instruction on 3DNOW hardware. Which is the *old* AMD chips. I don't think even the Athlon does that.
Anyway, it might be interesting to see a instruction-level annotated profile of
do_page_fault()or whatever
Yes.
Quoted 6 lines
Quoted 2 lines
So with
CONFIG_NUMA_BALANCING=ywe are taking a higher page fault rate, in exchange for a speedup.The thing is, so is autonuma.
And autonuma doesn't show any of these problems. [...]
AutoNUMA regresses on this workload, at least on my box:
v3.7 AutoNUMA | numa/core-v16 [ vs. v3.7]
----- -------- | ------------- -----------
|
[ SPECjbb transactions/sec ] |
[ higher is better ] |
|
SPECjbb single-1x32 524k 507k | 638k +21.7%It regresses by 3.3% over mainline. [I have not measured a THP-disabled number for AutoNUMA.]
Maybe it does not regress on David's box - I have just re-checked all of David's mails and AFAICS he has not reported AutoNUMA SPECjbb performance.
Quoted 1 line
Why are you ignoring that fact?
I'm not :-(
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Linus Torvalds
The same is true of all your arguments about Mel's numbers wrt THP etc. Your arguments are misleading - either intentionally, of because you yourself didn't think things through. For schednuma, it's not enough to be par with mainline with THP off - the competition (autonuma) has been beating mainline soundly in Mel's configuration. So the target to beat is not mainline, but the much *higher* performance that autonuma got.
Once the numa base patches are upstream, the bar will be raised automatically.
With the same infrastructure in place, we will be able to do an apples to apples comparison of the NUMA placement policies, and figure out which one will be the best to merge.
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 2 lines · Linus Torvalds
[...] And not look at vsyscalls or anything, but look at what schednuma does wrong!
I have started 4 independent lines of inquiry to figure out what's wrong on David's system, and all four are in the category of 'what does our tree do to cause a regression':
- suboptimal (== regressive) 4K fault handling by numa/core
- suboptimal (== regressive) placement by numa/core on David's assymetric-topology system
- vsyscalls escallating numa/core page fault overhead non-linearly
- TLB flushes escallating numacore page fault overhead non-linearly
I have sent patches for 3 of them, one is still work in progress, because it's non-trivial.
I'm absolutely open to every possibility and obviously any regression is numa/core's fault, full stop.
What would you have done differently to handle this particular regression?
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>