slab.c: remove branch hint
Subsystems (5)
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/5 process_64: remove branch hint
branch hint profiling of my nehalem machine, showed 96% incorrect branch hints:
6548732 174664120 96 __switch_to process_64.c
406
6548745 174565593 96 __switch_to process_64.c
410
--- arch/x86/kernel/process_64.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kernel/process_64.c +0 -0
@@ -403,11 +403,11 @@ __switch_to(struct task_struct *prev_p, struct
task_struct *next_p)
* This won't pick up thread selector changes, but I guess that is ok. */
savesegment(es, prev->es);
- if (unlikely(next->es | prev->es))
+ if (next->es | prev->es)
loadsegment(es, next->es);
savesegment(ds, prev->ds);
- if (unlikely(next->ds | prev->ds))
+ if (next->ds | prev->ds)
loadsegment(ds, next->ds);
-- 1.6.4.2
[tip:sched/core] sched, x86: Optimize branch hint in __switch_to()
Commit-ID: a3a1de0c34de6f5f8332cd6151c46af7813c0fcb Gitweb: http://git.kernel.org/tip/a3a1de0c34de6f5f8332cd6151c46af7813c0fcb Author: Tim Blechmann <[email protected]> AuthorDate: Tue, 24 Nov 2009 11:55:15 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 24 Nov 2009 12:20:04 +0100
sched, x86: Optimize branch hint in __switch_to()
Branch hint profiling on my nehalem machine showed 96% incorrect branch hints:
6548732 174664120 96 __switch_to process_64.c
406
6548745 174565593 96 __switch_to process_64.c
410LKML-Reference: <[email protected]>
--- arch/x86/kernel/process_64.c | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-)
arch/x86/kernel/process_64.c +2 -3
@@ -406,11 +406,10 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)* This won't pick up thread selector changes, but I guess that is ok. */ savesegment (es, prev->es); -if (unlikely (next->es | prev->es)) +if (next->es | prev->es)loadsegment (es, next->es); -savesegment (ds, prev->ds); -if (unlikely (next->ds | prev->ds)) +if (next->ds | prev->ds)loadsegment (ds, next->ds);
Re: [tip:sched/core] sched, x86: Optimize branch hint in __switch_to()
Quoted 37 lines · tip-bot for Tim Blechmann
Commit-ID: a3a1de0c34de6f5f8332cd6151c46af7813c0fcb
Gitweb: http://git.kernel.org/tip/a3a1de0c34de6f5f8332cd6151c46af7813c0fcb
Author: Tim Blechmann <[email protected]>
AuthorDate: Tue, 24 Nov 2009 11:55:15 +0100
Committer: Ingo Molnar <[email protected]>
CommitDate: Tue, 24 Nov 2009 12:20:04 +0100sched, x86: Optimize branch hint in
__switch_to()Branch hint profiling on my nehalem machine showed 96% incorrect branch hints:
6548732 174664120 96
__switch_toprocess_64.c 406
6548745 174565593 96__switch_toprocess_64.c 410Signed-off-by Tim Blechmann <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Mike Galbraith <[email protected]>Cc Paul Mackerras <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Cc Frederic Weisbecker <[email protected]>LKML-Reference: <[email protected]>
Signed-off-by Ingo Molnar <[email protected]>---
arch/x86/kernel/process_64.c | 5 ++---1 files changed, 2 insertions(+), 3 deletions(-)arch/x86/kernel/process_64.c +0
-0@@ -406,11 +406,10 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)* This won't pick up thread selector changes, but I guess that is ok. */ savesegment(es, prev->es);
arch/x86/kernel/process_64.c __switch_to(struct task_struct *prev_p, struct task_struct *next_p) · line 415
Quoted 10 lines
arch/x86/kernel/process_64.c +2
-3- if (unlikely(next->es | prev->es)) + if (next->es | prev->es) loadsegment(es, next->es); - savesegment(ds, prev->ds); - if (unlikely(next->ds | prev->ds)) + if (next->ds | prev->ds) loadsegment(ds, next->ds);
64-bit tasks should have %ds and %es set to null selectors. The only time they should be different is for 32-bit tasks. It lookx like some versions of GCC just aren't using the hint. I've tested it with 4.4.2 and the generated assembly is the same with or without the hint.
--
Brian Gerst
Re: [tip:sched/core] sched, x86: Optimize branch hint in __switch_to()
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 44 lines · Brian Gerst
Quoted 42 lines · tip-bot for Tim Blechmann
Commit-ID: a3a1de0c34de6f5f8332cd6151c46af7813c0fcb Gitweb: http://git.kernel.org/tip/a3a1de0c34de6f5f8332cd6151c46af7813c0fcb Author: Tim Blechmann <[email protected]> AuthorDate: Tue, 24 Nov 2009 11:55:15 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 24 Nov 2009 12:20:04 +0100sched, x86: Optimize branch hint in
__switch_to()Branch hint profiling on my nehalem machine showed 96% incorrect branch hints:
6548732 174664120 96
__switch_toprocess_64.c
406
6548745 174565593 96__switch_toprocess_64.c
410Signed-off-by Tim Blechmann <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Mike Galbraith <[email protected]>Cc Paul Mackerras <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Cc Frederic Weisbecker <[email protected]>LKML-Reference: <[email protected]>
Signed-off-by Ingo Molnar <[email protected]>--- arch/x86/kernel/process_64.c | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-)arch/x86/kernel/process_64.c +1
-2@@ -406,11 +406,10 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)* This won't pick up thread selector changes, but I guess that is ok. */ savesegment (es, prev->es); -if (unlikely (next->es | prev->es)) +if (next->es | prev->es)loadsegment (es, next->es); -savesegment (ds, prev->ds);
Quoted 8 lines
arch/x86/kernel/process_64.c __switch_to(struct task_struct *prev_p, struct task_struct *next_p) · line 415
Quoted 5 lines
arch/x86/kernel/process_64.c +1
-1-if (unlikely (next->ds | prev->ds)) +if (next->ds | prev->ds)loadsegment (ds, next->ds);64-bit tasks should have %ds and %es set to null selectors. The only time they should be different is for 32-bit tasks.
this doesn't seem to be the case on my machine for 96% of the calls ... i am just running 64bit binaries on this machine, no 32bit programs (that i know of (ubuntu karmic amd64))
tim
- --
[email protected]
http://tim.klingt.org
Relying on the government to protect your privacy is like asking a peeping tom to install your window blinds.
John Perry Barlow
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAksM8s0ACgkQdL+4qsZfVsscjwCffcvQz7/VTtIfH1XXgJ2SxJxT xWYAnjZ9aBD/OWmroVn68oEKqJZ7Pm3U
=C7kB
-----END PGP SIGNATURE-----
Re: [tip:sched/core] sched, x86: Optimize branch hint in __switch_to()
Quoted 57 lines · Tim Blechmann
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 44 lines · Brian Gerst
Quoted 42 lines · tip-bot for Tim Blechmann
Commit-ID: a3a1de0c34de6f5f8332cd6151c46af7813c0fcb
Gitweb: http://git.kernel.org/tip/a3a1de0c34de6f5f8332cd6151c46af7813c0fcb
Author: Tim Blechmann <[email protected]>
AuthorDate: Tue, 24 Nov 2009 11:55:15 +0100
Committer: Ingo Molnar <[email protected]>
CommitDate: Tue, 24 Nov 2009 12:20:04 +0100sched, x86: Optimize branch hint in
__switch_to()Branch hint profiling on my nehalem machine showed 96% incorrect branch hints:
6548732 174664120 96
__switch_toprocess_64.c 406
6548745 174565593 96__switch_toprocess_64.c 410Signed-off-by Tim Blechmann <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Mike Galbraith <[email protected]>Cc Paul Mackerras <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Cc Frederic Weisbecker <[email protected]>LKML-Reference: <[email protected]>
Signed-off-by Ingo Molnar <[email protected]>---
arch/x86/kernel/process_64.c | 5 ++---1 files changed, 2 insertions(+), 3 deletions(-)arch/x86/kernel/process_64.c +1
-2@@ -406,11 +406,10 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)* This won't pick up thread selector changes, but I guess that is ok. */ savesegment(es, prev->es); - if (unlikely(next->es | prev->es)) + if (next->es | prev->es) loadsegment(es, next->es); - savesegment(ds, prev->ds);Quoted 8 lines
arch/x86/kernel/process_64.c __switch_to(struct task_struct *prev_p, struct task_struct *next_p) · line 415
Quoted 5 lines
arch/x86/kernel/process_64.c +1
-1- if (unlikely(next->ds | prev->ds)) + if (next->ds | prev->ds) loadsegment(ds, next->ds);64-bit tasks should have %ds and %es set to null selectors. The only time they should be different is for 32-bit tasks.
Quoted 5 lines
this doesn't seem to be the case on my machine for 96% of the calls ... i am just running 64bit binaries on this machine, no 32bit programs (that i know of (ubuntu karmic amd64))
tim
Some early kernel threads inherited KERNEL_DS from the boot process. Patch coming soon.
--
Brian Gerst
index 780cd92..22db86a 100644
- /* set up data segments. actually 0 would do too */
- movl $__KERNEL_DS,%eax
+ /* set up data segments */
+ xorl %eax,%eax
movl %eax,%ds
movl %eax,%ss
movl %eax,%es
PATCH 1/1 x86, 64-bit: Set data segments to null after switching to 64-bit mode
In reply to Brian GerstThis prevents kernel threads from inheriting non-null segment selectors, and causing optimizations in __switch_to() to be ineffective.
--- arch/x86/kernel/head_64.S | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kernel/head_64.S +2 -2
@@ -212,8 +212,8 @@ ENTRY(secondary_startup_64) */ lgdt early_gdt_descr(%rip) - /* set up data segments. actually 0 would do too */ - movl $__KERNEL_DS,%eax + /* set up data segments */ + xorl %eax,%eax movl %eax,%ds movl %eax,%ss movl %eax,%es
Signature · 1 line
1.6.5.2
[tip:x86/asm] x86, 64-bit: Set data segments to null after switching to 64-bit mode
Commit-ID: 8ec6993d9f7d961014af970ded57542961fe9ad9 Gitweb: http://git.kernel.org/tip/8ec6993d9f7d961014af970ded57542961fe9ad9 Author: Brian Gerst <[email protected]> AuthorDate: Wed, 25 Nov 2009 11:17:36 -0500 Committer: Ingo Molnar <[email protected]> CommitDate: Thu, 26 Nov 2009 10:44:30 +0100
x86, 64-bit: Set data segments to null after switching to 64-bit mode
This prevents kernel threads from inheriting non-null segment selectors, and causing optimizations in __switch_to() to be ineffective.
LKML-Reference: <[email protected]>
--- arch/x86/kernel/head_64.S | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kernel/head_64.S +2 -2
@@ -212,8 +212,8 @@ ENTRY(secondary_startup_64) */ lgdt early_gdt_descr(%rip) - /* set up data segments. actually 0 would do too */ - movl $__KERNEL_DS,%eax + /* set up data segments */ + xorl %eax,%eax movl %eax,%ds movl %eax,%ss movl %eax,%es
Re: [PATCH] x86, 64-bit: Set data segments to null after switching to 64-bit mode
In reply to Brian GerstQuoted 4 lines · Brian Gerst
This prevents kernel threads from inheriting non-null segment selectors, and causing optimizations in
__switch_to()to be ineffective.Signed-off-by Brian Gerst <[email protected]>
with this patch, the branch hint is actually correct, giving 100% correct predictions. so please revert my earlier patch (commit a3a1de0c34de6f5f8332cd6151c46af7813c0fcb)
thnx, tim
Signature · 6 lines
[email protected] http://tim.klingt.org Desperation is the raw material of drastic change. Only those who can leave behind everything they have ever believed in can hope to escape. William S. Burroughs
PATCH 1/1 Revert "sched, x86: Optimize branch hint in __switch_to()"
In reply to tip-bot for Tim BlechmannThis reverts commit a3a1de0c34de6f5f8332cd6151c46af7813c0fcb.
Commit 8ec6993d9f7d961014af970ded57542961fe9ad9 cleared the es and ds selectors, so the original branch hints are correct now. Therefore the branch hint doesn't need to be removed.
--- arch/x86/kernel/process_64.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
arch/x86/kernel/process_64.c +3 -2
@@ -406,10 +406,11 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)* This won't pick up thread selector changes, but I guess that is ok. */ savesegment (es, prev->es); -if (next->es | prev->es) +if (unlikely (next->es | prev->es))loadsegment (es, next->es); +savesegment (ds, prev->ds); -if (next->ds | prev->ds) +if (unlikely (next->ds | prev->ds))loadsegment (ds, next->ds);
[tip:sched/core] Revert "sched, x86: Optimize branch hint in __switch_to()"
Commit-ID: be8147e68625a1adb111acfd6b98a492be4b74d0 Gitweb: http://git.kernel.org/tip/be8147e68625a1adb111acfd6b98a492be4b74d0 Author: Tim Blechmann <[email protected]> AuthorDate: Wed, 2 Dec 2009 12:32:10 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 2 Dec 2009 12:38:03 +0100
Revert "sched, x86: Optimize branch hint in __switch_to()"
This reverts commit a3a1de0c34de6f5f8332cd6151c46af7813c0fcb.
Commit 8ec6993d9f7d961014af970ded57542961fe9ad9 cleared the es and ds selectors, so the original branch hints are correct now. Therefore the branch hint doesn't need to be removed.
LKML-Reference: <[email protected]>
--- arch/x86/kernel/process_64.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
arch/x86/kernel/process_64.c +3 -2
@@ -406,10 +406,11 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)* This won't pick up thread selector changes, but I guess that is ok. */ savesegment (es, prev->es); -if (next->es | prev->es) +if (unlikely (next->es | prev->es))loadsegment (es, next->es); +savesegment (ds, prev->ds); -if (next->ds | prev->ds) +if (unlikely (next->ds | prev->ds))loadsegment (ds, next->ds);
PATCH 2/5 sched.c: change branch hint
In reply to a message not in the archivebranch hint profiling of my nehalem machine, showed over 90% incorrect branch hints:
10420275 170645395 94 context_switch sched.c
3043
10408421 171098521 94 context_switch sched.c
3050
--- kernel/sched.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/sched.c +0 -0
@@ -3040,14 +3040,14 @@ context_switch(struct rq *rq, struct task_struct
*prev,
*/
arch_start_context_switch(prev);
- if (unlikely(!mm)) {
+ if (likely(!mm)) {
next->active_mm = oldmm;
atomic_inc(&oldmm->mm_count);
enter_lazy_tlb(oldmm, next);
} else
switch_mm(oldmm, mm, next);
- if (unlikely(!prev->mm)) {
+ if (likely(!prev->mm)) {
prev->active_mm = NULL;
rq->prev_mm = oldmm;
}
Signature · 1 line
1.6.4.2
[tip:sched/core] sched: Optimize branch hint in context_switch()
Commit-ID: 710390d90f143a9ebb87a475215140f426792efd Gitweb: http://git.kernel.org/tip/710390d90f143a9ebb87a475215140f426792efd Author: Tim Blechmann <[email protected]> AuthorDate: Tue, 24 Nov 2009 11:55:27 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 24 Nov 2009 12:18:42 +0100
sched: Optimize branch hint in context_switch()
Branch hint profiling on my nehalem machine showed over 90% incorrect branch hints:
10420275 170645395 94 context_switch sched.c
3043
10408421 171098521 94 context_switch sched.c
3050
LKML-Reference: <[email protected]>
--- kernel/sched.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/sched.c +2 -2
@@ -2829,14 +2829,14 @@ context_switch(struct rq *rq, struct task_struct *prev,*/ arch_start_context_switch (prev); -if (unlikely (!mm)) { +if (likely (!mm)) { next->active_mm = oldmm;atomic_inc (&oldmm->mm_count);enter_lazy_tlb (oldmm, next); }else switch_mm (oldmm, mm, next); -if (unlikely (!prev->mm)) { +if (likely (!prev->mm)) { prev->active_mm =NULL ; rq->prev_mm = oldmm; }
PATCH 3/5 slab.c: remove branch hint
In reply to a message not in the archivebranch profiling on my nehalem machine showed 99% incorrect branch hints:
28459 7678524 99 __cache_alloc_node slab.c
3551
--- mm/slab.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
mm/slab.c +0 -0
@@ -3548,7 +3548,7 @@ __cache_alloc_node(struct kmem_cache *cachep,
gfp_t flags, int nodeid,
slab_irq_save(save_flags, this_cpu); this_node = cpu_to_node(this_cpu);
- if (unlikely(nodeid == -1))
+ if (nodeid == -1)
nodeid = this_node;
if (unlikely(!cachep->nodelists[nodeid])) {
Signature · 1 line
1.6.4.2
(Pekka Cc:-ed)
Quoted 26 lines · Tim Blechmann
branch profiling on my nehalem machine showed 99% incorrect branch hints:
28459 7678524 99 __cache_alloc_node slab.c3551
Signed-off-by Tim Blechmann <[email protected]>--- mm/slab.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)mm/slab.c +1
-1@@ -3548,7 +3548,7 @@ __cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,slab_irq_save (save_flags ,this_cpu ); this_node =cpu_to_node (this_cpu); -if (unlikely (nodeid ==-1 )) +if (nodeid ==-1 ) nodeid = this_node;if (unlikely (!cachep->nodelists[nodeid])) {Signature · 1 line
1.6.4.2
Quoted 14 lines · Ingo Molnar
(Pekka Cc:-ed)
Quoted 10 lines · Tim Blechmann
branch profiling on my nehalem machine showed 99% incorrect branch hints:
28459 7678524 99
__cache_alloc_nodeslab.c3551Signed-off-by Tim Blechmann <[email protected]>---
mm/slab.c | 2 +-1 files changed, 1 insertions(+), 1 deletions(-)
Quoted 12 lines
mm/slab.c __cache_alloc_node(struct kmem_cache *cachep, · line 3553
Quoted 12 lines
mm/slab.c +1
-1@@ -3548,7 +3548,7 @@ __cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, slab_irq_save(save_flags, this_cpu); this_node = cpu_to_node(this_cpu); - if (unlikely(nodeid == -1)) + if (nodeid == -1) nodeid = this_node; if (unlikely(!cachep->nodelists[nodeid])) {
That sounds odd to me. Can you see where the incorrectly predicted calls are coming from? Calling kmem_cache_alloc_node() with node set to -1 most of the time could be a real bug somewhere.
Quoted 15 lines · Pekka Enberg
Quoted 14 lines · Ingo Molnar
(Pekka Cc:-ed)
Quoted 10 lines · Tim Blechmann
branch profiling on my nehalem machine showed 99% incorrect branch hints:
? ?28459 ?7678524 ?99
__cache_alloc_node? ? ? ? ? ?slab.c? 3551Signed-off-by Tim Blechmann <[email protected]>---
?mm/slab.c | ? ?2 +-
?1 files changed, 1 insertions(+), 1 deletions(-)
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
mm/slab.c +0
-0Quoted 8 lines
mm/slab.c __cache_alloc_node(struct kmem_cache *cachep, · line 3553
Quoted 8 lines
mm/slab.c +1
-1@@ -3548,7 +3548,7 @@ __cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, ? ? ? slab_irq_save(save_flags, this_cpu); ? ? ? this_node = cpu_to_node(this_cpu); - ? ? if (unlikely(nodeid == -1)) + ? ? if (nodeid == -1) ? ? ? ? ? ? ? nodeid = this_node; ? ? ? if (unlikely(!cachep->nodelists[nodeid])) {That sounds odd to me. Can you see where the incorrectly predicted calls are coming from? Calling
kmem_cache_alloc_node()with node set to -1 most of the time could be a real bug somewhere.
I think it could occur in too limited tests - the branch prediction looks 'wrong' in certain tests - while it's OK in general.
Is there some easy to run workload you consider more or less representative of typical SLAB patterns?
<plug> You might want to pull even with the scheduler subsystem and in
addition to 'perf bench sched', add a 'perf bench slab' set of interesting testcases for SLAB performance testing. :-)
</plug>
Ingo
Ingo Molnar kirjoitti:
Quoted 7 lines
Quoted 3 lines
That sounds odd to me. Can you see where the incorrectly predicted calls are coming from? Calling
kmem_cache_alloc_node()with node set to -1 most of the time could be a real bug somewhere.I think it could occur in too limited tests - the branch prediction looks 'wrong' in certain tests - while it's OK in general.
Is there some easy to run workload you consider more or less representative of typical SLAB patterns?
I can think of three main classes: VFS, SCSI, or network intensive applications and benchmarks tend to bring out the worst in SLAB. There are probably some interesting NUMA related things that I'm not really aware of.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 15 lines · Pekka Enberg
Quoted 14 lines · Ingo Molnar
(Pekka Cc:-ed)
Quoted 10 lines · Tim Blechmann
branch profiling on my nehalem machine showed 99% incorrect branch hints:
28459 7678524 99 __cache_alloc_node slab.c3551
Signed-off-by Tim Blechmann <[email protected]>--- mm/slab.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
mm/slab.c +0
-0Quoted 8 lines
mm/slab.c __cache_alloc_node(struct kmem_cache *cachep, · line 3553
Quoted 8 lines
mm/slab.c +1
-1@@ -3548,7 +3548,7 @@ __cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,slab_irq_save (save_flags ,this_cpu ); this_node =cpu_to_node (this_cpu); -if (unlikely (nodeid ==-1 )) +if (nodeid ==-1 ) nodeid = this_node;if (unlikely (!cachep->nodelists[nodeid])) {That sounds odd to me. Can you see where the incorrectly predicted calls are coming from? Calling
kmem_cache_alloc_node()with node set to -1 most of the time could be a real bug somewhere.
i don't know, if there is any facility in the ftrace branch profiler to get call graph information, but i can try to manually dump backtraces in this condition path ...
could be a specific situation on my machine, though ...
tim
- --
[email protected]
http://tim.klingt.org
You don't have to call it music if the term shocks you.
John Cage
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAksLx0kACgkQdL+4qsZfVsskOQCglFQG3eYAfdgXoOAHAGTqaLcU 8e0AoIQNbzSRxttGFaXTF3PEh5O4aGEB
=3nmT
-----END PGP SIGNATURE-----
Quoted 15 lines · Pekka Enberg
Quoted 14 lines · Ingo Molnar
(Pekka Cc:-ed)
Quoted 10 lines · Tim Blechmann
branch profiling on my nehalem machine showed 99% incorrect branch hints:
28459 7678524 99 __cache_alloc_node slab.c3551
Signed-off-by Tim Blechmann <[email protected]>--- mm/slab.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
mm/slab.c +0
-0Quoted 8 lines
mm/slab.c __cache_alloc_node(struct kmem_cache *cachep, · line 3553
Quoted 8 lines
mm/slab.c +1
-1@@ -3548,7 +3548,7 @@ __cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,slab_irq_save (save_flags ,this_cpu ); this_node =cpu_to_node (this_cpu); -if (unlikely (nodeid ==-1 )) +if (nodeid ==-1 ) nodeid = this_node;if (unlikely (!cachep->nodelists[nodeid])) {That sounds odd to me. Can you see where the incorrectly predicted calls are coming from? Calling
kmem_cache_alloc_node()with node set to -1 most of the time could be a real bug somewhere.
when dumping the stack for the incorrectly hinted branches, i get the attached stack traces...
hth, tim
mm/slab.c +0 -0
@@ -3548,8 +3548,10 @@ __cache_alloc_node(struct kmem_cache *cachep,
gfp_t flags, int nodeid,
slab_irq_save(save_flags, this_cpu);
this_node = cpu_to_node(this_cpu);
- if (nodeid == -1)
+ if (nodeid == -1) {
+ dump_stack();
nodeid = this_node;+ }
if (unlikely(!cachep->nodelists[nodeid])) {
/* Node not bootstrapped yet */Signature · 8 lines
[email protected] http://tim.klingt.org It is better to make a piece of music than to perform one, better to perform one than to listen to one, better to listen to one than to misuse it as a means of distraction, entertainment, or acquisition of 'culture'. John Cage
Tim Blechmann kirjoitti:
Quoted 51 lines · Pekka Enberg
Quoted 15 lines · Ingo Molnar
Quoted 14 lines
(Pekka Cc:-ed)
Quoted 10 lines · Tim Blechmann
branch profiling on my nehalem machine showed 99% incorrect branch hints:
28459 7678524 99 __cache_alloc_node slab.c3551
Signed-off-by Tim Blechmann <[email protected]>--- mm/slab.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)Quoted 15 lines
Quoted 4 lines
Quoted 4 lines
mm/slab.c +0
-0Quoted 8 lines
mm/slab.c __cache_alloc_node(struct kmem_cache *cachep, · line 3553
Quoted 8 lines
mm/slab.c +1
-1@@ -3548,7 +3548,7 @@ __cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,slab_irq_save (save_flags ,this_cpu ); this_node =cpu_to_node (this_cpu); -if (unlikely (nodeid ==-1 )) +if (nodeid ==-1 ) nodeid = this_node;if (unlikely (!cachep->nodelists[nodeid])) {That sounds odd to me. Can you see where the incorrectly predicted calls are coming from? Calling
kmem_cache_alloc_node()with node set to -1 most of the time could be a real bug somewhere.when dumping the stack for the incorrectly hinted branches, i get the attached stack traces...
hth, tim
mm/slab.c +3
-1@@ -3548,8 +3548,10 @@ __cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,slab_irq_save (save_flags ,this_cpu ); this_node =cpu_to_node (this_cpu); -if (nodeid ==-1 ) +if (nodeid ==-1 ) { +dump_stack (); nodeid = this_node; + }if (unlikely (!cachep->nodelists[nodeid])) {
Quoted 4 lines
/* Node not bootstrapped yet */
OK, so it's the generic alloc_skb() function that keeps hitting kmem_cache_alloc_node() with "-1". Christoph, are you okay with removing the unlikely() annotation from __cache_alloc_node()?
Pekka
Quoted 3 lines · Pekka Enberg
OK, so it's the generic
alloc_skb()function that keeps hittingkmem_cache_alloc_node()with "-1". Christoph, are you okay with removing theunlikely()annotation from__cache_alloc_node()?
Yes. Lets look for other cases in the allocators too. kmem_cache_alloc_node used to be mainly used for off node allocations but the network alloc_skb() case shows that this is changing now.
I hope the users of kmem_cache_alloc_node() realize that using -1 is not equivalent to kmem_cache_alloc(). kmem_cache_alloc follows numa policies for memory allocations. kmem_cache_alloc_node() does not.
Quoted 9 lines · Christoph Lameter
Quoted 3 lines · Pekka Enberg
OK, so it's the generic
alloc_skb()function that keeps hittingkmem_cache_alloc_node()with "-1". Christoph, are you okay with removing theunlikely()annotation from__cache_alloc_node()?Yes. Lets look for other cases in the allocators too.
kmem_cache_alloc_nodeused to be mainly used for off node allocations but the networkalloc_skb()case shows that this is changing now.
Tim, can you please re-send the patch to me so I can apply it?
Quoted 1 line · Pekka Enberg
Tim, can you please re-send the patch to me so I can apply it?
SLUB has no issue since NUMA decisions are deferred to the page allocator.
PATCH 1/1 branch profiling on my nehalem machine showed 99% incorrect branch hints:
In reply to Pekka Enbergi have regenerated the patch for tip/tip ...
--
28459 7678524 99 __cache_alloc_node slab.c 3551
Discussion on lkml [1] led to the solution to remove this hint.
[1] http://patchwork.kernel.org/patch/63517/
--- mm/slab.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
mm/slab.c +1 -1
@@ -3306,7 +3306,7 @@ __cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,cache_alloc_debugcheck_before (cachep, flags);local_irq_save (save_flags); -if (unlikely (nodeid ==-1 )) +if (nodeid ==-1 ) nodeid =numa_node_id ();if (unlikely (!cachep->nodelists[nodeid])) {
Pekka Enberg
AppliedRe: [PATCH] branch profiling on my nehalem machine showed 99% incorrect branch hints:
Quoted 5 lines · Tim Blechmann
i have regenerated the patch for tip/tip ...
--
28459 7678524 99__cache_alloc_nodeslab.c3551
Quoted 8 lines
Discussion on lkml [1] led to the solution to remove this hint.
[1] http://patchwork.kernel.org/patch/63517/
Signed-off-by Tim Blechmann <[email protected]>--- mm/slab.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
Applied, thanks.
Please use plain text email for sending patches in the future.
PATCH 4/5 sched_fair.c: remove branch hint
In reply to a message not in the archivebranch hint profiling of my nehalem machine, showed 90% incorrect branch hints:
15728471 158903754 90 pick_next_task_fair sched_fair.c
1555
--- kernel/sched_fair.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched_fair.c +0 -0
@@ -1552,7 +1552,7 @@ static struct task_struct
*pick_next_task_fair(struct rq *rq)
struct cfs_rq *cfs_rq = &rq->cfs;
struct sched_entity *se;
- if (unlikely(!cfs_rq->nr_running))
+ if (!cfs_rq->nr_running)
return NULL;
do {
Signature · 1 line
1.6.4.2
[tip:sched/core] sched: Optimize branch hint in pick_next_task_fair()
Commit-ID: 36ace27e3e60d44ea69ce394b2e45386ae98d9d9 Gitweb: http://git.kernel.org/tip/36ace27e3e60d44ea69ce394b2e45386ae98d9d9 Author: Tim Blechmann <[email protected]> AuthorDate: Tue, 24 Nov 2009 11:55:45 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 24 Nov 2009 12:18:12 +0100
sched: Optimize branch hint in pick_next_task_fair()
Branch hint profiling on my nehalem machine showed 90% incorrect branch hints:
15728471 158903754 90 pick_next_task_fair
sched_fair.c 1555
LKML-Reference: <[email protected]>
--- kernel/sched_fair.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched_fair.c +1 -1
@@ -1704,7 +1704,7 @@ static struct task_struct *pick_next_task_fair(struct rq *rq)struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se; -if (unlikely (!cfs_rq->nr_running)) +if (!cfs_rq->nr_running)return NULL ; do {
PATCH 5/5 workqueue.c: remove branch hint
In reply to a message not in the archivebranch hint profiling of my nehalem machine, showed 96% incorrect branch hints:
131609 3840434 96 queue_delayed_work_on workqueue.c 258
--- kernel/workqueue.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/workqueue.c +0 -0
@@ -255,7 +255,7 @@ int queue_delayed_work_on(int cpu, struct
workqueue_struct *wq,
timer->data = (unsigned long)dwork; timer->function = delayed_work_timer_fn;
- if (unlikely(cpu >= 0))
+ if (cpu >= 0)
add_timer_on(timer, cpu);
else
add_timer(timer);
Signature · 1 line
1.6.4.2