a43455a1d57: +94.1% proc-vmstat.numa_hint_faults_local
A message that is not in the archive (sent off-list or not yet received): 53d70ee6.JsUEmW5dWsv8dev+%[email protected]
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
commit a43455a1d572daf7b730fe12eb747d1e17411365 ("sched/numa: Ensure task_numa_migrate() checks the preferred node")
ebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe
67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket
162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_localebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
147474 ~ 3% +70.6% 251650 ~ 5% ivb42/hackbench/50%-threads-pipe
94889 ~ 3% +46.3% 138815 ~ 5% lkp-snb01/hackbench/50%-threads-socket
242364 ~ 3% +61.1% 390465 ~ 5% TOTAL proc-vmstat.numa_pte_updatesebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
147104 ~ 3% +69.5% 249306 ~ 5% ivb42/hackbench/50%-threads-pipe
94431 ~ 3% +43.9% 135902 ~ 5% lkp-snb01/hackbench/50%-threads-socket
241535 ~ 3% +59.5% 385209 ~ 5% TOTAL proc-vmstat.numa_hint_faultsebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
308 ~ 8% +24.1% 382 ~ 5% lkp-snb01/hackbench/50%-threads-socket
308 ~ 8% +24.1% 382 ~ 5% TOTAL numa-vmstat.node0.nr_page_table_pagesebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
1234 ~ 8% +24.0% 1530 ~ 5% lkp-snb01/hackbench/50%-threads-socket
1234 ~ 8% +24.0% 1530 ~ 5% TOTAL numa-meminfo.node0.PageTablesebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
381 ~ 6% -17.9% 313 ~ 6% lkp-snb01/hackbench/50%-threads-socket
381 ~ 6% -17.9% 313 ~ 6% TOTAL numa-vmstat.node1.nr_page_table_pagesebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
1528 ~ 6% -18.0% 1253 ~ 6% lkp-snb01/hackbench/50%-threads-socket
1528 ~ 6% -18.0% 1253 ~ 6% TOTAL numa-meminfo.node1.PageTablesebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
24533 ~ 2% -16.2% 20560 ~ 3% ivb42/hackbench/50%-threads-pipe
13551 ~ 2% -10.7% 12096 ~ 2% lkp-snb01/hackbench/50%-threads-socket
38084 ~ 2% -14.2% 32657 ~ 3% TOTAL proc-vmstat.numa_pages_migratedebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
24533 ~ 2% -16.2% 20560 ~ 3% ivb42/hackbench/50%-threads-pipe
13551 ~ 2% -10.7% 12096 ~ 2% lkp-snb01/hackbench/50%-threads-socket
38084 ~ 2% -14.2% 32657 ~ 3% TOTAL proc-vmstat.pgmigrate_successebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
3538 ~ 7% +11.6% 3949 ~ 7% lkp-snb01/hackbench/50%-threads-socket
3538 ~ 7% +11.6% 3949 ~ 7% TOTAL numa-vmstat.node0.nr_anon_pagesebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
14154 ~ 7% +11.6% 15799 ~ 7% lkp-snb01/hackbench/50%-threads-socket
14154 ~ 7% +11.6% 15799 ~ 7% TOTAL numa-meminfo.node0.AnonPagesebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
3511 ~ 7% +11.0% 3898 ~ 7% lkp-snb01/hackbench/50%-threads-socket
3511 ~ 7% +11.0% 3898 ~ 7% TOTAL numa-vmstat.node0.nr_active_anonebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
14044 ~ 7% +11.1% 15597 ~ 7% lkp-snb01/hackbench/50%-threads-socket
14044 ~ 7% +11.1% 15597 ~ 7% TOTAL numa-meminfo.node0.Active(anon)ebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
187958 ~ 2% +56.6% 294375 ~ 5% ivb42/hackbench/50%-threads-pipe
124490 ~ 2% +35.0% 168004 ~ 4% lkp-snb01/hackbench/50%-threads-socket
312448 ~ 2% +48.0% 462379 ~ 5% TOTAL time.minor_page_faultsebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
11.47 ~ 1% -2.8% 11.15 ~ 1% ivb42/hackbench/50%-threads-pipe
11.47 ~ 1% -2.8% 11.15 ~ 1% TOTAL turbostat.RAM_Webe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
3.649e+08 ~ 0% -2.4% 3.562e+08 ~ 0% lkp-snb01/hackbench/50%-threads-socket 3.649e+08 ~ 0% -2.4% 3.562e+08 ~ 0% TOTAL time.involuntary_context_switches
ebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
1924472 ~ 0% -2.6% 1874425 ~ 0% ivb42/hackbench/50%-threads-pipe 1924472 ~ 0% -2.6% 1874425 ~ 0% TOTAL vmstat.system.in
ebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
1.38e+09 ~ 0% -1.8% 1.355e+09 ~ 0% lkp-snb01/hackbench/50%-threads-socket 1.38e+09 ~ 0% -1.8% 1.355e+09 ~ 0% TOTAL time.voluntary_context_switches
Legend:
~XX% - stddev percent
[+-]XX% - change percent
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.
Thanks,
Aaron
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu10/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu11/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu12/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu13/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu14/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu15/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu16/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu17/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu18/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu19/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu20/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu21/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu22/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu23/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu24/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu25/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu26/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu27/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu28/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu29/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu30/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu31/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu5/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu6/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu7/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu8/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu9/cpufreq/scaling_governor /usr/bin/hackbench -g 16 --threads -l 60000 /usr/bin/hackbench -g 16 --threads -l 60000 /usr/bin/hackbench -g 16 --threads -l 60000 /usr/bin/hackbench -g 16 --threads -l 60000 /usr/bin/hackbench -g 16 --threads -l 60000 /usr/bin/hackbench -g 16 --threads -l 60000 /usr/bin/hackbench -g 16 --threads -l 60000 /usr/bin/hackbench -g 16 --threads -l 60000 /usr/bin/hackbench -g 16 --threads -l 60000 /usr/bin/hackbench -g 16 --threads -l 60000 /usr/bin/hackbench -g 16 --threads -l 60000 /usr/bin/hackbench -g 16 --threads -l 60000 /usr/bin/hackbench -g 16 --threads -l 60000
Quoted 10 lines · On Tue, 29 Jul 2014 13:24:05 +0800 Aaron Lu <[email protected]> wrote:
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitmaster
commita43455a1d572daf7b730fe12eb747d1e17411365("sched/numa: Ensuretask_numa_migrate()checks the preferred node")
ebe06187bf2aec1a43455a1d572daf7b730fe12e
--------------- -------------------------94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_local
Hi Aaron,
Jirka Hladky has reported a regression with that changeset as well, and I have already spent some time debugging the issue.
I added tracing code to task_numa_compare() and saw a number of thread swaps with tiny improvements.
Does preventing those help your workload, or am I barking up the wrong tree again? (I have been looking at this for a while...)
---8<---
Subject: sched,numa: prevent task moves with marginal benefit
Commit a43455a1d57 makes task_numa_migrate() always check the preferred node for task placement. This is causing a performance regression with hackbench, as well as SPECjbb2005.
Tracing task_numa_compare() with a single instance of SPECjbb2005 on a 4 node system, I have seen several thread swaps with tiny improvements.
It appears that the hysteresis code that was added to task_numa_compare is not doing what we needed it to do, and a simple threshold could be better.
--- kernel/sched/fair.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-)
kernel/sched/fair.c +15 -9
@@ -924,10 +924,12 @@ static inline unsigned long group_faults_cpu(struct numa_group *group, int nid)/* * These return the fraction of accesses done by a particular task, or -* task group, on a particular numa node. The group weight is given a -* larger multiplier, in order to group tasks together that are almost -* evenly spread out between numa nodes. +* task group, on a particular numa node. The NUMA move threshold +* prevents task moves with marginal improvement, and is set to 5%. */ +#define NUMA_SCALE 1000 +#define NUMA_MOVE_THRESH 50 +static inline unsigned long task_weight (struct task_struct *p,int nid) {unsigned long total_faults;
@@ -940,7 +942,7 @@ static inline unsigned long task_weight(struct task_struct *p, int nid)if (!total_faults)return 0 ; -return 1000 *task_faults (p, nid) / total_faults; +return NUMA_SCALE *task_faults (p, nid) / total_faults; }static inline unsigned long group_weight (struct task_struct *p,int nid)
@@ -948,7 +950,7 @@ static inline unsigned long group_weight(struct task_struct *p, int nid)if (!p->numa_group || !p->numa_group->total_faults)return 0 ; -return 1000 *group_faults (p, nid) / p->numa_group->total_faults; +return NUMA_SCALE *group_faults (p, nid) / p->numa_group->total_faults; } bool should_numa_migrate_memory(struct task_struct *p, struct page * page,
@@ -1181,11 +1183,11 @@ static void task_numa_compare(struct task_numa_env *env, imp = taskimp +task_weight (cur, env->src_nid) -task_weight (cur, env->dst_nid);/* -* Add some hysteresis to prevent swapping the -* tasks within a group over tiny differences. +* Do not swap tasks within a group around unless +* there is a significant improvement. */ -if (cur->numa_group) - imp -= imp/16 ; +if (cur->numa_group && imp <NUMA_MOVE_THRESH ) + goto unlock; }else {/* * Compare the group weights. If a task is all by
@@ -1205,6 +1207,10 @@ static void task_numa_compare(struct task_numa_env *env, goto unlock;if (!cur) { +/* Only move if there is a significant improvement. */ +if (imp <NUMA_MOVE_THRESH ) + goto unlock; +/* Is there capacity@our destination? */ if (env->src_stats.has_free_capacity && !env->dst_stats.has_free_capacity)
Peter Zijlstra
maintainerQuoted 27 lines · Rik van Riel
Subject: sched,numa: prevent task moves with marginal benefit
Commit
a43455a1d57makestask_numa_migrate()always check the preferred node for task placement. This is causing a performance regression with hackbench, as well as SPECjbb2005.Tracing
task_numa_compare()with a single instance of SPECjbb2005 on a 4 node system, I have seen several thread swaps with tiny improvements.It appears that the hysteresis code that was added to
task_numa_compareis not doing what we needed it to do, and a simple threshold could be better.Reported-by Aaron Lu <[email protected]>Reported-by Jirka Hladky <[email protected]>Signed-off-by Rik van Riel <[email protected]>--- kernel/sched/fair.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-)kernel/sched/fair.c +0
-0@@ -924,10 +924,12 @@ static inline unsigned long group_faults_cpu(struct numa_group *group, int nid)
kernel/sched/fair.c static inline unsigned long group_faults_cpu(struct numa_group *group, int nid) · line 931
Quoted 10 lines
kernel/sched/fair.c +4
-3/* * These return the fraction of accesses done by a particular task, or -* task group, on a particular numa node. The group weight is given a -* larger multiplier, in order to group tasks together that are almost -* evenly spread out between numa nodes. +* task group, on a particular numa node. The NUMA move threshold +* prevents task moves with marginal improvement, and is set to 5%. */ +#define NUMA_SCALE 1000 +#define NUMA_MOVE_THRESH 50
Please make that 1024, there's no reason not to use power of two here. This base 10 factor thing annoyed me no end already, its time for it to die.
Quoted 6 lines · On Tue, 29 Jul 2014 10:17:12 +0200 Peter Zijlstra <[email protected]> wrote:
Review
Quoted 2 lines
(quoted diff) +2
-0+#define NUMA_SCALE 1000 +#define NUMA_MOVE_THRESH 50Please make that 1024, there's no reason not to use power of two here. This base 10 factor thing annoyed me no end already, its time for it to die.
That's easy enough. However, it would be good to know whether this actually helps with the regression Aaron found :)
---8<---
Subject: sched,numa: prevent task moves with marginal benefit
Commit a43455a1d57 makes task_numa_migrate() always check the preferred node for task placement. This is causing a performance regression with hackbench, as well as SPECjbb2005.
Tracing task_numa_compare() with a single instance of SPECjbb2005 on a 4 node system, I have seen several thread swaps with tiny improvements.
It appears that the hysteresis code that was added to task_numa_compare is not doing what we needed it to do, and a simple threshold could be better.
Aaron, does this patch help, or am I barking up the wrong tree?
--- kernel/sched/fair.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-)
kernel/sched/fair.c +15 -9
@@ -924,10 +924,12 @@ static inline unsigned long group_faults_cpu(struct numa_group *group, int nid)/* * These return the fraction of accesses done by a particular task, or -* task group, on a particular numa node. The group weight is given a -* larger multiplier, in order to group tasks together that are almost -* evenly spread out between numa nodes. +* task group, on a particular numa node. The NUMA move threshold +* prevents task moves with marginal improvement, and is set to 5%. */ +#define NUMA_SCALE 1024 +#define NUMA_MOVE_THRESH (5 * NUMA_SCALE / 100) +static inline unsigned long task_weight (struct task_struct *p,int nid) {unsigned long total_faults;
@@ -940,7 +942,7 @@ static inline unsigned long task_weight(struct task_struct *p, int nid)if (!total_faults)return 0 ; -return 1000 *task_faults (p, nid) / total_faults; +return NUMA_SCALE *task_faults (p, nid) / total_faults; }static inline unsigned long group_weight (struct task_struct *p,int nid)
@@ -948,7 +950,7 @@ static inline unsigned long group_weight(struct task_struct *p, int nid)if (!p->numa_group || !p->numa_group->total_faults)return 0 ; -return 1000 *group_faults (p, nid) / p->numa_group->total_faults; +return NUMA_SCALE *group_faults (p, nid) / p->numa_group->total_faults; } bool should_numa_migrate_memory(struct task_struct *p, struct page * page,
@@ -1181,11 +1183,11 @@ static void task_numa_compare(struct task_numa_env *env, imp = taskimp +task_weight (cur, env->src_nid) -task_weight (cur, env->dst_nid);/* -* Add some hysteresis to prevent swapping the -* tasks within a group over tiny differences. +* Do not swap tasks within a group around unless +* there is a significant improvement. */ -if (cur->numa_group) - imp -= imp/16 ; +if (cur->numa_group && imp <NUMA_MOVE_THRESH ) + goto unlock; }else {/* * Compare the group weights. If a task is all by
@@ -1205,6 +1207,10 @@ static void task_numa_compare(struct task_numa_env *env, goto unlock;if (!cur) { +/* Only move if there is a significant improvement. */ +if (imp <NUMA_MOVE_THRESH ) + goto unlock; +/* Is there capacity@our destination? */ if (env->src_stats.has_free_capacity && !env->dst_stats.has_free_capacity)
Quoted 10 lines · Rik van Riel
Quoted 6 lines · On Tue, 29 Jul 2014 10:17:12 +0200 Peter Zijlstra <[email protected]> wrote:
Review
Quoted 2 lines
(quoted diff) +2
-0+#define NUMA_SCALE 1000 +#define NUMA_MOVE_THRESH 50Please make that 1024, there's no reason not to use power of two here. This base 10 factor thing annoyed me no end already, its time for it to die.
Quoted 2 lines
That's easy enough. However, it would be good to know whether this actually helps with the regression Aaron found :)
Sorry for the delay.
I applied the last patch and queued the hackbench job to the ivb42 test machine for it to run 5 times, and here is the result(regarding the proc-vmstat.numa_hint_faults_local field):
173565
201262
192317
198342
198595
avg:
192816
It seems it is still very big than previous kernels.
BTW, to highlight changes, we only include metrics that have changed a lot in the report, which means, for metrics that don't show in the report, it means it doesn't change much. But just in case, here is the throughput metric regarding commit a43455a1d(compared to its parent):
ebe06187bf2aec1 a43455a1d572daf7b730fe12e
--------------- -------------------------
118881 ~ 0% +1.2% 120325 ~ 0% ivb42/hackbench/50%-threads-pipe 78410 ~ 0% +0.6% 78857 ~ 0% lkp-snb01/hackbench/50%-threads-socket 197292 ~ 0% +1.0% 199182 ~ 0% TOTAL hackbench.throughput
Feel free to let me know if you need more information.
Thanks,
Aaron
Quoted 92 lines
---8<---
Subject: sched,numa: prevent task moves with marginal benefit
Commit
a43455a1d57makestask_numa_migrate()always check the preferred node for task placement. This is causing a performance regression with hackbench, as well as SPECjbb2005.Tracing
task_numa_compare()with a single instance of SPECjbb2005 on a 4 node system, I have seen several thread swaps with tiny improvements.It appears that the hysteresis code that was added to
task_numa_compareis not doing what we needed it to do, and a simple threshold could be better.Aaron, does this patch help, or am I barking up the wrong tree?
Reported-by Aaron Lu <[email protected]>Reported-by Jirka Hladky <[email protected]>Signed-off-by Rik van Riel <[email protected]>--- kernel/sched/fair.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-)kernel/sched/fair.c +15
-9@@ -924,10 +924,12 @@ static inline unsigned long group_faults_cpu(struct numa_group *group, int nid)/* * These return the fraction of accesses done by a particular task, or -* task group, on a particular numa node. The group weight is given a -* larger multiplier, in order to group tasks together that are almost -* evenly spread out between numa nodes. +* task group, on a particular numa node. The NUMA move threshold +* prevents task moves with marginal improvement, and is set to 5%. */ +#define NUMA_SCALE 1024 +#define NUMA_MOVE_THRESH (5 * NUMA_SCALE / 100) +static inline unsigned long task_weight (struct task_struct *p,int nid) {unsigned long total_faults;@@ -940,7 +942,7 @@ static inline unsigned long task_weight(struct task_struct *p, int nid)if (!total_faults)return 0 ; -return 1000 *task_faults (p, nid) / total_faults; +return NUMA_SCALE *task_faults (p, nid) / total_faults; }static inline unsigned long group_weight (struct task_struct *p,int nid)@@ -948,7 +950,7 @@ static inline unsigned long group_weight(struct task_struct *p, int nid)if (!p->numa_group || !p->numa_group->total_faults)return 0 ; -return 1000 *group_faults (p, nid) / p->numa_group->total_faults; +return NUMA_SCALE *group_faults (p, nid) / p->numa_group->total_faults; } bool should_numa_migrate_memory(struct task_struct *p, struct page * page,@@ -1181,11 +1183,11 @@ static void task_numa_compare(struct task_numa_env *env, imp = taskimp +task_weight (cur, env->src_nid) -task_weight (cur, env->dst_nid);/* -* Add some hysteresis to prevent swapping the -* tasks within a group over tiny differences. +* Do not swap tasks within a group around unless +* there is a significant improvement. */ -if (cur->numa_group) - imp -= imp/16 ; +if (cur->numa_group && imp <NUMA_MOVE_THRESH ) + goto unlock; }else {/* * Compare the group weights. If a task is all by @@ -1205,6 +1207,10 @@ static void task_numa_compare(struct task_numa_env *env, goto unlock;if (!cur) { +/* Only move if there is a significant improvement. */ +if (imp <NUMA_MOVE_THRESH ) + goto unlock; +/* Is there capacity at our destination? */ if (env->src_stats.has_free_capacity && !env->dst_stats.has_free_capacity)
Quoted 27 lines · Aaron Lu
Quoted 10 lines · Rik van Riel
Quoted 6 lines · On Tue, 29 Jul 2014 10:17:12 +0200 Peter Zijlstra <[email protected]> wrote:
Review
Quoted 2 lines
(quoted diff) +2
-0+#define NUMA_SCALE 1000 +#define NUMA_MOVE_THRESH 50Please make that 1024, there's no reason not to use power of two here. This base 10 factor thing annoyed me no end already, its time for it to die.
Quoted 2 lines
That's easy enough. However, it would be good to know whether this actually helps with the regression Aaron found :)
Sorry for the delay.
I applied the last patch and queued the hackbench job to the ivb42 test machine for it to run 5 times, and here is the
result(regarding the proc-vmstat.numa_hint_faults_local field):
173565
201262
192317
198342
198595
avg:
192816
Quoted 1 line
It seems it is still very big than previous kernels.
It looks like a step in the right direction, though.
Could you try running with a larger threshold?
Quoted 13 lines
kernel/sched/fair.c static inline unsigned long group_faults_cpu(struct numa_group *group, int nid) · line 931
Quoted 13 lines
+++
b/kernel/sched/fair.ckernel/sched/fair.c +4
-3@@ -924,10 +924,12 @@ static inline unsigned long group_faults_cpu(struct numa_group *group, int nid)/* * These return the fraction of accesses done by a particular task, or -* task group, on a particular numa node. The group weight is given a -* larger multiplier, in order to group tasks together that are almost -* evenly spread out between numa nodes. +* task group, on a particular numa node. The NUMA move threshold +* prevents task moves with marginal improvement, and is set to 5%. */ +#define NUMA_SCALE 1024 +#define NUMA_MOVE_THRESH (5 * NUMA_SCALE / 100)
It would be good to see if changing NUMA_MOVE_THRESH to (NUMA_SCALE / 8) does the trick.
I will run the same thing here with SPECjbb2005.
Quoted 37 lines · Rik van Riel
Quoted 27 lines · Aaron Lu
Quoted 10 lines · Rik van Riel
Quoted 6 lines · On Tue, 29 Jul 2014 10:17:12 +0200 Peter Zijlstra <[email protected]> wrote:
Review
Quoted 2 lines
(quoted diff) +2
-0+#define NUMA_SCALE 1000 +#define NUMA_MOVE_THRESH 50Please make that 1024, there's no reason not to use power of two here. This base 10 factor thing annoyed me no end already, its time for it to die.
Quoted 2 lines
That's easy enough. However, it would be good to know whether this actually helps with the regression Aaron found :)
Sorry for the delay.
I applied the last patch and queued the hackbench job to the ivb42 test machine for it to run 5 times, and here is the
result(regarding the proc-vmstat.numa_hint_faults_local field):
173565
201262
192317
198342
198595
avg:
192816Quoted 1 line
It seems it is still very big than previous kernels.
It looks like a step in the right direction, though.
Could you try running with a larger threshold?
Quoted 3 lines
Quoted 3 lines
+++
b/kernel/sched/fair.c(quoted diff) +0
-0@@ -924,10 +924,12 @@ static inline unsigned long group_faults_cpu(struct numa_group *group, int nid)
Quoted 13 lines
Quoted 10 lines
kernel/sched/fair.c static inline unsigned long group_faults_cpu(struct numa_group *group, int nid) · line 931
Quoted 10 lines
kernel/sched/fair.c +4
-3/* * These return the fraction of accesses done by a particular task, or -* task group, on a particular numa node. The group weight is given a -* larger multiplier, in order to group tasks together that are almost -* evenly spread out between numa nodes. +* task group, on a particular numa node. The NUMA move threshold +* prevents task moves with marginal improvement, and is set to 5%. */ +#define NUMA_SCALE 1024 +#define NUMA_MOVE_THRESH (5 * NUMA_SCALE / 100)It would be good to see if changing
NUMA_MOVE_THRESHto (NUMA_SCALE/ 8) does the trick.
With your 2nd patch and the above change, the result is:
"proc-vmstat.numa_hint_faults_local": [
199708,
209152,
200638,
187324,
196654
],
avg:
198695
Regards,
Aaron
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 46 lines · Aaron Lu
Quoted 32 lines · Rik van Riel
Quoted 14 lines · Aaron Lu
Quoted 5 lines · Rik van Riel
Quoted 2 lines · On Tue, 29 Jul 2014 10:17:12 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 1 line
+#define
NUMA_SCALE1000 +#defineNUMA_MOVE_THRESH50Quoted 7 lines
Quoted 3 lines
Please make that 1024, there's no reason not to use power of two here. This base 10 factor thing annoyed me no end already, its time for it to die.
That's easy enough. However, it would be good to know whether this actually helps with the regression Aaron found :)
Quoted 8 lines
Sorry for the delay.
I applied the last patch and queued the hackbench job to the ivb42 test machine for it to run 5 times, and here is the
result(regarding the proc-vmstat.numa_hint_faults_local field): 173565 201262 192317 198342 198595 avg: 192816It seems it is still very big than previous kernels.
It looks like a step in the right direction, though.
Could you try running with a larger threshold?
Quoted 4 lines
Quoted 4 lines
+++
b/kernel/sched/fair.c@@ -924,10 +924,12 @@ static inline unsigned longgroup_faults_cpu(struct numa_group *group, int nid)Quoted 12 lines
Quoted 9 lines
Quoted 9 lines
/* * These return the fraction of accesses done by a particular task, or - * task group, on a particular numa node. The group weight is given a - * larger multiplier, in order to group tasks together that are almost - * evenly spread out between numa nodes. + * task group, on a particular numa node. The NUMA move threshold + * prevents task moves with marginal improvement, and is set to 5%. */ +#define
NUMA_SCALE1024 +#defineNUMA_MOVE_THRESH(5 *NUMA_SCALE/ 100)It would be good to see if changing
NUMA_MOVE_THRESHto (NUMA_SCALE/ 8) does the trick.
Quoted 6 lines
With your 2nd patch and the above change, the result is:
"proc-vmstat.numa_hint_faults_local": [ 199708, 209152, 200638, 187324, 196654 ],
avg: 198695
OK, so it is still a little higher than your original 162245.
I guess this is to be expected, since the code will be more successful at placing a task on the right node, which results in the task scanning its memory more rapidly for a little bit.
Are you seeing any changes in throughput?
- --
All rights reversed
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJT2eC/AAoJEM553pKExN6DIFMH/23LsoEJ8cUqMTdWUzhXesEb TW0yncraZ6tDkGHopTU4oFmck93XUUVSJRVjLC3lxvxAIdWt8M4GCbWN8RD1yicX Ii9s18+2r2vkc30gkIgh2yahaqQUun9sUkuaQ4BaKlbP+hwQzB3OfU1GjR7iStFE t04krgCAL+xL63H/4mN0Y9ZjOBUz2QYbkspS21+oEWKkFY2FyyQn+hOSnA6lSvqy o7v4tmC8jtRXsQY+hfy1aOtMUZO5sRcYHOttlxgjE5MbnW/whhsC+oB7cWw646St LhvhhIykl/g2Bz+E3KbfnREGn5OO7NmEhv3am2Dj5XsNHnEfxYJH/m/aTA4az/s= =/IeV
-----END PGP SIGNATURE-----
Quoted 50 lines · Rik van Riel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 46 lines · Aaron Lu
Quoted 44 lines · Rik van Riel
Quoted 14 lines · Aaron Lu
Quoted 5 lines · Rik van Riel
Quoted 2 lines · On Tue, 29 Jul 2014 10:17:12 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 1 line
+#define
NUMA_SCALE1000 +#defineNUMA_MOVE_THRESH50Quoted 7 lines
Quoted 3 lines
Please make that 1024, there's no reason not to use power of two here. This base 10 factor thing annoyed me no end already, its time for it to die.
That's easy enough. However, it would be good to know whether this actually helps with the regression Aaron found :)
Quoted 8 lines
Sorry for the delay.
I applied the last patch and queued the hackbench job to the ivb42 test machine for it to run 5 times, and here is the
result(regarding the proc-vmstat.numa_hint_faults_local field): 173565 201262 192317 198342 198595 avg: 192816It seems it is still very big than previous kernels.
It looks like a step in the right direction, though.
Could you try running with a larger threshold?
Quoted 4 lines
Quoted 4 lines
+++
b/kernel/sched/fair.c@@ -924,10 +924,12 @@ static inline unsigned longgroup_faults_cpu(struct numa_group *group, int nid)Quoted 9 lines
Quoted 9 lines
/* * These return the fraction of accesses done by a particular task, or - * task group, on a particular numa node. The group weight is given a - * larger multiplier, in order to group tasks together that are almost - * evenly spread out between numa nodes. + * task group, on a particular numa node. The NUMA move threshold + * prevents task moves with marginal improvement, and is set to 5%. */ +#define
NUMA_SCALE1024 +#defineNUMA_MOVE_THRESH(5 *NUMA_SCALE/ 100)It would be good to see if changing
NUMA_MOVE_THRESHto (NUMA_SCALE/ 8) does the trick.
Quoted 8 lines
Quoted 6 lines
With your 2nd patch and the above change, the result is:
"proc-vmstat.numa_hint_faults_local": [ 199708, 209152, 200638, 187324, 196654 ],
avg: 198695
OK, so it is still a little higher than your original 162245.
The original number is 94500 for ivb42 machine, the 162245 is the sum of the two numbers above it that are tested on two machines - one is the number for ivb42 and one is for lkp-snb01. Sorry if that is not clear.
And for the numbers I have given with your patch applied, they are all for ivb42 alone.
Quoted 6 lines
I guess this is to be expected, since the code will be more successful at placing a task on the right node, which results in the task scanning its memory more rapidly for a little bit.
Are you seeing any changes in throughput?
The throughput has almost no change. Your 2nd patch with scale changed has seen a decrease of 0.1% compared to your original commit that triggered the report, and that original commit has a increase of 1.2% compared to its parent commit.
Regards,
Aaron
Quoted 8 lines · On Thu, 31 Jul 2014 13:04:54 +0800 Aaron Lu <[email protected]> wrote:
Quoted 7 lines · Rik van Riel
Quoted 2 lines · Aaron Lu
kernel/sched/fair.c static inline unsigned long group_faults_cpu(struct numa_group *group, int nid) · line 931
Quoted 2 lines
kernel/sched/fair.c +2
-0+#define NUMA_SCALE 1024 +#define NUMA_MOVE_THRESH (5 * NUMA_SCALE / 100)It would be good to see if changing
NUMA_MOVE_THRESHto (NUMA_SCALE/ 8) does the trick.
FWIW, running with NUMA_MOVE_THRESH set to (NUMA_SCALE / 8) seems to resolve the SPECjbb2005 threshold on my system.
I will run some more sanity tests later today...
Signature · 1 line
All rights reversed.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 17 lines · Aaron Lu
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
+++
b/kernel/sched/fair.c@@ -924,10 +924,12 @@ static inline unsigned longgroup_faults_cpu(struct numa_group *group, int nid)Quoted 12 lines
Quoted 9 lines
Quoted 9 lines
/* * These return the fraction of accesses done by a particular task, or - * task group, on a particular numa node. The group weight is given a - * larger multiplier, in order to group tasks together that are almost - * evenly spread out between numa nodes. + * task group, on a particular numa node. The NUMA move threshold + * prevents task moves with marginal improvement, and is set to 5%. */ +#define
NUMA_SCALE1024 +#defineNUMA_MOVE_THRESH(5 *NUMA_SCALE/ 100)It would be good to see if changing
NUMA_MOVE_THRESHto (NUMA_SCALE/ 8) does the trick.
Quoted 1 line
With your 2nd patch and the above change, the result is:
Peter,
the threshold does not seem to make a difference for the performance tests on my system, I guess you can drop this patch :)
- --
All rights reversed
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJT4U/xAAoJEM553pKExN6DY4oH/ihJDmcCSZ0sKqGbyzJqLrFY KWCEXhfiN6hQJBrmeOvrbzlHsMH0LzYfgTVnc1nteAcnUXiBeqkgxwf+S1dmvoFr DZSxC+9tQ68ho0YcLd7rpEMfsnwOQAB9BgX8GxxwMb8q5zZ9Bz3r9NKVF0P2D3cj eeJ8Z3EGaKOteVhwAPVPeuTf7xwhqoqp4ujLgTL7BcaifqvGhi3+uo9/KcavE15d eale3MuhbCIsAQeyB4SwgGwilE/oZTPTos4BNdUrIyxO4nDajbeLb1qsLSHYcirH CA7++bTE9V6TvO1tBLVpeYdSAGcDKKUBHM6N+0UDwkR/Tp4oRyQ115Peo2H34ak= =kFxZ
-----END PGP SIGNATURE-----
Peter Zijlstra
maintainerQuoted 1 line · Aaron Lu
118881 ~ 0% +1.2% 120325 ~ 0% ivb42/hackbench/50%-threads-pipe
What kind of IVB is that EP or EX (or rather, how many sockets)? Also what arguments to hackbench do you use?
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Aaron Lu
118881 ~ 0% +1.2% 120325 ~ 0% ivb42/hackbench/50%-threads-pipe
What kind of IVB is that EP or EX (or rather, how many sockets)? Also what arguments to hackbench do you use?
2 sockets EP.
The cmdline is:/usr/bin/hackbench -g 24 --threads --pipe -l 60000
Regards,
Aaron
Peter Zijlstra
maintainerQuoted 14 lines · Rik van Riel
Quoted 10 lines · On Tue, 29 Jul 2014 13:24:05 +0800 Aaron Lu <[email protected]> wrote:
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitmaster
commita43455a1d572daf7b730fe12eb747d1e17411365("sched/numa: Ensuretask_numa_migrate()checks the preferred node")
ebe06187bf2aec1a43455a1d572daf7b730fe12e
--------------- -------------------------94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_local
Quoted 4 lines
Hi Aaron,
Jirka Hladky has reported a regression with that changeset as well, and I have already spent some time debugging the issue.
So assuming those numbers above are the difference in numa_hint_local_faults, the report is actually a significant _improvement_, not a regression.
On my IVB-EP I get similar numbers; using:
PRE=`grep numa_hint_faults_local /proc/vmstat | cut -d' ' -f2` perf bench sched messaging -g 24 -t -p -l 60000 POST=`grep numa_hint_faults_local /proc/vmstat | cut -d' ' -f2` echo $((POST-PRE))
tip/mater+origin/master tip/master+origin/master-a43455a1d57
local total local total
faults time faults time
19971 51.384 10104 50.838 17193 50.564 9116 50.208 13435 49.057 8332 51.344 23794 50.795 9954 51.364 20255 49.463 9598 51.258
18929.6 50.2526 9420.8 51.0024
3863.61 0.96 717.78 0.49
So that patch improves both local faults and runtime. Its good (even though for the runtime we're still inside stdev overlap, so ideally I'd do more runs).
Now I also did a run with the proposed patch, NUMA_SCALE/8 variant, and that slightly reduces both again:
tip/master+origin/master+patch
local total
faults time
21296 50.541
12771 50.54
13872 52.224
23352 50.85
16516 50.705
17561.4 50.972
4613.32 0.71
So for hackbench a43455a1d57 is good and the proposed patch is making things worse.
Let me see if I can still find my SPECjbb2005 copy to see what that does.
Peter Zijlstra
maintainerQuoted 15 lines · Peter Zijlstra
Quoted 14 lines · Rik van Riel
Quoted 10 lines · On Tue, 29 Jul 2014 13:24:05 +0800 Aaron Lu <[email protected]> wrote:
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitmaster
commita43455a1d572daf7b730fe12eb747d1e17411365("sched/numa: Ensuretask_numa_migrate()checks the preferred node")
ebe06187bf2aec1a43455a1d572daf7b730fe12e
--------------- -------------------------94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_local
Quoted 7 lines
Quoted 4 lines
Hi Aaron,
Jirka Hladky has reported a regression with that changeset as well, and I have already spent some time debugging the issue.
Let me see if I can still find my SPECjbb2005 copy to see what that does.
Jirka, what kind of setup were you seeing SPECjbb regressions?
I'm not seeing any on 2 sockets with a single SPECjbb instance, I'll go check one instance per socket now.
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Rik van Riel
Quoted 10 lines · On Tue, 29 Jul 2014 13:24:05 +0800 Aaron Lu <[email protected]> wrote:
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitmaster
commita43455a1d572daf7b730fe12eb747d1e17411365("sched/numa: Ensuretask_numa_migrate()checks the preferred node")
ebe06187bf2aec1a43455a1d572daf7b730fe12e
--------------- -------------------------94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_localHi Aaron,
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
Jirka Hladky has reported a regression with that changeset as well, and I have already spent some time debugging the issue.
Let me see if I can still find my SPECjbb2005 copy to see what that does.
Jirka, what kind of setup were you seeing SPECjbb regressions?
I'm not seeing any on 2 sockets with a single SPECjbb instance, I'll go check one instance per socket now.
Peter, I'm seeing regressions for
SINGLE SPECjbb instance for number of warehouses being the same as total number of cores in the box.
Example: 4 NUMA node box, each CPU has 6 cores => biggest regression is for 24 warehouses.
See the attached snapshot.
Jirka
Peter Zijlstra
maintainerQuoted 28 lines · Jirka Hladky
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Rik van Riel
Quoted 10 lines · On Tue, 29 Jul 2014 13:24:05 +0800 Aaron Lu <[email protected]> wrote:
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitmaster
commita43455a1d572daf7b730fe12eb747d1e17411365("sched/numa: Ensuretask_numa_migrate()checks the preferred node")
ebe06187bf2aec1a43455a1d572daf7b730fe12e
--------------- -------------------------94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_localHi Aaron,
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
Jirka Hladky has reported a regression with that changeset as well, and I have already spent some time debugging the issue.
Let me see if I can still find my SPECjbb2005 copy to see what that does.
Jirka, what kind of setup were you seeing SPECjbb regressions?
I'm not seeing any on 2 sockets with a single SPECjbb instance, I'll go check one instance per socket now.
Quoted 7 lines
Peter, I'm seeing regressions for
SINGLE SPECjbb instance for number of warehouses being the same as total number of cores in the box.
Example: 4 NUMA node box, each CPU has 6 cores => biggest regression is for 24 warehouses.
IVB-EP: 2 node, 10 cores, 2 thread per core:
tip/master+origin/master:
Warehouses Thrput
4 196781
8 358064
12 511318
16 589251
20 656123
24 710789
28 765426
32 787059
36 777899
* 40 748568Throughput 18258
Warehouses Thrput
4 201598
8 363470
12 512968
16 584289
20 605299
24 720142
28 776066
32 791263
36 776965
* 40 760572Throughput 18551
tip/master+origin/master-a43455a1d57
SPEC scores
Warehouses Thrput
4 198667
8 362481
12 503344
16 582602
20 647688
24 731639
28 786135
32 794124
36 774567
* 40 757559Throughput 18477
Given that there's fairly large variance between the two runs with the commit in, I'm not sure I can say there's a problem here.
The one run without the patch is more or less between the two runs with the patch.
And doing this many runs takes ages, so I'm not tempted to either make the runs longer or do more of them.
Lemme try on a 4 node box though, who knows.
Quoted 90 lines · Peter Zijlstra
Quoted 28 lines · Jirka Hladky
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Rik van Riel
Quoted 10 lines · On Tue, 29 Jul 2014 13:24:05 +0800 Aaron Lu <[email protected]> wrote:
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitmaster
commita43455a1d572daf7b730fe12eb747d1e17411365("sched/numa: Ensuretask_numa_migrate()checks the preferred node")
ebe06187bf2aec1a43455a1d572daf7b730fe12e
--------------- -------------------------94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_localHi Aaron,
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
Jirka Hladky has reported a regression with that changeset as well, and I have already spent some time debugging the issue.
Let me see if I can still find my SPECjbb2005 copy to see what that does.
Jirka, what kind of setup were you seeing SPECjbb regressions?
I'm not seeing any on 2 sockets with a single SPECjbb instance, I'll go check one instance per socket now.
Quoted 7 lines
Peter, I'm seeing regressions for
SINGLE SPECjbb instance for number of warehouses being the same as total number of cores in the box.
Example: 4 NUMA node box, each CPU has 6 cores => biggest regression is for 24 warehouses.
IVB-EP: 2 node, 10 cores, 2 thread per core:
tip/master+origin/master:
Warehouses Thrput 4 196781 8 358064 12 511318 16 589251 20 656123 24 710789 28 765426 32 787059 36 777899 * 40 748568Throughput 18258
Warehouses Thrput 4 201598 8 363470 12 512968 16 584289 20 605299 24 720142 28 776066 32 791263 36 776965 * 40 760572Throughput 18551
tip/master+origin/master-a43455a1d57
SPEC scores Warehouses Thrput 4 198667 8 362481 12 503344 16 582602 20 647688 24 731639 28 786135 32 794124 36 774567 * 40 757559Throughput 18477
Given that there's fairly large variance between the two runs with the commit in, I'm not sure I can say there's a problem here.
Quoted 7 lines
The one run without the patch is more or less between the two runs with the patch.
And doing this many runs takes ages, so I'm not tempted to either make the runs longer or do more of them.
Lemme try on a 4 node box though, who knows.
IVB-EP: 2 node, 10 cores, 2 thread per core
=> on such system, I run only 20 warenhouses as maximum. (number of nodes * number of PHYSICAL cores)
The kernels you have tested shows following results:
656123/605299/647688
I'm doing 3 iterations (3 runs) to get some statistics. To speed up the test significantly please do the run with 20 warehouses only (or in general with #warehouses == number of nodes * number of PHYSICAL cores)
Jirka
Peter Zijlstra
maintainerQuoted 4 lines · Jirka Hladky
I'm doing 3 iterations (3 runs) to get some statistics. To speed up the test significantly please do the run with 20 warehouses only
(or in general with #warehouses == number of nodes * number of PHYSICAL cores)
Yeah, went and did that for my 4 node machine, its got a ton more cores, but I matches the warehouses to it:
-a43455a1d57 tip/master
979996.47 1144715.44
876146 1098499.07
1058974.18 1019499.38
1055951.59 1139405.22
970504.01 1099659.09
988314.45 1100355.64 (avg)
75059.546179565 50085.7473975167(stdev)
So for 5 runs, tip/master (which includes the offending patch) wins hands down.
Each run is 2 minutes.
Peter Zijlstra
maintainerQuoted 17 lines · Peter Zijlstra
Quoted 4 lines · Jirka Hladky
I'm doing 3 iterations (3 runs) to get some statistics. To speed up the test significantly please do the run with 20 warehouses only
(or in general with #warehouses == number of nodes * number of PHYSICAL cores)Yeah, went and did that for my 4 node machine, its got a ton more cores, but I matches the warehouses to it:
-a43455a1d57 tip/master
979996.47 1144715.44
876146 1098499.07
1058974.18 1019499.38
1055951.59 1139405.22
970504.01 1099659.09
Quoted 6 lines
988314.45 1100355.64 (avg)
75059.546179565 50085.7473975167(stdev)So for 5 runs, tip/master (which includes the offending patch) wins hands down.
Each run is 2 minutes.
Because Rik asked for a43455a1d57^1 numbers:
546423.08
546558.63
545990.01
546015.98
some a43455a1d57 numbers:
538652.93
544333.57
542684.77
same setup and everything. So clearly the patches after that made 'some' difference indeed, seeing how tip/master is almost twice that.
So the reason I didn't so a43455a1d57^1 vs a43455a1d57 is because we already fingered a commit, after that what you test is the revert of that commit, because revert is what you typically end up doing if a commit is fail.
But on the state of tip/master, taking that commit out is a net negative for everything I've tested.
Quoted 7 lines · Jirka Hladky
Peter, I'm seeing regressions for
SINGLE SPECjbb instance for number of warehouses being the same as total number of cores in the box.
Example: 4 NUMA node box, each CPU has 6 cores => biggest regression is for 24 warehouses.
By looking at your graph, that's around a 10% difference.
So I'm not seeing anywhere near as bad a regression on a 80-core box. Testing single with 80 warehouses, I get:
tip/master baseline:
677476.36 bops
705826.70 bops
704870.87 bops
681741.20 bops
707014.59 bops
Avg: 695385.94 bops
tip/master + patch (NUMA_SCALE/8 variant):
698242.66 bops
693873.18 bops
707852.28 bops
691785.96 bops
747206.03 bopsthis
Avg: 707792.022 bops
So both these are pretty similar, however, when reverting, on avg we increase the amount of bops a mere ~4%:
tip/master + reverted:
778416.02 bops
702602.62 bops
712557.32 bops
713982.90 bops
783300.36 bops
Avg: 738171.84 bops
Are there perhaps any special specjbb options you are using?
Quoted 4 lines · Davidlohr Bueso
So both these are pretty similar, however, when reverting, on avg we increase the amount of bops a mere ~4%:
tip/master + reverted:
Just to be clear, this is reverting a43455a1d57.
Quoted 41 lines · Davidlohr Bueso
Quoted 7 lines · Jirka Hladky
Peter, I'm seeing regressions for
SINGLE SPECjbb instance for number of warehouses being the same as total number of cores in the box.
Example: 4 NUMA node box, each CPU has 6 cores => biggest regression is for 24 warehouses.
By looking at your graph, that's around a 10% difference.
So I'm not seeing anywhere near as bad a regression on a 80-core box. Testing single with 80 warehouses, I get:
tip/master baseline:
677476.36 bops
705826.70 bops
704870.87 bops
681741.20 bops
707014.59 bopsAvg: 695385.94 bops
tip/master + patch (NUMA_SCALE/8 variant):
698242.66 bops
693873.18 bops
707852.28 bops
691785.96 bops
747206.03 bopsthisAvg: 707792.022 bops
So both these are pretty similar, however, when reverting, on avg we increase the amount of bops a mere ~4%:
tip/master + reverted:
778416.02 bops
702602.62 bops
712557.32 bops
713982.90 bops
783300.36 bops
Quoted 4 lines
Avg: 738171.84 bops
Are there perhaps any special specjbb options you are using?
I see the regression only on this box. It has 4 "Ivy Bridge-EX" Xeon E7-4890 v2 CPUs.
http://ark.intel.com/products/75251
http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#.22Ivy_Bridge-EX.22_.2822_nm.29_Expandable_2
Please rerun the test on box with Ivy Bridge CPUs. It seems that older CPU generations are not affected.
Thanks
Jirka
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 10 lines · Jirka Hladky
I see the regression only on this box. It has 4 "Ivy Bridge-EX" Xeon E7-4890 v2 CPUs.
http://ark.intel.com/products/75251
http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#.22Ivy_Bridge-EX.22_.2822_nm.29_Expandable_2Please rerun the test on box with Ivy Bridge CPUs. It seems that older CPU generations are not affected.
That would have been good info to know :)
I've been spending about a month trying to reproduce your issue on a Westmere E7-4860.
Good thing I found all kinds of other scheduler issues along the way...
- --
All rights reversed
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJT3GZBAAoJEM553pKExN6D4FcH/2c/kYOZkbJeLBEWJHB0yWNR tqI2Lt/qxPxOKADlDylJwj2Dq8R19Cc4tnJZAdPh+wgCivFefseQY0MI1TI8CO/Z vEH+dCG8hokygFxKqAX9udI0MD1OxfTKKIk4fdjInZ632JG+JHnqVH6qWxBsriXD 151jzCR/zQEjg6gyCc8YsL06Q9YHyVv7dakggtRkYnE1GIUAtTDhFttRpNYoiVQQ y/d32adq//PywTmsyWwJMu1ZGe1eGC57JBYzjoUo2iOlFQ9QR+fe4W2/6ZCbekwK O8ZYbrJzDGrNQP2yDYd+o040KeVfzYkOtwz7+/40TYIvqFiuvKxEAxbJ32+krxA= =XxCE
-----END PGP SIGNATURE-----
Quoted 13 lines · Rik van Riel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 8 lines · Jirka Hladky
I see the regression only on this box. It has 4 "Ivy Bridge-EX" Xeon E7-4890 v2 CPUs.
http://ark.intel.com/products/75251
http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#.22Ivy_Bridge-EX.22_.2822_nm.29_Expandable_2
Quoted 8 lines
Quoted 2 lines
Please rerun the test on box with Ivy Bridge CPUs. It seems that older CPU generations are not affected.
That would have been good info to know :)
I've been spending about a month trying to reproduce your issue on a Westmere E7-4860.
Good thing I found all kinds of other scheduler issues along the way...
Hi Rik,
till recently I have seen the regression on all systems.
With the latest kernel, only Ivy Bridge system seems to be affected.
Jirka
Peter Zijlstra
maintainerQuoted 2 lines · Jirka Hladky
I see the regression only on this box. It has 4 "Ivy Bridge-EX" Xeon E7-4890 v2 CPUs.
That's the exact CPU I've got in the 4 node machine I did the tests on.
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Rik van Riel
Quoted 10 lines · On Tue, 29 Jul 2014 13:24:05 +0800 Aaron Lu <[email protected]> wrote:
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitmaster
commita43455a1d572daf7b730fe12eb747d1e17411365("sched/numa: Ensuretask_numa_migrate()checks the preferred node")
ebe06187bf2aec1a43455a1d572daf7b730fe12e
--------------- -------------------------94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_local
Quoted 6 lines
Quoted 4 lines
Hi Aaron,
Jirka Hladky has reported a regression with that changeset as well, and I have already spent some time debugging the issue.
So assuming those numbers above are the difference in
Yes, they are.
It means, for commit ebe06187bf2aec1, the number for num_hint_local_faults is 94500 for ivb42 machine and 67745 for lkp-snb01 machine. The 3%, 4% following that number means the deviation of the different runs to their average(we usually run it multiple times to phase out possible sharp values). We should probably remove that percentage, as they cause confusion if no detailed explanation and may not mean much to the commit author and others(if the deviation is big enough, we should simply drop that result).
The percentage in the middle is the change between the two commits.
Another thing is the meaning of the numbers, it doesn't seem that evident they are for proc-vmstat.numa_hint_faults_local. Maybe something like this is better?
ebe06187bf2aec1 a43455a1d572daf7b730fe12e proc-vmstat.numa_hint_faults_local
--------------- ------------------------- -----------------------------
94500 +115.6% 203711 ivb42/hackbench/50%-threads-pipe
67745 +64.1% 111174 lkp-snb01/hackbench/50%-threads-socket
162245 +94.1% 314885 TOTAL Regards,
Aaron
Quoted 52 lines
numa_hint_local_faults, the report is actually a significant _improvement_, not a regression.On my IVB-EP I get similar numbers; using:
PRE=`grep numa_hint_faults_local /proc/vmstat | cut -d' ' -f2` perf bench sched messaging -g 24 -t -p -l 60000 POST=`grep numa_hint_faults_local /proc/vmstat | cut -d' ' -f2` echo $((POST-PRE))tip/mater+origin/master tip/master+origin/master-a43455a1d57
local total local total
faults time faults time19971 51.384 10104 50.838 17193 50.564 9116 50.208 13435 49.057 8332 51.344 23794 50.795 9954 51.364 20255 49.463 9598 51.25818929.6 50.2526 9420.8 51.0024
3863.61 0.96 717.78 0.49So that patch improves both local faults and runtime. Its good (even though for the runtime we're still inside stdev overlap, so ideally I'd do more runs).
Now I also did a run with the proposed patch, NUMA_SCALE/8 variant, and that slightly reduces both again:
tip/master+origin/master+patch
local total
faults time21296 50.541
12771 50.54
13872 52.224
23352 50.85
16516 50.70517561.4 50.972
4613.32 0.71So for hackbench
a43455a1d57is good and the proposed patch is making things worse.Let me see if I can still find my SPECjbb2005 copy to see what that does.
Peter Zijlstra
maintainerQuoted 6 lines · Aaron Lu
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
ebe06187bf2aec1a43455a1d572daf7b730fe12e
--------------- -------------------------94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_local
Quoted 8 lines
It means, for commit
ebe06187bf2aec1, the number fornum_hint_local_faultsis 94500 for ivb42 machine and 67745 for lkp-snb01 machine. The 3%, 4% following that number means the deviation of the different runs to their average(we usually run it multiple times to phase out possible sharp values). We should probably remove that percentage, as they cause confusion if no detailed explanation and may not mean much to the commit author andothers(if the deviation is big enough, we should simply drop that result).
Nah, variance is good, but the typical symbol would be +- or the fancy ±.
~ when used as a unary op means 'approx' or 'about' or 'same order' ~ when used as a binary op means equivalence, a weaker equal, often in the vein of the unary op meaning.
Also see: http://en.wikipedia.org/wiki/Tilde#Mathematics
So while I think having a measure of variance is good, I think you picked entirely the wrong symbol.
Quoted 19 lines · Peter Zijlstra
Quoted 6 lines · Aaron Lu
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
ebe06187bf2aec1a43455a1d572daf7b730fe12e
--------------- -------------------------94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_localQuoted 8 lines
It means, for commit
ebe06187bf2aec1, the number fornum_hint_local_faultsis 94500 for ivb42 machine and 67745 for lkp-snb01 machine. The 3%, 4% following that number means the deviation of the different runs to their average(we usually run it multiple times to phase out possible sharp values). We should probably remove that percentage, as they cause confusion if no detailed explanation and may not mean much to the commit author andothers(if the deviation is big enough, we should simply drop that result).Nah, variance is good, but the typical symbol would be +- or the fancy ±.
Quoted 8 lines
~ when used as a unary op means 'approx' or 'about' or 'same order' ~ when used as a binary op means equivalence, a weaker equal, often in the vein of the unary op meaning.
Also see: http://en.wikipedia.org/wiki/Tilde#Mathematics
So while I think having a measure of variance is good, I think you picked entirely the wrong symbol.
Or, maybe you can use σ (lower case sigma) to indicate stddev, :)
Quoted 19 lines · Peter Zijlstra
Quoted 6 lines · Aaron Lu
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
ebe06187bf2aec1a43455a1d572daf7b730fe12e
--------------- -------------------------94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_localQuoted 8 lines
It means, for commit
ebe06187bf2aec1, the number fornum_hint_local_faultsis 94500 for ivb42 machine and 67745 for lkp-snb01 machine. The 3%, 4% following that number means the deviation of the different runs to their average(we usually run it multiple times to phase out possible sharp values). We should probably remove that percentage, as they cause confusion if no detailed explanation and may not mean much to the commit author andothers(if the deviation is big enough, we should simply drop that result).Nah, variance is good, but the typical symbol would be +- or the fancy ±.
Quoted 8 lines
~ when used as a unary op means 'approx' or 'about' or 'same order' ~ when used as a binary op means equivalence, a weaker equal, often in the vein of the unary op meaning.
Also see: http://en.wikipedia.org/wiki/Tilde#Mathematics
So while I think having a measure of variance is good, I think you picked entirely the wrong symbol.
Good point! We'll first try ± for the stddev percent and fall back to +- if it turn out to not work well in some cases.
Thanks,
Fengguang
Quoted 34 lines · Aaron Lu
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Rik van Riel
Quoted 10 lines · On Tue, 29 Jul 2014 13:24:05 +0800 Aaron Lu <[email protected]> wrote:
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitmaster
commita43455a1d572daf7b730fe12eb747d1e17411365("sched/numa: Ensuretask_numa_migrate()checks the preferred node")
ebe06187bf2aec1a43455a1d572daf7b730fe12e
--------------- -------------------------94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_localQuoted 6 lines
Quoted 4 lines
Hi Aaron,
Jirka Hladky has reported a regression with that changeset as well, and I have already spent some time debugging the issue.
So assuming those numbers above are the difference in
Yes, they are.
It means, for commit
ebe06187bf2aec1, the number fornum_hint_local_faultsis 94500 for ivb42 machine and 67745 for lkp-snb01 machine. The 3%, 4% following that number means the deviation of the different runs to their average(we usually run it multiple times to phase out possible sharp values). We should probably remove that percentage, as they cause confusion if no detailed explanation and may not mean much to the commit author andothers(if the deviation is big enough, we should simply drop that result).
Quoted 5 lines
The percentage in the middle is the change between the two commits.
Another thing is the meaning of the numbers, it doesn't seem that evident they are for proc-vmstat.numa_hint_faults_local. Maybe something like this is better?
Instead of removing info, why not document what each piece of data represents. Or add headers to the table. etc.
Thanks,
Davidlohr
Peter Zijlstra
maintainerQuoted 3 lines · Davidlohr Bueso
Instead of removing info, why not document what each piece of data represents. Or add headers to the table. etc.
Yes headers are good, knowing exactly what a number is often removes a lot of confusion ;-)
Quoted 65 lines · Peter Zijlstra
Quoted 14 lines · Rik van Riel
Quoted 10 lines · On Tue, 29 Jul 2014 13:24:05 +0800 Aaron Lu <[email protected]> wrote:
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitmaster
commita43455a1d572daf7b730fe12eb747d1e17411365("sched/numa: Ensuretask_numa_migrate()checks the preferred node")
ebe06187bf2aec1a43455a1d572daf7b730fe12e
--------------- -------------------------94500 ~ 3% +115.6% 203711 ~ 6% ivb42/hackbench/50%-threads-pipe 67745 ~ 4% +64.1% 111174 ~ 5% lkp-snb01/hackbench/50%-threads-socket 162245 ~ 3% +94.1% 314885 ~ 6% TOTAL proc-vmstat.numa_hint_faults_localQuoted 4 lines
Hi Aaron,
Jirka Hladky has reported a regression with that changeset as well, and I have already spent some time debugging the issue.
So assuming those numbers above are the difference in
numa_hint_local_faults, the report is actually a significant _improvement_, not a regression.On my IVB-EP I get similar numbers; using:
PRE=`grep numa_hint_faults_local /proc/vmstat | cut -d' ' -f2` perf bench sched messaging -g 24 -t -p -l 60000 POST=`grep numa_hint_faults_local /proc/vmstat | cut -d' ' -f2` echo $((POST-PRE))tip/mater+origin/master tip/master+origin/master-a43455a1d57
local total local total
faults time faults time19971 51.384 10104 50.838 17193 50.564 9116 50.208 13435 49.057 8332 51.344 23794 50.795 9954 51.364 20255 49.463 9598 51.25818929.6 50.2526 9420.8 51.0024
3863.61 0.96 717.78 0.49So that patch improves both local faults and runtime. Its good (even though for the runtime we're still inside stdev overlap, so ideally I'd do more runs).
Now I also did a run with the proposed patch, NUMA_SCALE/8 variant, and that slightly reduces both again:
tip/master+origin/master+patch
local total
faults time21296 50.541
12771 50.54
13872 52.224
23352 50.85
16516 50.705
Quoted 5 lines
17561.4 50.972
4613.32 0.71So for hackbench
a43455a1d57is good and the proposed patch is making things worse.
It also seems to be the case on a 8-socket 80 core DL980:
tip/master baseline:
67276 169.590 [sec]
82400 188.406 [sec]
87827 201.122 [sec]
96659 228.243 [sec]
83180 192.422 [sec]
tip/master + a43455a1d57 reverted
36686 170.373 [sec]
52670 187.904 [sec]
55723 203.597 [sec]
41780 174.354 [sec]
36070 173.179 [sec]
Runtimes are pretty much all over the place, cannot really say if it's gotten slower or faster. However, on avg, we nearly double the amount of hint local faults with the commit in question.
After adding the proposed fix (NUMA_SCALE/8 variant), it goes down again, closer to without a43455a1d57"
tip/master + patch
50591 175.272 [sec]
57858 191.969 [sec]
77564 215.429 [sec]
50613 179.384 [sec]
61673 201.694 [sec]
Quoted 2 lines
Let me see if I can still find my SPECjbb2005 copy to see what that does.
I'll try to dig it up as well.