sched/fair: Fix group power_orig computation
tip-bot for Peter Zijlstra
merge botCommit-ID: 863bffc80898b8df295ebac111af2335ec05f85d Gitweb: http://git.kernel.org/tip/863bffc80898b8df295ebac111af2335ec05f85d Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 28 Aug 2013 11:44:39 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Thu, 12 Sep 2013 19:14:43 +0200
sched/fair: Fix group power_orig computation
When looking at the code I noticed we don't actually compute sgp->power_orig correctly for groups, fix that.
Currently the only consumer of that value is fix_small_capacity() which is only used on POWER7+ and that code excludes this case by being limited to SD_SHARE_CPUPOWER which is only ever set on the SMT domain which must be the lowest domain and this has singleton groups.
So nothing should be affected by this change.
--- kernel/sched/fair.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
kernel/sched/fair.c +11 -5
@@ -4450,7 +4450,7 @@ void update_group_power(struct sched_domain *sd, int cpu) {struct sched_domain *child = sd->child;struct sched_group *group, *sdg = sd->groups; -unsigned long power; +unsigned long power, power_orig;unsigned long interval; interval =msecs_to_jiffies (sd->balance_interval);
@@ -4462,7 +4462,7 @@ void update_group_power(struct sched_domain *sd, int cpu)return ; } - power =0 ; + power_orig = power =0 ;if (child->flags &SD_OVERLAP ) {/*
@@ -4470,8 +4470,12 @@ void update_group_power(struct sched_domain *sd, int cpu)* span the current group. */ -for_each_cpu (cpu,sched_group_cpus (sdg)) - power +=power_of (cpu); +for_each_cpu (cpu,sched_group_cpus (sdg)) { +struct sched_group *sg =cpu_rq (cpu)->sd->groups; + + power_orig += sg->sgp->power_orig; + power += sg->sgp->power; + } }else {/* * !SD_OVERLAP domains can assume that child groups
@@ -4480,12 +4484,14 @@ void update_group_power(struct sched_domain *sd, int cpu) group = child->groups;do { + power_orig += group->sgp->power_orig; power += group->sgp->power; group = group->next; }while (group != child->groups); } - sdg->sgp->power_orig = sdg->sgp->power = power; + sdg->sgp->power_orig = power_orig; + sdg->sgp->power = power; }/*
Quoted 17 lines · tip-bot for Peter Zijlstra
Commit-ID: 863bffc80898b8df295ebac111af2335ec05f85d Gitweb: http://git.kernel.org/tip/863bffc80898b8df295ebac111af2335ec05f85d Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 28 Aug 2013 11:44:39 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Thu, 12 Sep 2013 19:14:43 +0200sched/fair: Fix group
power_origcomputationWhen looking at the code I noticed we don't actually compute
sgp->power_origcorrectly for groups, fix that.Currently the only consumer of that value is
fix_small_capacity()which is only used on POWER7+ and that code excludes this case by being limited toSD_SHARE_CPUPOWERwhich is only ever set on the SMT domain which must be the lowest domain and this has singleton groups.
Quoted 6 lines
So nothing should be affected by this change.
Cc Michael Neuling <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Ingo Molnar <[email protected]>
FWIW, this doesn't seem to break POWER7.
Thanks!
Mikey
Quoted 58 lines
--- kernel/sched/fair.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)kernel/sched/fair.c +11
-5@@ -4450,7 +4450,7 @@ void update_group_power(struct sched_domain *sd, int cpu) {struct sched_domain *child = sd->child;struct sched_group *group, *sdg = sd->groups; -unsigned long power; +unsigned long power, power_orig;unsigned long interval; interval =msecs_to_jiffies (sd->balance_interval);@@ -4462,7 +4462,7 @@ void update_group_power(struct sched_domain *sd, int cpu)return ; } - power =0 ; + power_orig = power =0 ;if (child->flags &SD_OVERLAP ) {/* @@ -4470,8 +4470,12 @@ void update_group_power(struct sched_domain *sd, int cpu)* span the current group. */ -for_each_cpu (cpu,sched_group_cpus (sdg)) - power +=power_of (cpu); +for_each_cpu (cpu,sched_group_cpus (sdg)) { +struct sched_group *sg =cpu_rq (cpu)->sd->groups; + + power_orig += sg->sgp->power_orig; + power += sg->sgp->power; + } }else {/* * !SD_OVERLAP domains can assume that child groups @@ -4480,12 +4484,14 @@ void update_group_power(struct sched_domain *sd, int cpu) group = child->groups;do { + power_orig += group->sgp->power_orig; power += group->sgp->power; group = group->next; }while (group != child->groups); } - sdg->sgp->power_orig = sdg->sgp->power = power; + sdg->sgp->power_orig = power_orig; + sdg->sgp->power = power; }/*
With Commit-id 863bffc80898 (sched/fair: Fix group power_orig computation) which is part of latest tip/master, numa machine may fail to boot both on powerpc and x86_64.
On powerpc
[ 0.710162] Unable to handle kernel paging request for data at address 0x00000010 [ 0.710170] Faulting instruction address: 0xc0000000000f3db4 [ 0.710177] Oops: Kernel access of bad area, sig: 11 [#1] [ 0.710182] SMP NR_CPUS=1024 NUMA pSeries [ 0.710190] Modules linked in: [ 0.710199] CPU: 53 PID: 1 Comm: swapper/53 Not tainted 3.12.0-tip_master+ #1 [ 0.710205] task: c000001713980000 ti: c000001713a00000 task.ti: c000001713a00000 [ 0.710211] NIP: c0000000000f3db4 LR: c0000000000f3de0 CTR: 0000000000000000 [ 0.710217] REGS: c000001713a036d0 TRAP: 0300 Not tainted (3.12.0-tip_master+) [ 0.710223] MSR: 8000000000009032 <SF,EE,ME,IR,DR,RI> CR: 48002044 XER: 20000003 [ 0.710243] SOFTE: 1 [ 0.710246] CFAR: c00000000000908c [ 0.710250] DAR: 0000000000000010, DSISR: 40000000 [ 0.710254]
GPR00: c000000000e85e00 c000001713a03950 c0000000015a7a58 0000000000000030
GPR04: 0000000000000030 0000000000000000 0000000000000400 000000001dcd6500
GPR08: 0000000000000000 0000000000000000 0000000000000000 c00000000160c958
GPR12: 0000000000000000 c000000007edb980 c0000007e2e374a8 c000000001607a58
GPR16: c000001ef38f8f58 0000000000000180 c000000001770258 0000000000000001
GPR20: 000000000000005f c000001ef38f8f40 0000000000000001 c0000000014b9acf
GPR24: 0000000000000000 c000001712884a40 c000000001607a58 c000000000e87a58
GPR28: c00000000160fc54 c000001712884a58 0000000000000000 0000000000000000
[ 0.710349] NIP [c0000000000f3db4] .update_group_power+0xd4/0x2e0 [ 0.710355] LR [c0000000000f3de0] .update_group_power+0x100/0x2e0 [ 0.710360] Call Trace: [ 0.710364] [c000001713a03950] [c0000000000f3d28] .update_group_power+0x48/0x2e0 (unreliable) [ 0.710375] [c000001713a03a00] [c0000000000ed73c] .build_sched_domains+0xadc/0xd90 [ 0.710385] [c000001713a03b70] [c000000000bf39b0] .sched_init_smp+0x528/0x66c [ 0.710394] [c000001713a03ce0] [c000000000bd46a8] .kernel_init_freeable+0x200/0x398 [ 0.710405] [c000001713a03db0] [c00000000000bc04] .kernel_init+0x24/0x140 [ 0.710413] [c000001713a03e30] [c00000000000a16c] .ret_from_kernel_thread+0x5c/0x70 [ 0.710419] Instruction dump: [ 0.710424] 3bb90018 3bc00000 3be00000 3860ffff 3f62ff8e 48000034 60000000 397a4f00 [ 0.710439] 381be3a8 7d2b482a 7d204a14 e9290950 <e9290010> e9290010 81690004 80090008 [ 0.710465] ---[ end trace b5091a0959b24fe3 ]--- [ 0.710469] [ 2.710530] Kernel panic - not syncing: Fatal exception [ 2.710943] Rebooting in 10 seconds..
On x86_64
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.12.0-tip_v312+ #5 Hardware name: IBM System x3750 M4 -[8722C1A]-/00D1432, BIOS -[KOE116JUS-1.10]- 09/25/2012 task: ffff8810373a6040 ti: ffff8810373a8000 task.ti: ffff8810373a8000 RIP: 0010:[<ffffffff8108aa53>] [<ffffffff8108aa53>] update_group_power+0xa3/0x130 RSP: 0000:ffff8810373a9db8 EFLAGS: 00010283 RAX: 0000000000000008 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000040 RBP: ffff8810373a9de8 R08: ffff88203632f818 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000 R13: 00000000001d3cc0 R14: ffff88203632f818 R15: ffff88203632f800 FS: 0000000000000000(0000) GS:ffff88103de00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000010 CR3: 0000000001a0b000 CR4: 00000000000407f0 Stack: ffff8810373a9dd8 ffff881035aa0c00 ffff88203632f800 ffff8820362fa800 0000000000000008 0000000000000008 ffff8810373a9e58 ffffffff8108627b ffff8810373a9e18 ffffffff8129fcf8 00000000373a76e0 000000000000003f Call Trace: [<ffffffff8108627b>] build_sched_domains+0x37b/0x3f0 [<ffffffff8129fcf8>] ? alloc_cpumask_var_node+0x58/0x80 [<ffffffff81d59783>] sched_init_smp+0x8f/0x13c [<ffffffff81d3db78>] kernel_init_freeable+0x27b/0x303 [<ffffffff81593c6e>] ? kernel_init+0xe/0xf0 [<ffffffff810a0ead>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81593c60>] ? rest_init+0xd0/0xd0 [<ffffffff81593c6e>] kernel_init+0xe/0xf0 [<ffffffff815a9eac>] ret_from_fork+0x7c/0xb0 [<ffffffff81593c60>] ? rest_init+0xd0/0xd0 Code: ad 00 4d 8d 77 18 45 31 e4 31 db b8 ff ff ff ff eb 2d 66 0f 1f 44 00 00 48 63 d0 48 8b 14 d5
c0 9d b4 81 49 8b 94 15 50 09 00 00 <48> 8b 52 10 48 8b 52 10 8b 4a 08 8b 52 04 49 01 cc 48 01 d3 8 3
RIP [<ffffffff8108aa53>] update_group_power+0xa3/0x130
RSP <ffff8810373a9db8>
CR2: 0000000000000010
---[ end trace cd8cb7fb261d7bea ]--- Kernel panic - not syncing: Fatal exception
This can be fixed by a simple check below.
Signature · 4 lines
Thanks and Regards Srikar Dronamraju -------->8---------------------------------------------
Quoted 1 line
From
bfc5dced04472c6c499aa3c6773ddef42d83fefcMon Sep 17 00:00:00 2001
From: Srikar Dronamraju <[email protected]>
Date: Tue, 12 Nov 2013 03:05:31 -0500
Subject: [PATCH] sched: Check sched_domain before computing group power.
After Commit-id 863bffc80898 (sched/fair: Fix group power_orig computation), we might end up computing group power before the sched_domain for a cpu is updated.
Check for rq->sd before updating group power.
--- kernel/sched/fair.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)
kernel/sched/fair.c +6 -1
@@ -5354,8 +5354,13 @@ void update_group_power(struct sched_domain *sd, int cpu)*/ for_each_cpu (cpu,sched_group_cpus (sdg)) { -struct sched_group *sg =cpu_rq (cpu)->sd->groups; +struct rq *rq =cpu_rq (cpu); +struct sched_group *sg; +if (!rq->sd) +continue ; + + sg = rq->sd->groups; power_orig += sg->sgp->power_orig; power += sg->sgp->power; }
Signature · 1 line
1.7.1
Peter Zijlstra
maintainerQuoted 11 lines · Srikar Dronamraju
From: Srikar Dronamraju <[email protected]>
Date: Tue, 12 Nov 2013 03:05:31 -0500
Subject: [PATCH] sched: Checksched_domainbefore computing group power.After Commit-id
863bffc80898(sched/fair: Fix grouppower_origcomputation), we might end up computing group power before thesched_domainfor a cpu is updated.Check for
rq->sdbefore updating group power.Signed-off-by Srikar Dronamraju <[email protected]>
Thanks! My head hurts a bit from going through the overlap init paths but this does indeed revert to the previous behaviour.
I'm not entirely sure if either are fully correct, but given we initialize the sgp->power to some 'reasonable' default we can rely on runtime updates to correct any funnies.
Quoted 26 lines
--- kernel/sched/fair.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)kernel/sched/fair.c +6
-1@@ -5354,8 +5354,13 @@ void update_group_power(struct sched_domain *sd, int cpu)*/ for_each_cpu (cpu,sched_group_cpus (sdg)) { -struct sched_group *sg =cpu_rq (cpu)->sd->groups; +struct rq *rq =cpu_rq (cpu); +struct sched_group *sg; +if (!rq->sd) +continue ; + + sg = rq->sd->groups; power_orig += sg->sgp->power_orig; power += sg->sgp->power; }Signature · 1 line
1.7.1
PATCH v2 1/1 sched: Check sched_domain before computing group power.
After Commit-id 863bffc80898 (sched/fair: Fix group power_orig computation), we might end up computing group power before the sched_domain for a cpu is updated.
Update with cpu_power, if rq->sd is not yet updated.
---
Changelog since v1: Fix divide by zero errors that can result because power/power_orig was set to 0.
kernel/sched/fair.c | 10 +++++++++- 1 files changed, 9 insertions(+), 1 deletions(-)
kernel/sched/fair.c +9 -1
@@ -5354,8 +5354,16 @@ void update_group_power(struct sched_domain *sd, int cpu)*/ for_each_cpu (cpu,sched_group_cpus (sdg)) { -struct sched_group *sg =cpu_rq (cpu)->sd->groups; +struct rq *rq =cpu_rq (cpu); +struct sched_group *sg; +if (!rq->sd) { + power_orig +=power_of (cpu); + power +=power_of (cpu); +continue ; + } + + sg = rq->sd->groups; power_orig += sg->sgp->power_orig; power += sg->sgp->power; }
Signature · 1 line
1.7.1
Peter Zijlstra
maintainerRe: [PATCH v2] sched: Check sched_domain before computing group power.
Quoted 10 lines · Srikar Dronamraju
After Commit-id
863bffc80898(sched/fair: Fix grouppower_origcomputation), we might end up computing group power before thesched_domainfor a cpu is updated.Update with
cpu_power, ifrq->sdis not yet updated.Signed-off-by Srikar Dronamraju <[email protected]>---
Changelog since v1: Fix divide by zero errors that can result because power/power_orig was set to 0.
Hurm.. can you provide the actual topology of the machine that triggers this? My brain hurts trying to thing through the weird cases of this code.
Re: [PATCH v2] sched: Check sched_domain before computing group power.
Quoted 5 lines
Hurm.. can you provide the actual topology of the machine that triggers this? My brain hurts trying to thing through the weird cases of this code.
Hope this helps. Please do let me know if you were looking for pdf output.
Machine (251GB)
NUMANode P#0 (63GB)
Socket P#0
L3 (20MB)
L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB)
L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB)
L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB)
Core P#0 Core P#1 Core P#2 Core P#3 Core P#4 Core P#5 Core P#6 Core P#7
PU P#0 PU P#1 PU P#2 PU P#3 PU P#4 PU P#5 PU P#6 PU P#7
PU P#32 PU P#33 PU P#34 PU P#35 PU P#36 PU P#37 PU P#38 PU P#39NUMANode P#1 (63GB)
Socket P#1
L3 (20MB)
L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB)
L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB)
L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB)
Core P#0 Core P#1 Core P#2 Core P#3 Core P#4 Core P#5 Core P#6 Core P#7
PU P#8 PU P#9 PU P#10 PU P#11 PU P#12 PU P#13 PU P#14 PU P#15
PU P#40 PU P#41 PU P#42 PU P#43 PU P#44 PU P#45 PU P#46 PU P#47NUMANode P#2 (63GB)
Socket P#2
L3 (20MB)
L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB)
L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB)
L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB)
Core P#0 Core P#1 Core P#2 Core P#3 Core P#4 Core P#5 Core P#6 Core P#7
PU P#16 PU P#17 PU P#18 PU P#19 PU P#20 PU P#21 PU P#22 PU P#23
PU P#48 PU P#49 PU P#50 PU P#51 PU P#52 PU P#53 PU P#54 PU P#55NUMANode P#3 (62GB)
Socket P#3
L3 (20MB)
L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB)
L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB)
L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB)
Core P#0 Core P#1 Core P#2 Core P#3 Core P#4 Core P#5 Core P#6 Core P#7
PU P#24 PU P#25 PU P#26 PU P#27 PU P#28 PU P#29 PU P#30 PU P#31
PU P#56 PU P#57 PU P#58 PU P#59 PU P#60 PU P#61 PU P#62 PU P#63
PCI 1000:005b
sda sdb sdc sdd
sde sdf sdg sdh
PCI 19a2:0710
eth0
PCI 19a2:0710
eth1
PCI 19a2:0710
eth2
PCI 19a2:0710
eth3
PCI 102b:0534
PCI 8086:1d02
PCI 1000:0073Host: kong.in.ibm.com
Indexes: physical
Date: Tuesday 12 November 2013 10:38:18 PM IST
Peter Zijlstra
maintainerRe: [PATCH v2] sched: Check sched_domain before computing group power.
Quoted 7 lines · Srikar Dronamraju
Quoted 5 lines
Hurm.. can you provide the actual topology of the machine that triggers this? My brain hurts trying to thing through the weird cases of this code.
Hope this helps. Please do let me know if you were looking for pdf output.
PDFs go into /dev/null..
the below misses the interesting bits; being the node distance table. Also a complete sched_debug domain print is useful.
Re: [PATCH v2] sched: Check sched_domain before computing group power.
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Srikar Dronamraju
Quoted 5 lines
Hurm.. can you provide the actual topology of the machine that triggers this? My brain hurts trying to thing through the weird cases of this code.
Quoted 7 lines
Quoted 1 line
Hope this helps. Please do let me know if you were looking for pdf output.
PDFs go into
/dev/null..the below misses the interesting bits; being the node distance table. Also a complete
sched_debugdomain print is useful.
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 32 33 34 35 36 37 38 39 node 0 size: 64191 MB
node 0 free: 63194 MB
node 1 cpus: 8 9 10 11 12 13 14 15 40 41 42 43 44 45 46 47 node 1 size: 64481 MB
node 1 free: 63515 MB
node 2 cpus: 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55 node 2 size: 64481 MB
node 2 free: 63536 MB
node 3 cpus: 24 25 26 27 28 29 30 31 56 57 58 59 60 61 62 63 node 3 size: 63968 MB
node 3 free: 62981 MB
node distances:
node 0 1 2 3
0: 10 11 11 12 1: 11 10 12 11 2: 11 12 10 11 3: 12 11 11 10
x86: Booting SMP configuration:
.... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 .... node #1, CPUs: #8 #9 #10 #11 #12 #13 #14 #15 .... node #2, CPUs: #16 #17 #18 #19 #20 #21 #22 #23 .... node #3, CPUs: #24 #25 #26 #27 #28 #29 #30 #31 .... node #0, CPUs: #32 #33 #34 #35 #36 #37 #38 #39 .... node #1, CPUs: #40 #41 #42 #43 #44 #45 #46 #47 .... node #2, CPUs: #48 #49 #50 #51 #52 #53 #54 #55 .... node #3, CPUs: #56 #57 #58 #59 #60 #61 #62 #63
x86: Booted up 4 nodes, 64 CPUs
smpboot: Total of 64 processors activated (308393.92 BogoMIPS)
CPU0 attaching sched-domain:
domain 0: span 0,32 level SIBLING
groups: 0 (cpu_power = 588) 32 (cpu_power = 588)
domain 1: span 0-7,32-39 level MC
groups: 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU1 attaching sched-domain:
domain 0: span 1,33 level SIBLING
groups: 1 (cpu_power = 589) 33 (cpu_power = 589)
domain 1: span 0-7,32-39 level MC
groups: 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU2 attaching sched-domain:
domain 0: span 2,34 level SIBLING
groups: 2 (cpu_power = 589) 34 (cpu_power = 589)
domain 1: span 0-7,32-39 level MC
groups: 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU3 attaching sched-domain:
domain 0: span 3,35 level SIBLING
groups: 3 (cpu_power = 589) 35 (cpu_power = 589)
domain 1: span 0-7,32-39 level MC
groups: 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU4 attaching sched-domain:
domain 0: span 4,36 level SIBLING
groups: 4 (cpu_power = 589) 36 (cpu_power = 589)
domain 1: span 0-7,32-39 level MC
groups: 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU5 attaching sched-domain:
domain 0: span 5,37 level SIBLING
groups: 5 (cpu_power = 589) 37 (cpu_power = 589)
domain 1: span 0-7,32-39 level MC
groups: 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU6 attaching sched-domain:
domain 0: span 6,38 level SIBLING
groups: 6 (cpu_power = 589) 38 (cpu_power = 589)
domain 1: span 0-7,32-39 level MC
groups: 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU7 attaching sched-domain:
domain 0: span 7,39 level SIBLING
groups: 7 (cpu_power = 588) 39 (cpu_power = 588)
domain 1: span 0-7,32-39 level MC
groups: 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU8 attaching sched-domain:
domain 0: span 8,40 level SIBLING
groups: 8 (cpu_power = 588) 40 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU9 attaching sched-domain:
domain 0: span 9,41 level SIBLING
groups: 9 (cpu_power = 588) 41 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU10 attaching sched-domain:
domain 0: span 10,42 level SIBLING
groups: 10 (cpu_power = 588) 42 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU11 attaching sched-domain:
domain 0: span 11,43 level SIBLING
groups: 11 (cpu_power = 588) 43 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU12 attaching sched-domain:
domain 0: span 12,44 level SIBLING
groups: 12 (cpu_power = 588) 44 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU13 attaching sched-domain:
domain 0: span 13,45 level SIBLING
groups: 13 (cpu_power = 588) 45 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU14 attaching sched-domain:
domain 0: span 14,46 level SIBLING
groups: 14 (cpu_power = 588) 46 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU15 attaching sched-domain:
domain 0: span 15,47 level SIBLING
groups: 15 (cpu_power = 588) 47 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU16 attaching sched-domain:
domain 0: span 16,48 level SIBLING
groups: 16 (cpu_power = 588) 48 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU17 attaching sched-domain:
domain 0: span 17,49 level SIBLING
groups: 17 (cpu_power = 588) 49 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU18 attaching sched-domain:
domain 0: span 18,50 level SIBLING
groups: 18 (cpu_power = 588) 50 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU19 attaching sched-domain:
domain 0: span 19,51 level SIBLING
groups: 19 (cpu_power = 588) 51 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU20 attaching sched-domain:
domain 0: span 20,52 level SIBLING
groups: 20 (cpu_power = 588) 52 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU21 attaching sched-domain:
domain 0: span 21,53 level SIBLING
groups: 21 (cpu_power = 588) 53 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU22 attaching sched-domain:
domain 0: span 22,54 level SIBLING
groups: 22 (cpu_power = 588) 54 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU23 attaching sched-domain:
domain 0: span 23,55 level SIBLING
groups: 23 (cpu_power = 588) 55 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU24 attaching sched-domain:
domain 0: span 24,56 level SIBLING
groups: 24 (cpu_power = 588) 56 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU25 attaching sched-domain:
domain 0: span 25,57 level SIBLING
groups: 25 (cpu_power = 588) 57 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU26 attaching sched-domain:
domain 0: span 26,58 level SIBLING
groups: 26 (cpu_power = 588) 58 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU27 attaching sched-domain:
domain 0: span 27,59 level SIBLING
groups: 27 (cpu_power = 588) 59 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU28 attaching sched-domain:
domain 0: span 28,60 level SIBLING
groups: 28 (cpu_power = 588) 60 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU29 attaching sched-domain:
domain 0: span 29,61 level SIBLING
groups: 29 (cpu_power = 588) 61 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU30 attaching sched-domain:
domain 0: span 30,62 level SIBLING
groups: 30 (cpu_power = 588) 62 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU31 attaching sched-domain:
domain 0: span 31,63 level SIBLING
groups: 31 (cpu_power = 588) 63 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU32 attaching sched-domain:
domain 0: span 0,32 level SIBLING
groups: 32 (cpu_power = 588) 0 (cpu_power = 588)
domain 1: span 0-7,32-39 level MC
groups: 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU33 attaching sched-domain:
domain 0: span 1,33 level SIBLING
groups: 33 (cpu_power = 589) 1 (cpu_power = 589)
domain 1: span 0-7,32-39 level MC
groups: 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU34 attaching sched-domain:
domain 0: span 2,34 level SIBLING
groups: 34 (cpu_power = 589) 2 (cpu_power = 589)
domain 1: span 0-7,32-39 level MC
groups: 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU35 attaching sched-domain:
domain 0: span 3,35 level SIBLING
groups: 35 (cpu_power = 589) 3 (cpu_power = 589)
domain 1: span 0-7,32-39 level MC
groups: 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU36 attaching sched-domain:
domain 0: span 4,36 level SIBLING
groups: 36 (cpu_power = 589) 4 (cpu_power = 589)
domain 1: span 0-7,32-39 level MC
groups: 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU37 attaching sched-domain:
domain 0: span 5,37 level SIBLING
groups: 37 (cpu_power = 589) 5 (cpu_power = 589)
domain 1: span 0-7,32-39 level MC
groups: 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU38 attaching sched-domain:
domain 0: span 6,38 level SIBLING
groups: 38 (cpu_power = 589) 6 (cpu_power = 589)
domain 1: span 0-7,32-39 level MC
groups: 6,38 (cpu_power = 1178) 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU39 attaching sched-domain:
domain 0: span 7,39 level SIBLING
groups: 39 (cpu_power = 588) 7 (cpu_power = 588)
domain 1: span 0-7,32-39 level MC
groups: 7,39 (cpu_power = 1176) 0,32 (cpu_power = 1176) 1,33 (cpu_power = 1178) 2,34 (cpu_power = 1178) 3,35 (cpu_power = 1178) 4,36 (cpu_power = 1178) 5,37 (cpu_power = 1178) 6,38 (cpu_power = 1178) domain 2: span 0-23,32-55 level NUMA
groups: 0-7,32-39 (cpu_power = 9420) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU40 attaching sched-domain:
domain 0: span 8,40 level SIBLING
groups: 40 (cpu_power = 588) 8 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU41 attaching sched-domain:
domain 0: span 9,41 level SIBLING
groups: 41 (cpu_power = 588) 9 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU42 attaching sched-domain:
domain 0: span 10,42 level SIBLING
groups: 42 (cpu_power = 588) 10 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU43 attaching sched-domain:
domain 0: span 11,43 level SIBLING
groups: 43 (cpu_power = 588) 11 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU44 attaching sched-domain:
domain 0: span 12,44 level SIBLING
groups: 44 (cpu_power = 588) 12 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU45 attaching sched-domain:
domain 0: span 13,45 level SIBLING
groups: 45 (cpu_power = 588) 13 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU46 attaching sched-domain:
domain 0: span 14,46 level SIBLING
groups: 46 (cpu_power = 588) 14 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 14,46 (cpu_power = 1176) 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU47 attaching sched-domain:
domain 0: span 15,47 level SIBLING
groups: 47 (cpu_power = 588) 15 (cpu_power = 588)
domain 1: span 8-15,40-47 level MC
groups: 15,47 (cpu_power = 1176) 8,40 (cpu_power = 1176) 9,41 (cpu_power = 1176) 10,42 (cpu_power = 1176) 11,43 (cpu_power = 1176) 12,44 (cpu_power = 1176) 13,45 (cpu_power = 1176) 14,46 (cpu_power = 1176) domain 2: span 0-15,24-47,56-63 level NUMA
groups: 8-15,40-47 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU48 attaching sched-domain:
domain 0: span 16,48 level SIBLING
groups: 48 (cpu_power = 588) 16 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU49 attaching sched-domain:
domain 0: span 17,49 level SIBLING
groups: 49 (cpu_power = 588) 17 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU50 attaching sched-domain:
domain 0: span 18,50 level SIBLING
groups: 50 (cpu_power = 588) 18 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU51 attaching sched-domain:
domain 0: span 19,51 level SIBLING
groups: 51 (cpu_power = 588) 19 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU52 attaching sched-domain:
domain 0: span 20,52 level SIBLING
groups: 52 (cpu_power = 588) 20 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU53 attaching sched-domain:
domain 0: span 21,53 level SIBLING
groups: 53 (cpu_power = 588) 21 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU54 attaching sched-domain:
domain 0: span 22,54 level SIBLING
groups: 54 (cpu_power = 588) 22 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 22,54 (cpu_power = 1176) 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU55 attaching sched-domain:
domain 0: span 23,55 level SIBLING
groups: 55 (cpu_power = 588) 23 (cpu_power = 588)
domain 1: span 16-23,48-55 level MC
groups: 23,55 (cpu_power = 1176) 16,48 (cpu_power = 1176) 17,49 (cpu_power = 1176) 18,50 (cpu_power = 1176) 19,51 (cpu_power = 1176) 20,52 (cpu_power = 1176) 21,53 (cpu_power = 1176) 22,54 (cpu_power = 1176) domain 2: span 0-7,16-39,48-63 level NUMA
groups: 16-23,48-55 (cpu_power = 9408) 24-31,56-63 (cpu_power = 9408) 0-7,32-39 (cpu_power = 9420)
domain 3: span 0-63 level NUMA
groups: 0-23,32-55 (cpu_power = 28236) 8-31,40-63 (cpu_power = 28224)CPU56 attaching sched-domain:
domain 0: span 24,56 level SIBLING
groups: 56 (cpu_power = 588) 24 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU57 attaching sched-domain:
domain 0: span 25,57 level SIBLING
groups: 57 (cpu_power = 588) 25 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU58 attaching sched-domain:
domain 0: span 26,58 level SIBLING
groups: 58 (cpu_power = 588) 26 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU59 attaching sched-domain:
domain 0: span 27,59 level SIBLING
groups: 59 (cpu_power = 588) 27 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU60 attaching sched-domain:
domain 0: span 28,60 level SIBLING
groups: 60 (cpu_power = 588) 28 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU61 attaching sched-domain:
domain 0: span 29,61 level SIBLING
groups: 61 (cpu_power = 588) 29 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU62 attaching sched-domain:
domain 0: span 30,62 level SIBLING
groups: 62 (cpu_power = 588) 30 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 30,62 (cpu_power = 1176) 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)CPU63 attaching sched-domain:
domain 0: span 31,63 level SIBLING
groups: 63 (cpu_power = 588) 31 (cpu_power = 588)
domain 1: span 24-31,56-63 level MC
groups: 31,63 (cpu_power = 1176) 24,56 (cpu_power = 1176) 25,57 (cpu_power = 1176) 26,58 (cpu_power = 1176) 27,59 (cpu_power = 1176) 28,60 (cpu_power = 1176) 29,61 (cpu_power = 1176) 30,62 (cpu_power = 1176) domain 2: span 8-31,40-63 level NUMA
groups: 24-31,56-63 (cpu_power = 9408) 8-15,40-47 (cpu_power = 9408) 16-23,48-55 (cpu_power = 9408)
domain 3: span 0-63 level NUMA
groups: 8-31,40-63 (cpu_power = 28224) 0-23,32-55 (cpu_power = 28236)Signature · 2 lines
Thanks and Regards Srikar Dronamraju
A message that is not in the archive (sent off-list or not yet received): CAM4v1pNMn=5oZDiX3fUp9uPkZTPJgk=vEKEjevzvpwn=PjTzXg@mail.gmail.com
Re: [PATCH v2] sched: Check sched_domain before computing group power.
Quoted 11 lines · Preeti Murthy
Hi Srikar,
update_group_power()is called only during load balancing duringupdate_sg_lb_stats().
Load balancing begins at the base domain of the CPU,rq(cpu)->sd. This is checked for
NULL. So how canupdate_group_power()be called in a scenario where the base domain
of the CPU is not initialized? I say 'initialized' since you check for NULL on rq(cpu)->sd.
update_group_power() also gets called from init_sched_groups_power(). And if you see the oops message, we know that the oops happens from that path. In build_sched_domains(), we do cpu_attach_domain() what updates rq->sd after the call to init_sched_groups_power(). So by the time init_sched_groups_power() is called rq->sd is not yet initialized.
We only hit oops case, when the sd->flags has SD_OVERLAP set.
Quoted 2 lines
In the changelog, you say 'updated'. Are you saying that it has a stale value?
I said, "before the sched_domain for a cpu is updated", so its not yet updated or has stale value. As I said earlier in this mail, the initialization happens after we do a update_group_power().
Quoted 2 lines
Please do elaborate on how you observed this.
Does this clarify?
Signature · 2 lines
Thanks and Regards Srikar Dronamraju
Re: [PATCH v2] sched: Check sched_domain before computing group power.
Hi,
Quoted 27 lines · Srikar Dronamraju
Quoted 11 lines · Preeti Murthy
Hi Srikar,
update_group_power()is called only during load balancing duringupdate_sg_lb_stats().
Load balancing begins at the base domain of the CPU,rq(cpu)->sd. This is checked for
NULL. So how canupdate_group_power()be called in a scenario where the base domain
of the CPU is not initialized? I say 'initialized' since you check for NULL on rq(cpu)->sd.
update_group_power()also gets called frominit_sched_groups_power(). And if you see the oops message, we know that the oops happens from that path. Inbuild_sched_domains(), we docpu_attach_domain()what updatesrq->sdafter the call toinit_sched_groups_power(). So by the timeinit_sched_groups_power()is calledrq->sdis not yet initialized.We only hit oops case, when the
sd->flagshasSD_OVERLAPset.Quoted 2 lines
In the changelog, you say 'updated'. Are you saying that it has a stale value?
Quoted 8 lines
I said, "before the
sched_domainfor a cpu is updated", so its not yet updated or has stale value. As I said earlier in this mail, the initialization happens after we do aupdate_group_power().Quoted 2 lines
Please do elaborate on how you observed this.
Does this clarify?
Yes it clarifies, thank you.
However I was thinking that a better fix would be to reorder the way we call update_group_power() and cpu_attach_domain(). Why do we need to do update_group_power() of the groups of the sched domains that would probably degenerate in cpu_attach_domain()? So it seemed best to move update_group_power() to after cpu_attach_domain() so that it saves unnecessary iterations over sched domains which could degenerate, and it fixes the issue that you have brought out as well. See below for the patch:
-------------------------------------------------------------------------------
sched: Update power of sched groups after sched domains have been attached to CPUs
From: Preeti U Murthy <[email protected]>
Avoid iterating unnecessarily over the sched domains which could potentially degenerate, while updating sched groups' power. This can be done by moving the call to init_sched_groups_power() to after cpu_attach_domain(), when the possibility of degenerating sched domains is examined and appropriately sched domains are degenerated.
But claim_allocations() which does a NULL on the struct sd_data members for each sched domain should iterate over all the initally built sched domains. So move claim_allocations() to a loop where we build sched groups for each domain. We would not require to reference sd_data after sched domains and sched groups have been built.
Another use of this re-ordering is with reference to the commit "sched/fair: Fix group power_orig computation". After this commit, we end up dereferencing cpu_rq(cpu)->sd in update_group_power(). This would lead to a NULL pointer since this parameter is updated after call to update_group_power() in build_sched_domains() during initialization of sched domains per CPU. The below change prevents this from occuring.
--- kernel/sched/core.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
kernel/sched/core.c +10 -10
@@ -6211,17 +6211,7 @@ static int build_sched_domains(const struct cpumask *cpu_map,if (build_sched_groups (sd, i)) goto error; } - } - } - -/* Calculate CPU power for physical packages and nodes */ -for (i = nr_cpumask_bits-1 ; i >=0 ; i--) { -if (!cpumask_test_cpu (i, cpu_map)) -continue ; - -for (sd = *per_cpu_ptr (d.sd, i); sd; sd = sd->parent) {claim_allocations (i, sd); -init_sched_groups_power (i, sd); } }
@@ -6233,6 +6223,16 @@ static int build_sched_domains(const struct cpumask *cpu_map, }rcu_read_unlock (); +/* Calculate CPU power for physical packages and nodes */ +for (i = nr_cpumask_bits-1 ; i >=0 ; i--) { +if (!cpumask_test_cpu (i, cpu_map)) +continue ; + +for (sd = *per_cpu_ptr (d.sd, i); sd; sd = sd->parent) +init_sched_groups_power (i, sd); + } + + ret =0 ; error:__free_domain_allocs (&d, alloc_state, cpu_map);
Quoted 1 line
Regards
Preeti U. Murthy
Peter Zijlstra
maintainerRe: [PATCH v2] sched: Check sched_domain before computing group power.
Quoted 7 lines · Preeti U Murthy
However I was thinking that a better fix would be to reorder the way we call
update_group_power()andcpu_attach_domain(). Why do we need to doupdate_group_power()of the groups of the sched domains that would probably degenerate incpu_attach_domain()? So it seemed best to moveupdate_group_power()to aftercpu_attach_domain()so that it saves unnecessary iterations over sched domains which could degenerate, and it fixes the issue that you have brought out as well. See below for the patch:
So how is publishing the domain tree before we've set these values at all going to help avoid the divide-by-zero problem?
Also its just terribly bad form to publish something before you're done with initialization.
Re: [PATCH v2] sched: Check sched_domain before computing group power.
Hi Peter,
Quoted 11 lines · Peter Zijlstra
Quoted 7 lines · Preeti U Murthy
However I was thinking that a better fix would be to reorder the way we call
update_group_power()andcpu_attach_domain(). Why do we need to doupdate_group_power()of the groups of the sched domains that would probably degenerate incpu_attach_domain()? So it seemed best to moveupdate_group_power()to aftercpu_attach_domain()so that it saves unnecessary iterations over sched domains which could degenerate, and it fixes the issue that you have brought out as well. See below for the patch:So how is publishing the domain tree before we've set these values at all going to help avoid the divide-by-zero problem?
We are still doing initialization of cpu power and power_orig during building of sched domains right? Except that it is being done after CPUs have base domains attached to them.
But if you are talking about the check in sched_debug_one() on if power_orig has been initialized, then yes, this patch fails. I am sorry I overlooked the sched_debug() checks in cpu_attach_domain().
Quoted 3 lines
Also its just terribly bad form to publish something before you're done with initialization.
You are right. cpu_rq(cpu)->sd is going to be used by anyone intending to iterate through the sched domains. By the time we publish this, every parameter related to sched domains and groups need to be initialized. The fact that sched_domain_debug() is going to be called by cpu_attach_domain() to do one final sanity check on all the parameters of the sched domains further emphasizes that we cannot have anything un-initialised at this stage. So clearly my patch is incorrect. Please add my Reviewed-by to Srikar's patch.
Thanks.
Regards
Preeti U. Murthy
Quoted 1 line
Peter Zijlstra
maintainerRe: [PATCH v2] sched: Check sched_domain before computing group power.
In reply to Srikar DronamrajuQuoted 10 lines · Srikar Dronamraju
After Commit-id
863bffc80898(sched/fair: Fix grouppower_origcomputation), we might end up computing group power before thesched_domainfor a cpu is updated.Update with
cpu_power, ifrq->sdis not yet updated.Signed-off-by Srikar Dronamraju <[email protected]>---
Changelog since v1: Fix divide by zero errors that can result because power/power_orig was set to 0.
Duh yes!, init_sched_groups_power() is called before we attach the actual domains, so the above will _always_ fail for the build_sched_domain() case.
I was a bit puzzled how we could ever have 0 since surely at least the current cpu should have some !0 contribution, but no!
---
Subject: sched: Check sched_domain before computing group power
From: Srikar Dronamraju <[email protected]>
Date: Tue, 12 Nov 2013 22:11:26 +0530
After Commit-id 863bffc80898 (sched/fair: Fix group power_orig computation), we can dereference rq->sd before its set.
Fix this by falling back to power_of() in this case and add a comment explaining things.
--- kernel/sched/fair.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-)
kernel/sched/fair.c +24 -3
@@ -5354,10 +5354,31 @@ void update_group_power(struct sched_dom*/ for_each_cpu (cpu,sched_group_cpus (sdg)) { -struct sched_group *sg =cpu_rq (cpu)->sd->groups; +struct sched_group_power *sgp; +struct rq *rq =cpu_rq (cpu); - power_orig += sg->sgp->power_orig; - power += sg->sgp->power; +/* +* build_sched_domains() -> init_sched_groups_power() +* gets here before we've attached the domains to the +* runqueues. +* +* Use power_of(), which is set irrespective of domains +* in update_cpu_power(). +* +* This avoids power/power_orig from being 0 and +* causing divide-by-zero issues on boot. +* +* Runtime updates will correct power_orig. +*/ +if (!rq->sd) { + power_orig +=power_of (cpu); + power +=power_of (cpu); +continue ; + } + + sgp = rq->sd->groups->sgp; + power_orig += sgp->power_orig; + power += sgp->power; } }else {/*
Re: [PATCH v2] sched: Check sched_domain before computing group power.
Quoted 4 lines
kernel/sched/fair.c +4
-0+/* +* build_sched_domains() -> init_sched_groups_power() +* gets here before we've attached the domains to the +* runqueues.
kernel/sched/fair.c void update_group_power(struct sched_dom · line 5373
Quoted 10 lines
kernel/sched/fair.c +10
-0+* +* Use power_of(), which is set irrespective of domains +* in update_cpu_power(). +* +* This avoids power/power_orig from being 0 and +* causing divide-by-zero issues on boot. +* +* Runtime updates will correct power_orig. +*/ +if (!rq->sd) {
Because this condition is only true during boot up, I am now thinking if we should do mark this as unlikely i.e if (unlikely(!rq->sd)) {
Peter,
Please do let me know if you agree and if you want to spin of a new version?
Quoted 12 lines
kernel/sched/fair.c +8
-0+ power_orig +=power_of (cpu); + power +=power_of (cpu); +continue ; + } + + sgp = rq->sd->groups->sgp; + power_orig += sgp->power_orig; + power += sgp->power; } }else {/*
Signature · 2 lines
Thanks and Regards Srikar Dronamraju
Peter Zijlstra
maintainerRe: [PATCH v2] sched: Check sched_domain before computing group power.
Quoted 9 lines · Srikar Dronamraju
Quoted 9 lines
kernel/sched/fair.c +9
-0+/* +* build_sched_domains() -> init_sched_groups_power() +* gets here before we've attached the domains to the +* runqueues. +* +* Use power_of(), which is set irrespective of domains +* in update_cpu_power(). +* +* This avoids power/power_orig from being 0 and
Quoted 8 lines
kernel/sched/fair.c void update_group_power(struct sched_dom · line 5373
Quoted 5 lines
kernel/sched/fair.c +5
-0+* causing divide-by-zero issues on boot. +* +* Runtime updates will correct power_orig. +*/ +if (!rq->sd) {Because this condition is only true during boot up, I am now thinking if we should do mark this as unlikely i.e if (
unlikely(!rq->sd)) {
Makes sense, edited the patch.
tip-bot for Srikar Dronamraju
merge bot[tip:sched/urgent] sched: Check sched_domain before computing group power
In reply to Peter Zijlstrasched: Check sched_domain before computing group power
Commit-ID: 9abf24d465180f5f2eb26a43545348262f16b771 Gitweb: http://git.kernel.org/tip/9abf24d465180f5f2eb26a43545348262f16b771 Author: Srikar Dronamraju <[email protected]> AuthorDate: Tue, 12 Nov 2013 22:11:26 +0530 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 19 Nov 2013 17:01:15 +0100
sched: Check sched_domain before computing group power
After commit 863bffc80898 ("sched/fair: Fix group power_orig computation"), we can dereference rq->sd before it is set.
Fix this by falling back to power_of() in this case and add a comment explaining things.
--- kernel/sched/fair.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-)
kernel/sched/fair.c +24 -3
@@ -5379,10 +5379,31 @@ void update_group_power(struct sched_domain *sd, int cpu)*/ for_each_cpu (cpu,sched_group_cpus (sdg)) { -struct sched_group *sg =cpu_rq (cpu)->sd->groups; +struct sched_group_power *sgp; +struct rq *rq =cpu_rq (cpu); - power_orig += sg->sgp->power_orig; - power += sg->sgp->power; +/* +* build_sched_domains() -> init_sched_groups_power() +* gets here before we've attached the domains to the +* runqueues. +* +* Use power_of(), which is set irrespective of domains +* in update_cpu_power(). +* +* This avoids power/power_orig from being 0 and +* causing divide-by-zero issues on boot. +* +* Runtime updates will correct power_orig. +*/ +if (unlikely (!rq->sd)) { + power_orig +=power_of (cpu); + power +=power_of (cpu); +continue ; + } + + sgp = rq->sd->groups->sgp; + power_orig += sgp->power_orig; + power += sgp->power; } }else {/*
Re: [tip:sched/urgent] sched: Check sched_domain before computing group power
Quoted 54 lines · tip-bot for Srikar Dronamraju
Commit-ID: 9abf24d465180f5f2eb26a43545348262f16b771 Gitweb: http://git.kernel.org/tip/9abf24d465180f5f2eb26a43545348262f16b771 Author: Srikar Dronamraju <[email protected]> AuthorDate: Tue, 12 Nov 2013 22:11:26 +0530 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 19 Nov 2013 17:01:15 +0100sched: Check
sched_domainbefore computing group powerAfter commit
863bffc80898("sched/fair: Fix grouppower_origcomputation"), we can dereferencerq->sdbefore it is set.Fix this by falling back to
power_of()in this case and add a comment explaining things.Signed-off-by Srikar Dronamraju <[email protected]>[ Added comment and tweaked patch. ]Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Ingo Molnar <[email protected]>--- kernel/sched/fair.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-)kernel/sched/fair.c +17
-3@@ -5379,10 +5379,31 @@ void update_group_power(struct sched_domain *sd, int cpu)*/ for_each_cpu (cpu,sched_group_cpus (sdg)) { -struct sched_group *sg =cpu_rq (cpu)->sd->groups; +struct sched_group_power *sgp; +struct rq *rq =cpu_rq (cpu); - power_orig += sg->sgp->power_orig; - power += sg->sgp->power; +/* +* build_sched_domains() -> init_sched_groups_power() +* gets here before we've attached the domains to the +* runqueues. +* +* Use power_of(), which is set irrespective of domains +* in update_cpu_power(). +* +* This avoids power/power_orig from being 0 and +* causing divide-by-zero issues on boot. +* +* Runtime updates will correct power_orig. +*/ +if (unlikely (!rq->sd)) { + power_orig +=power_of (cpu);
kernel/sched/fair.c void update_group_power(struct sched_domain *sd, int cpu) · line 5409
Quoted 10 lines
kernel/sched/fair.c +7
-0+ power +=power_of (cpu); +continue ; + } + + sgp = rq->sd->groups->sgp; + power_orig += sgp->power_orig; + power += sgp->power; } }else {/*
This one seems fix NULL reference in compute_group_power.
but get following on current Linus tree plus tip/sched/urgent.
divide error: 0000 [#1] SMP
[ 28.190477] Modules linked in: [ 28.192012] CPU: 11 PID: 484 Comm: kworker/u324:0 Not tainted 3.12.0-yh-10487-g4b94e59-dirty #2044 [ 28.210488] Hardware name: Oracle Corporation Sun Fire [ 28.229877] task: ffff88ff25205140 ti: ffff88ff2520a000 task.ti: ffff88ff2520a000 [ 28.236139] RIP: 0010:[<ffffffff810d9ff4>] [<ffffffff810d9ff4>] find_busiest_group+0x2b4/0x8a0 [ 28.252075] RSP: 0000:ffff88ff2520b9a8 EFLAGS: 00010046 [ 28.269591] RAX: 0000000000013fff RBX: 00000000ffffffff RCX: 00000000000000a0 [ 28.272977] RDX: 0000000000000000 RSI: 0000000000014000 RDI: 0000000000000050 [ 28.291968] RBP: ffff88ff2520bb08 R08: 00000000000003b6 R09: 0000000000000000 [ 28.309327] R10: 0000000000000000 R11: 0000000000000002 R12: ffff88ff2520ba90 [ 28.314222] R13: ffff887f2491c000 R14: 0000000000014000 R15: ffff88ff2520bba0 [ 28.331408] FS: 0000000000000000(0000) GS:ffff887f7d800000(0000) knlGS:0000000000000000 [ 28.349333] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 28.351524] CR2: 0000000000000168 CR3: 0000000002c14000 CR4: 00000000000007e0 [ 28.370245] Stack: [ 28.371466] 000000002520b9b8 000000000000000b 0000000000000048 0000000000000000 [ 28.389951] ffff887f2491c018 ffff88ff2520ba20 000000000000003c 00000000000002b6 [ 28.395085] 00000000000002b6 00000000000002b6 0000000000002df0 0000000100000001 [ 28.412079] Call Trace: [ 28.413617] [<ffffffff810da798>] load_balance+0x1b8/0x8c0 [ 28.429692] [<ffffffff810ec67b>] ? __lock_acquire+0xadb/0xce0 [ 28.433037] [<ffffffff810db3c1>] idle_balance+0x101/0x1c0 [ 28.450328] [<ffffffff810db304>] ? idle_balance+0x44/0x1c0 [ 28.453420] [<ffffffff8214a13b>] __schedule+0x2cb/0xa10 [ 28.469847] [<ffffffff810e66d8>] ? trace_hardirqs_off_caller+0x28/0x160 [ 28.473782] [<ffffffff810e681d>] ? trace_hardirqs_off+0xd/0x10 [ 28.490654] [<ffffffff810d1c14>] ? local_clock+0x34/0x60 [ 28.493723] [<ffffffff810b837b>] ? worker_thread+0x2db/0x370 [ 28.510363] [<ffffffff8214f450>] ? _raw_spin_unlock_irq+0x30/0x40 [ 28.514002] [<ffffffff8214a935>] schedule+0x65/0x70 [ 28.530380] [<ffffffff810b8380>] worker_thread+0x2e0/0x370 [ 28.533450] [<ffffffff810ea19d>] ? trace_hardirqs_on+0xd/0x10 [ 28.550976] [<ffffffff810b80a0>] ? manage_workers.isra.17+0x330/0x330 [ 28.554356] [<ffffffff810bf598>] kthread+0x108/0x110 [ 28.571857] [<ffffffff810bf490>] ? __init_kthread_worker+0x70/0x70 [ 28.588961] [<ffffffff82157cec>] ret_from_fork+0x7c/0xb0 [ 28.592017] [<ffffffff810bf490>] ? __init_kthread_worker+0x70/0x70 [ 28.609904] Code: 89 85 b8 fe ff ff 49 8b 45 10 41 8b 7d 0c 44 8b 50 08 44 8b 70 04 89 f8 48 c1 e0 0a 45 89 d1 49 8d 44 01 ff 48 89 c2 48 c1 fa 3f <49> f7 f9 31 d2 49 89 c1 89 f8 44 89 f7 41 f7 f1 48 81 c7 00 02 [ 28.641210] RIP [<ffffffff810d9ff4>] find_busiest_group+0x2b4/0x8a0 [ 28.650476] RSP <ffff88ff2520b9a8> [ 28.651754] divide error: 0000 [#2] [ 28.651762] ---[ end trace
bcaaa28065586d41 ]---
Peter Zijlstra
maintainerRe: [tip:sched/urgent] sched: Check sched_domain before computing group power
Quoted 43 lines · Yinghai Lu
Quoted 27 lines
kernel/sched/fair.c +17
-3@@ -5379,10 +5379,31 @@ void update_group_power(struct sched_domain *sd, int cpu)*/ for_each_cpu (cpu,sched_group_cpus (sdg)) { -struct sched_group *sg =cpu_rq (cpu)->sd->groups; +struct sched_group_power *sgp; +struct rq *rq =cpu_rq (cpu); - power_orig += sg->sgp->power_orig; - power += sg->sgp->power; +/* +* build_sched_domains() -> init_sched_groups_power() +* gets here before we've attached the domains to the +* runqueues. +* +* Use power_of(), which is set irrespective of domains +* in update_cpu_power(). +* +* This avoids power/power_orig from being 0 and +* causing divide-by-zero issues on boot. +* +* Runtime updates will correct power_orig. +*/ +if (unlikely (!rq->sd)) { + power_orig +=power_of (cpu);kernel/sched/fair.c void update_group_power(struct sched_domain *sd, int cpu) · line 5409
Quoted 10 lines
kernel/sched/fair.c +7
-0+ power +=power_of (cpu); +continue ; + } + + sgp = rq->sd->groups->sgp; + power_orig += sgp->power_orig; + power += sgp->power; } }else {/* This one seems fix NULL reference in
compute_group_power.but get following on current Linus tree plus tip/sched/urgent.
divide error: 0000 [#1] SMP
Quoted 8 lines
[ 28.190477] Modules linked in: [ 28.192012] CPU: 11 PID: 484 Comm: kworker/u324:0 Not tainted 3.12.0-yh-10487-g4b94e59-dirty #2044 [ 28.210488] Hardware name: Oracle Corporation Sun Fire [ 28.229877] task: ffff88ff25205140 ti: ffff88ff2520a000 task.ti: ffff88ff2520a000 [ 28.236139] RIP: 0010:[<ffffffff810d9ff4>] [<ffffffff810d9ff4>]
find_busiest_group+0x2b4/0x8a0
Hurmph.. what kind of hardware is that? and is there anything funny you do to make it do this?
Re: [tip:sched/urgent] sched: Check sched_domain before computing group power
Quoted 15 lines · Peter Zijlstra
Quoted 6 lines
This one seems fix NULL reference in
compute_group_power.but get following on current Linus tree plus tip/sched/urgent.
divide error: 0000 [#1] SMP
Quoted 8 lines
[ 28.190477] Modules linked in: [ 28.192012] CPU: 11 PID: 484 Comm: kworker/u324:0 Not tainted 3.12.0-yh-10487-g4b94e59-dirty #2044 [ 28.210488] Hardware name: Oracle Corporation Sun Fire [ 28.229877] task: ffff88ff25205140 ti: ffff88ff2520a000 task.ti: ffff88ff2520a000 [ 28.236139] RIP: 0010:[<ffffffff810d9ff4>] [<ffffffff810d9ff4>]
find_busiest_group+0x2b4/0x8a0
Quoted 2 lines
Hurmph.. what kind of hardware is that? and is there anything funny you do to make it do this?
intel nehanem-ex or westmere-ex 8 sockets system.
I tried without my local patches, the problem is still there.
Yinghai
Re: [tip:sched/urgent] sched: Check sched_domain before computing group power
Quoted 16 lines · Yinghai Lu
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines
This one seems fix NULL reference in
compute_group_power.but get following on current Linus tree plus tip/sched/urgent.
divide error: 0000 [#1] SMP
[ 28.190477] Modules linked in: [ 28.192012] CPU: 11 PID: 484 Comm: kworker/u324:0 Not tainted 3.12.0-yh-10487-g4b94e59-dirty #2044 [ 28.210488] Hardware name: Oracle Corporation Sun Fire [ 28.229877] task: ffff88ff25205140 ti: ffff88ff2520a000 task.ti: ffff88ff2520a000 [ 28.236139] RIP: 0010:[<ffffffff810d9ff4>] [<ffffffff810d9ff4>]
find_busiest_group+0x2b4/0x8a0
Quoted 6 lines
Quoted 2 lines
Hurmph.. what kind of hardware is that? and is there anything funny you do to make it do this?
intel nehanem-ex or westmere-ex 8 sockets system.
I tried without my local patches, the problem is still there.
original one in linus's tree:
[ 8.952728] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 8.965697] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [ 8.969495] IP: [<ffffffff810d7b53>] update_group_power+0x1d3/0x250 [ 8.987159] PGD 0 [ 8.989280] Oops: 0000 [#1] SMP [ 8.991686] Modules linked in: [ 8.993803] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.12.0-yh-02845-g527d151 #2048 [ 9.009175] Hardware name: Oracle Corporation Sun Fire X4800 M2 / , BIOS 15013200 04/19/2012 [ 9.028433] task: ffff883f24e28000 ti: ffff883f24e24000 task.ti: ffff883f24e24000 [ 9.033249] RIP: 0010:[<ffffffff810d7b53>] [<ffffffff810d7b53>] update_group_power+0x1d3/0x250 [ 9.051193] RSP: 0000:ffff883f24e25d68 EFLAGS: 00010283 [ 9.068162] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000 [ 9.071838] RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 00000000000000a0 [ 9.090260] RBP: ffff883f24e25d98 R08: ffff88ffc4891020 R09: 0000000000000000 [ 9.107870] R10: ffff88ffc4890818 R11: 0000000000000001 R12: 00000000001d40c0 [ 9.111527] R13: ffff88ffc4891018 R14: ffff88ffc4891000 R15: 0000000000000000 [ 9.131279] FS: 0000000000000000(0000) GS:ffff883f7d600000(0000) knlGS:0000000000000000 [ 9.148870] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 9.151914] CR2: 0000000000000010 CR3: 0000000002c14000 CR4: 00000000000007f0 [ 9.168645] Stack: [ 9.169871] ffff883f24e25d88 ffff88ffc4891000 ffff88ffc4870000 0000000000000001 [ 9.188660] 0000000000000001 ffff883f23b0d400 ffff883f24e25e58 ffffffff810ce094 [ 9.193232] ffff883f24e25dd8 0000000000000246 0000000000000003 ffffffff000000a0 [ 9.210992] Call Trace: [ 9.212524] [<ffffffff810ce094>] build_sched_domains+0x6f4/0x980 [ 9.229900] [<ffffffff83042a2b>] sched_init_smp+0x95/0x146 [ 9.233236] [<ffffffff83023023>] kernel_init_freeable+0x148/0x259 [ 9.250019] [<ffffffff82121bce>] ? kernel_init+0xe/0x130 [ 9.253356] [<ffffffff82121bc0>] ? rest_init+0xd0/0xd0 [ 9.268882] [<ffffffff82121bce>] kernel_init+0xe/0x130 [ 9.271661] [<ffffffff8215176c>] ret_from_fork+0x7c/0xb0 [ 9.288882] [<ffffffff82121bc0>] ? rest_init+0xd0/0xd0 [ 9.292476] Code: ff 31 db b8 ff ff ff ff 4d 8d 6e 18 eb 31 66 2e 0f 1f 84 00 00 00 00 00 48 63 d0 48 8b 14 d5 40 c4 e2 82 49 8b 94 14 08 09 00 00 <48> 8b 52 10 48 8b 52 10 8b 4a 08 8b 52 04 49 01 cf 48 01 d3 83 [ 9.335669] RIP [<ffffffff810d7b53>] update_group_power+0x1d3/0x250 [ 9.348090] RSP <ffff883f24e25d68> [ 9.350240] CR2: 0000000000000010 [ 9.351803] ---[ end trace a21cca9ad6b48d40 ]--- [ 9.367839] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009 [ 9.367839]
Re: [tip:sched/urgent] sched: Check sched_domain before computing group power
Quoted 7 lines · Yinghai Lu
original one in linus's tree:
[ 8.952728] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 8.965697] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [ 8.969495] IP: [<ffffffff810d7b53>] update_group_power+0x1d3/0x250
This should have been fixed by Srikar's patch, no?
Re: [tip:sched/urgent] sched: Check sched_domain before computing group power
Quoted 11 lines · David Rientjes
Quoted 7 lines · Yinghai Lu
original one in linus's tree:
[ 8.952728] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 8.965697] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [ 8.969495] IP: [<ffffffff810d7b53>] update_group_power+0x1d3/0x250This should have been fixed by Srikar's patch, no?
maybe not related, now in another system, linus's tree + Srikar's patch.
got
[ 33.546361] divide error: 0000 [#1] SMP [ 33.589436] Modules linked in: [ 33.592869] CPU: 15 PID: 567 Comm: kworker/u482:0 Not tainted 3.13.0-rc1-yh-00324-gcf1be1c-dirty #10 [ 33.603075] Hardware name: Oracle Corporation [ 33.609571] calling ipc_ns_init+0x0/0x14 @ 1 [ 33.609575] initcall ipc_ns_init+0x0/0x14 returned 0 after 0 usecs [ 33.609577] calling init_mmap_min_addr+0x0/0x16 @ 1 [ 33.609579] initcall init_mmap_min_addr+0x0/0x16 returned 0 after 0 usecs [ 33.609583] calling init_cpufreq_transition_notifier_list+0x0/0x1b @ 1 [ 33.609621] initcall init_cpufreq_transition_notifier_list+0x0/0x1b returned 0 after 0 usecs [ 33.609624] calling net_ns_init+0x0/0xfa @ 1 [ 33.677194] task: ffff897c5ba5c8c0 ti: ffff897c5ba8e000 task.ti: ffff897c5ba8e000 [ 33.685558] RIP: 0010:[<ffffffff810dbf2c>] [<ffffffff810dbf2c>] find_busiest_group+0x2ac/0x880 [ 33.695310] RSP: 0000:ffff897c5ba8f9a8 EFLAGS: 00010046 [ 33.701253] RAX: 000000000001dfff RBX: 00000000ffffffff RCX: 000000000001e000 [ 33.709226] RDX: 0000000000000000 RSI: 0000000000000078 RDI: 0000000000000000 [ 33.717198] RBP: ffff897c5ba8fb08 R08: 0000000000000000 R09: 0000000000000000 [ 33.725178] R10: 0000000000000000 R11: 000000000001e000 R12: ffff897c5ba8fa90 [ 33.733156] R13: ffff897c5ad61d80 R14: 0000000000000000 R15: ffff897c5ba8fba0 [ 33.741132] FS: 0000000000000000(0000) GS:ffff897d7c200000(0000) knlGS:0000000000000000 [ 33.750164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 33.756593] CR2: 0000000000000168 CR3: 0000000002a14000 CR4: 00000000001407e0 [ 33.764571] Stack: [ 33.766822] 0000000000000000 0000000000000046 0000000000000048 0000000000000000 [ 33.775141] ffff897c5ad61d98 ffff897c5ba8fa20 0000000000000036 00000000000003ab [ 33.783461] 00000000000003ab 0000000000000139 00000000000044e8 0000000100000003 [ 33.791789] Call Trace: [ 33.794549] [<ffffffff810dc6c8>] load_balance+0x1c8/0x8d0 [ 33.800701] [<ffffffff810ee65b>] ? __lock_acquire+0xadb/0xce0 [ 33.807222] [<ffffffff810dd2d1>] idle_balance+0x101/0x1c0 [ 33.813355] [<ffffffff810dd214>] ? idle_balance+0x44/0x1c0 [ 33.819618] [<ffffffff8207a5bb>] __schedule+0x2cb/0xa10 [ 33.825584] [<ffffffff810e86c8>] ? trace_hardirqs_off_caller+0x28/0x160 [ 33.833089] [<ffffffff810e880d>] ? trace_hardirqs_off+0xd/0x10 [ 33.839731] [<ffffffff810d3b84>] ? local_clock+0x34/0x60 [ 33.845788] [<ffffffff810ba7bb>] ? worker_thread+0x2db/0x370 [ 33.852241] [<ffffffff8207f8a0>] ? _raw_spin_unlock_irq+0x30/0x40 [ 33.859150] [<ffffffff8207ad65>] schedule+0x65/0x70 [ 33.864700] [<ffffffff810ba7c0>] worker_thread+0x2e0/0x370 [ 33.870932] [<ffffffff810ec17d>] ? trace_hardirqs_on+0xd/0x10 [ 33.877472] [<ffffffff810ba4e0>] ? manage_workers.isra.17+0x330/0x330 [ 33.884789] [<ffffffff810c18c8>] kthread+0x108/0x110 [ 33.890441] [<ffffffff810c17c0>] ? __init_kthread_worker+0x70/0x70 [ 33.897465] [<ffffffff8208812c>] ret_from_fork+0x7c/0xb0 [ 33.903504] [<ffffffff810c17c0>] ? __init_kthread_worker+0x70/0x70 [ 33.910508] Code: 89 85 b8 fe ff ff 49 8b 45 10 41 8b 75 0c 44 8b 50 08 44 8b 58 04 89 f0 48 c1 e0 0a 45 89 d1 49 8d 44 01 ff 48 89 c2 48 c1 fa 3f <49> f7 f9 31 d2 49 89 c1 89 f0 44 89 de 41 f7 f1 48 81 c6 00 02 [ 33.932375] RIP [<ffffffff810dbf2c>] find_busiest_group+0x2ac/0x880 [ 33.939491] RSP <ffff897c5ba8f9a8> [ 33.943418] ---[ end trace 7a833c0cac54cac8 ]---
Peter Zijlstra
maintainerRe: [tip:sched/urgent] sched: Check sched_domain before computing group power
Quoted 11 lines · Yinghai Lu
Quoted 10 lines · David Rientjes
Quoted 7 lines · Yinghai Lu
original one in linus's tree:
[ 8.952728] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 8.965697] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [ 8.969495] IP: [<ffffffff810d7b53>] update_group_power+0x1d3/0x250
Quoted 3 lines
Quoted 1 line
This should have been fixed by Srikar's patch, no?
maybe not related, now in another system, linus's tree + Srikar's patch.
And this another such -EX system? And from what I can tell its during boot, right?
Totally weird.
Re: [tip:sched/urgent] sched: Check sched_domain before computing group power
Quoted 12 lines · Peter Zijlstra
Quoted 11 lines · Yinghai Lu
Quoted 10 lines · David Rientjes
Quoted 7 lines · Yinghai Lu
original one in linus's tree:
[ 8.952728] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 8.965697] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [ 8.969495] IP: [<ffffffff810d7b53>] update_group_power+0x1d3/0x250
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
This should have been fixed by Srikar's patch, no?
maybe not related, now in another system, linus's tree + Srikar's patch.
And this another such -EX system? And from what I can tell its during boot, right?
Ivybridge EX, yes.
Re: [tip:sched/urgent] sched: Check sched_domain before computing group power
In reply to Yinghai LuQuoted 59 lines · Yinghai Lu
On Wed, Nov 27, 2013 at 7:02 PM, David Rientjes <[email protected]> wrote:
maybe not related, now in another system, linus's tree + Srikar's patch.
got
[ 33.546361] divide error: 0000 [#1] SMP [ 33.589436] Modules linked in: [ 33.592869] CPU: 15 PID: 567 Comm: kworker/u482:0 Not tainted 3.13.0-rc1-yh-00324-gcf1be1c-dirty #10 [ 33.603075] Hardware name: Oracle Corporation [ 33.609571] calling ipc_ns_init+0x0/0x14 @ 1 [ 33.609575] initcall ipc_ns_init+0x0/0x14 returned 0 after 0 usecs [ 33.609577] calling init_mmap_min_addr+0x0/0x16 @ 1 [ 33.609579] initcall init_mmap_min_addr+0x0/0x16 returned 0 after 0 usecs [ 33.609583] calling init_cpufreq_transition_notifier_list+0x0/0x1b @ 1 [ 33.609621] initcall init_cpufreq_transition_notifier_list+0x0/0x1b returned 0 after 0 usecs [ 33.609624] calling net_ns_init+0x0/0xfa @ 1 [ 33.677194] task: ffff897c5ba5c8c0 ti: ffff897c5ba8e000 task.ti: ffff897c5ba8e000 [ 33.685558] RIP: 0010:[<ffffffff810dbf2c>] [<ffffffff810dbf2c>] find_busiest_group+0x2ac/0x880 [ 33.695310] RSP: 0000:ffff897c5ba8f9a8 EFLAGS: 00010046 [ 33.701253] RAX: 000000000001dfff RBX: 00000000ffffffff RCX: 000000000001e000 [ 33.709226] RDX: 0000000000000000 RSI: 0000000000000078 RDI: 0000000000000000 [ 33.717198] RBP: ffff897c5ba8fb08 R08: 0000000000000000 R09: 0000000000000000 [ 33.725178] R10: 0000000000000000 R11: 000000000001e000 R12: ffff897c5ba8fa90 [ 33.733156] R13: ffff897c5ad61d80 R14: 0000000000000000 R15: ffff897c5ba8fba0 [ 33.741132] FS: 0000000000000000(0000) GS:ffff897d7c200000(0000) knlGS:0000000000000000 [ 33.750164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 33.756593] CR2: 0000000000000168 CR3: 0000000002a14000 CR4: 00000000001407e0 [ 33.764571] Stack: [ 33.766822] 0000000000000000 0000000000000046 0000000000000048 0000000000000000 [ 33.775141] ffff897c5ad61d98 ffff897c5ba8fa20 0000000000000036 00000000000003ab [ 33.783461] 00000000000003ab 0000000000000139 00000000000044e8 0000000100000003 [ 33.791789] Call Trace: [ 33.794549] [<ffffffff810dc6c8>] load_balance+0x1c8/0x8d0 [ 33.800701] [<ffffffff810ee65b>] ? __lock_acquire+0xadb/0xce0 [ 33.807222] [<ffffffff810dd2d1>] idle_balance+0x101/0x1c0 [ 33.813355] [<ffffffff810dd214>] ? idle_balance+0x44/0x1c0 [ 33.819618] [<ffffffff8207a5bb>] __schedule+0x2cb/0xa10 [ 33.825584] [<ffffffff810e86c8>] ? trace_hardirqs_off_caller+0x28/0x160 [ 33.833089] [<ffffffff810e880d>] ? trace_hardirqs_off+0xd/0x10 [ 33.839731] [<ffffffff810d3b84>] ? local_clock+0x34/0x60 [ 33.845788] [<ffffffff810ba7bb>] ? worker_thread+0x2db/0x370 [ 33.852241] [<ffffffff8207f8a0>] ? _raw_spin_unlock_irq+0x30/0x40 [ 33.859150] [<ffffffff8207ad65>] schedule+0x65/0x70 [ 33.864700] [<ffffffff810ba7c0>] worker_thread+0x2e0/0x370 [ 33.870932] [<ffffffff810ec17d>] ? trace_hardirqs_on+0xd/0x10 [ 33.877472] [<ffffffff810ba4e0>] ? manage_workers.isra.17+0x330/0x330 [ 33.884789] [<ffffffff810c18c8>] kthread+0x108/0x110 [ 33.890441] [<ffffffff810c17c0>] ? __init_kthread_worker+0x70/0x70 [ 33.897465] [<ffffffff8208812c>] ret_from_fork+0x7c/0xb0
Quoted 8 lines
[ 33.903504] [<ffffffff810c17c0>] ? __init_kthread_worker+0x70/0x70 [ 33.910508] Code: 89 85 b8 fe ff ff 49 8b 45 10 41 8b 75 0c 44 8b 50 08 44 8b 58 04 89 f0 48 c1 e0 0a 45 89 d1 49 8d 44 01 ff 48 89 c2 48 c1 fa 3f <49> f7 f9 31 d2 49 89 c1 89 f0 44 89 de 41 f7 f1 48 81 c6 00 02 [ 33.932375] RIP [<ffffffff810dbf2c>] find_busiest_group+0x2ac/0x880 [ 33.939491] RSP <ffff897c5ba8f9a8> [ 33.943418] ---[ end trace 7a833c0cac54cac8 ]---
Hi, PeterZ,
This divide_by_zero could be workaround with attached patch.
Yinghai
--- kernel/sched/core.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/sched/core.c +3 -0
@@ -5737,6 +5737,9 @@ static int __sdt_alloc(const struct cpumif (!sgp)return -ENOMEM ; +/* avoid divide-by-zero in sg_capacity() */ + sgp->power_orig =1 ; + *per_cpu_ptr (sdd->sgp, j) = sgp; } }
Peter Zijlstra
maintainerRe: [tip:sched/urgent] sched: Check sched_domain before computing group power
Quoted 9 lines · Yinghai Lu
--- kernel/sched/core.c | 3 +++ 1 file changed, 3 insertions(+)Index:
linux-2.6/kernel/sched/core.c
===================================================================
kernel/sched/core.c static int __sdt_alloc(const struct cpum · line 5745
Quoted 9 lines
kernel/sched/core.c +3
-0@@ -5737,6 +5737,9 @@ static int __sdt_alloc(const struct cpumif (!sgp)return -ENOMEM ; +/* avoid divide-by-zero in sg_capacity() */ + sgp->power_orig =1 ; + *per_cpu_ptr (sdd->sgp, j) = sgp; } }
Ooh, sg_capacity() is generating the /0..
Does the below work too?
--- kernel/sched/core.c | 1 + 1 file changed, 1 insertion(+)
kernel/sched/core.c +1 -0
@@ -5115,6 +5115,7 @@ build_overlap_sched_groups(struct sched_domain *sd, int cpu)* die on a /0 trap. */ sg->sgp->power =SCHED_POWER_SCALE *cpumask_weight (sg_span); + sg->sgp->power_orig = sg->sgp->power;/* * Make sure the first group of this domain contains the
Re: [tip:sched/urgent] sched: Check sched_domain before computing group power
Quoted 13 lines · Peter Zijlstra
Ooh,
sg_capacity()is generating the /0..Does the below work too?
--- kernel/sched/core.c | 1 + 1 file changed, 1 insertion(+)kernel/sched/core.c +0
-0
kernel/sched/core.c build_overlap_sched_groups(struct sched_domain *sd, int cpu) · line 5121
Quoted 7 lines
kernel/sched/core.c +1
-0@@ -5115,6 +5115,7 @@ build_overlap_sched_groups(struct sched_domain *sd, int cpu)* die on a /0 trap. */ sg->sgp->power =SCHED_POWER_SCALE *cpumask_weight (sg_span); + sg->sgp->power_orig = sg->sgp->power;/* * Make sure the first group of this domain contains the
Yes, it works.
Peter Zijlstra
maintainerRe: [tip:sched/urgent] sched: Check sched_domain before computing group power
In reply to Yinghai LuQuoted 16 lines · Yinghai Lu
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines
This one seems fix NULL reference in
compute_group_power.but get following on current Linus tree plus tip/sched/urgent.
divide error: 0000 [#1] SMP
[ 28.190477] Modules linked in: [ 28.192012] CPU: 11 PID: 484 Comm: kworker/u324:0 Not tainted 3.12.0-yh-10487-g4b94e59-dirty #2044 [ 28.210488] Hardware name: Oracle Corporation Sun Fire [ 28.229877] task: ffff88ff25205140 ti: ffff88ff2520a000 task.ti: ffff88ff2520a000 [ 28.236139] RIP: 0010:[<ffffffff810d9ff4>] [<ffffffff810d9ff4>]
find_busiest_group+0x2b4/0x8a0
Quoted 6 lines
Quoted 2 lines
Hurmph.. what kind of hardware is that? and is there anything funny you do to make it do this?
intel nehanem-ex or westmere-ex 8 sockets system.
I tried without my local patches, the problem is still there.
And I suppose a kernel before
863bffc80898 ("sched/fair: Fix group power_orig computation")
work fine, eh?
I'll further assume that your RIP points to:
sds.avg_load = (SCHED_POWER_SCALE * sds.total_load) / sds.total_pwr;
indicating that sds.total_pwr := 0.
update_sd_lb_stats() computes it like:
sds->total_pwr += sgs->group_power;
which comes out of update_sg_lb_stats() like:
sgs->group_power = group->sgp->power;
Which we compute in update_group_power() similarly to how we did before 863bffc80898.
Which leaves me a bit puzzled.
Re: [tip:sched/urgent] sched: Check sched_domain before computing group power
Quoted 40 lines · Peter Zijlstra
Quoted 16 lines · Yinghai Lu
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines
This one seems fix NULL reference in
compute_group_power.but get following on current Linus tree plus tip/sched/urgent.
divide error: 0000 [#1] SMP
[ 28.190477] Modules linked in: [ 28.192012] CPU: 11 PID: 484 Comm: kworker/u324:0 Not tainted 3.12.0-yh-10487-g4b94e59-dirty #2044 [ 28.210488] Hardware name: Oracle Corporation Sun Fire [ 28.229877] task: ffff88ff25205140 ti: ffff88ff2520a000 task.ti: ffff88ff2520a000 [ 28.236139] RIP: 0010:[<ffffffff810d9ff4>] [<ffffffff810d9ff4>]
find_busiest_group+0x2b4/0x8a0Quoted 6 lines
Quoted 2 lines
Hurmph.. what kind of hardware is that? and is there anything funny you do to make it do this?
intel nehanem-ex or westmere-ex 8 sockets system.
I tried without my local patches, the problem is still there.
And I suppose a kernel before
863bffc80898("sched/fair: Fix grouppower_origcomputation")work fine, eh?
I'll further assume that your RIP points to:
sds.avg_load = (SCHED_POWER_SCALE * sds.total_load) / sds.total_pwr;indicating that sds.total_pwr := 0.
update_sd_lb_stats()computes it like:sds->total_pwr += sgs->group_power;
Quoted 8 lines
which comes out of
update_sg_lb_stats()like:sgs->group_power = group->sgp->power;Which we compute in
update_group_power()similarly to how we did before863bffc80898.Which leaves me a bit puzzled.
Hi,
for linus tree i need to revert commit-863bffc.
commit-863bffc
for linus tree + sched/urgent, I need to revert
commit-42eb088
commit-9abf24d
commit-863bffc
.
If only revert commit-42eb088, still have problem.
if only revert commit-9abf24d, commit-863bffc, still have problem.
Assume you need to dump sched/urgent,
and revert commit-863bffc directly from Linus's tree.
Thanks
Yinghai
Peter Zijlstra
maintainerRe: [tip:sched/urgent] sched: Check sched_domain before computing group power
Quoted 38 lines · Yinghai Lu
Quoted 16 lines · Peter Zijlstra
And I suppose a kernel before
863bffc80898("sched/fair: Fix grouppower_origcomputation")work fine, eh?
I'll further assume that your RIP points to:
sds.avg_load = (SCHED_POWER_SCALE * sds.total_load) / sds.total_pwr;indicating that sds.total_pwr := 0.
update_sd_lb_stats()computes it like:sds->total_pwr += sgs->group_power;Quoted 8 lines
which comes out of
update_sg_lb_stats()like:sgs->group_power = group->sgp->power;Which we compute in
update_group_power()similarly to how we did before863bffc80898.Which leaves me a bit puzzled.
Hi,
for linus tree i need to revert commit-863bffc.
commit-863bffcfor linus tree + sched/urgent, I need to revert
commit-42eb088
commit-9abf24d
commit-863bffc
.
If only revert commit-42eb088, still have problem.
if only revert commit-9abf24d, commit-863bffc, still have problem.
Quoted 2 lines
Assume you need to dump sched/urgent,
and revert commit-863bffc directly from Linus's tree.
That doesn't answer any of the questions above and only raises more questions.
I also will not revert until a little later, I really need to understand this. My wsm-ep system boots just fine, so there's something funny somewhere.
I also cannot see the difference between 863bffc^1 and 9abf24d.
Also, you mentioning 42eb088 is new; what does that have to do with anything? You cannot revert that without also reverting 37dc6b50cee9, but you don't mention that commit at all.
David Rientjes
Acked-byRe: [tip:sched/urgent] sched: Check sched_domain before computing group power
In reply to tip-bot for Srikar DronamrajuQuoted 15 lines · tip-bot for Srikar Dronamraju
Commit-ID: 9abf24d465180f5f2eb26a43545348262f16b771 Gitweb: http://git.kernel.org/tip/9abf24d465180f5f2eb26a43545348262f16b771 Author: Srikar Dronamraju <[email protected]> AuthorDate: Tue, 12 Nov 2013 22:11:26 +0530 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 19 Nov 2013 17:01:15 +0100sched: Check
sched_domainbefore computing group powerAfter commit
863bffc80898("sched/fair: Fix grouppower_origcomputation"), we can dereferencerq->sdbefore it is set.Fix this by falling back to
power_of()in this case and add a comment explaining things.
Quoted 6 lines
Signed-off-by Srikar Dronamraju <[email protected]>[ Added comment and tweaked patch. ]Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Ingo Molnar <[email protected]>
Fixes a boot failure for me, thanks.