Skip to contentSkip to conversation

x86, hotplug: fix llc shared map unreleased during cpu hotplug

PATCH Superseded Wanpeng Li 4 participants 9 messages Jul 22, 2014 – Jul 29, 2014 x86

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Wanpeng Li

author
#0 lore raw

PATCH v2 1/1 x86, hotplug: fix llc shared map unreleased during cpu hotplug

[  220.262093] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
[  220.262104] IP: [<ffffffff810e7ac9>] find_busiest_group+0x2b9/0xa30
[  220.262111] PGD 5a9d5067 PUD 13067 PMD 0
[  220.262117] Oops: 0000 [#3] SMP
[...]
[  220.262245] Call Trace:
[  220.262252]  [<ffffffff810e8396>] load_balance+0x156/0x980
[  220.262259]  [<ffffffff816eeffe>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0
[  220.262266]  [<ffffffff810e9aa3>] idle_balance+0xe3/0x150
[  220.262270]  [<ffffffff816ec4e7>] __schedule+0x797/0x8d0
[  220.262277]  [<ffffffff816ec934>] schedule+0x24/0x70
[  220.262283]  [<ffffffff816e9cd9>] schedule_timeout+0x119/0x1f0
[  220.262294]  [<ffffffff810bb6e0>] ? lock_timer_base+0x70/0x70
[  220.262301]  [<ffffffff816e9dc9>] schedule_timeout_uninterruptible+0x19/0x20
[  220.262308]  [<ffffffff810bd3e8>] msleep+0x18/0x20
[  220.262317]  [<ffffffff813aa11a>] lock_device_hotplug_sysfs+0x2a/0x50
[  220.262323]  [<ffffffff813aa16e>] online_store+0x2e/0x80
[  220.262358]  [<ffffffff813a873b>] dev_attr_store+0x1b/0x20
[  220.262366]  [<ffffffff812292fd>] sysfs_write_file+0xdd/0x160
[  220.262377]  [<ffffffff811b7e78>] vfs_write+0xc8/0x170
[  220.262384]  [<ffffffff811b83ca>] SyS_write+0x5a/0xa0
[  220.262388]  [<ffffffff816f76b9>] system_call_fastpath+0x16/0x1b

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

---
v1 -> v2:
* fix subject line

 arch/x86/kernel/smpboot.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kernel/smpboot.c +3 -0
index 5492798..0134ec7 100644
@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
 
 	for_each_cpu(sibling, cpu_sibling_mask(cpu))
 		cpumask_clear_cpu(cpu, cpu_sibling_mask(sibling));
+	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
+		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
+	cpumask_clear(cpu_llc_shared_mask(cpu));
 	cpumask_clear(cpu_sibling_mask(cpu));
 	cpumask_clear(cpu_core_mask(cpu));
 	c->phys_proc_id = 0;
Signature · 1 line
1.9.1
author
#2 lore raw
Quoted 6 lines · Gong
Quoted 2 lines · Wanpeng Li

Subject: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu
hotplug

See this link:
https://lkml.org/lkml/2014/7/17/78

Interesting, thanks for your pointing out.

Regards,
Wanpeng Li

maintainer
#3 lore raw
Quoted 9 lines · Wanpeng Li
Quoted 6 lines · Gong
Quoted 2 lines · Wanpeng Li

Subject: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu
hotplug

See this link:
https://lkml.org/lkml/2014/7/17/78

Interesting, thanks for your pointing out.

Well, we might end up doing both.

Yasuaki's patch makes the core numbering stable which makes a lot of sense in and of itself.

However, if some code somewhere relies implicitly on the fact that a set bit in LLC mask means that core is online and goes and does stuff on it, Wanpeng's reported splat will happen.

So probably we will have to have *stable* core numbers *and* clear the LLC mask in order to be on the safe side of things.

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#4 lore raw
In reply to Wanpeng Li
Quoted 23 lines · Wanpeng Li
[  220.262093] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
[  220.262104] IP: [<ffffffff810e7ac9>] find_busiest_group+0x2b9/0xa30
[  220.262111] PGD 5a9d5067 PUD 13067 PMD 0
[  220.262117] Oops: 0000 [#3] SMP
[...]
[  220.262245] Call Trace:
[  220.262252]  [<ffffffff810e8396>] load_balance+0x156/0x980
[  220.262259]  [<ffffffff816eeffe>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0
[  220.262266]  [<ffffffff810e9aa3>] idle_balance+0xe3/0x150
[  220.262270]  [<ffffffff816ec4e7>] __schedule+0x797/0x8d0
[  220.262277]  [<ffffffff816ec934>] schedule+0x24/0x70
[  220.262283]  [<ffffffff816e9cd9>] schedule_timeout+0x119/0x1f0
[  220.262294]  [<ffffffff810bb6e0>] ? lock_timer_base+0x70/0x70
[  220.262301]  [<ffffffff816e9dc9>] schedule_timeout_uninterruptible+0x19/0x20
[  220.262308]  [<ffffffff810bd3e8>] msleep+0x18/0x20
[  220.262317]  [<ffffffff813aa11a>] lock_device_hotplug_sysfs+0x2a/0x50
[  220.262323]  [<ffffffff813aa16e>] online_store+0x2e/0x80
[  220.262358]  [<ffffffff813a873b>] dev_attr_store+0x1b/0x20
[  220.262366]  [<ffffffff812292fd>] sysfs_write_file+0xdd/0x160
[  220.262377]  [<ffffffff811b7e78>] vfs_write+0xc8/0x170
[  220.262384]  [<ffffffff811b83ca>] SyS_write+0x5a/0xa0
[  220.262388]  [<ffffffff816f76b9>] system_call_fastpath+0x16/0x1b
Quoted 6 lines

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

I posted a latest patch as follows:
https://lkml.org/lkml/2014/7/22/1018

Could you confirm the patch fixes your issue?

Thanks,
Yasuaki Ishimatsu

Quoted 23 lines

---
v1 -> v2:
* fix subject line

  arch/x86/kernel/smpboot.c | 3 +++
  1 file changed, 3 insertions(+)
arch/x86/kernel/smpboot.c +3 -0
index 5492798..0134ec7 100644
@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
 
  	for_each_cpu(sibling, cpu_sibling_mask(cpu))
  		cpumask_clear_cpu(cpu, cpu_sibling_mask(sibling));
+	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
+		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
+	cpumask_clear(cpu_llc_shared_mask(cpu));
  	cpumask_clear(cpu_sibling_mask(cpu));
  	cpumask_clear(cpu_core_mask(cpu));
  	c->phys_proc_id = 0;
author
#5 lore raw

Hi Yasuaki,

Quoted 31 lines · Yasuaki Ishimatsu
Quoted 23 lines · Wanpeng Li
[  220.262093] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
[  220.262104] IP: [<ffffffff810e7ac9>] find_busiest_group+0x2b9/0xa30
[  220.262111] PGD 5a9d5067 PUD 13067 PMD 0
[  220.262117] Oops: 0000 [#3] SMP
[...]
[  220.262245] Call Trace:
[  220.262252]  [<ffffffff810e8396>] load_balance+0x156/0x980
[  220.262259]  [<ffffffff816eeffe>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0
[  220.262266]  [<ffffffff810e9aa3>] idle_balance+0xe3/0x150
[  220.262270]  [<ffffffff816ec4e7>] __schedule+0x797/0x8d0
[  220.262277]  [<ffffffff816ec934>] schedule+0x24/0x70
[  220.262283]  [<ffffffff816e9cd9>] schedule_timeout+0x119/0x1f0
[  220.262294]  [<ffffffff810bb6e0>] ? lock_timer_base+0x70/0x70
[  220.262301]  [<ffffffff816e9dc9>] schedule_timeout_uninterruptible+0x19/0x20
[  220.262308]  [<ffffffff810bd3e8>] msleep+0x18/0x20
[  220.262317]  [<ffffffff813aa11a>] lock_device_hotplug_sysfs+0x2a/0x50
[  220.262323]  [<ffffffff813aa16e>] online_store+0x2e/0x80
[  220.262358]  [<ffffffff813a873b>] dev_attr_store+0x1b/0x20
[  220.262366]  [<ffffffff812292fd>] sysfs_write_file+0xdd/0x160
[  220.262377]  [<ffffffff811b7e78>] vfs_write+0xc8/0x170
[  220.262384]  [<ffffffff811b83ca>] SyS_write+0x5a/0xa0
[  220.262388]  [<ffffffff816f76b9>] system_call_fastpath+0x16/0x1b
Quoted 6 lines

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 4 lines

I posted a latest patch as follows:
https://lkml.org/lkml/2014/7/22/1018

Could you confirm the patch fixes your issue?

Sorry for the late, there is still call trace w/ your patch applied. The call trace is in attachment.

Regards,
Wanpeng Li

Quoted 18 lines

Thanks,
Yasuaki Ishimatsu

Quoted 14 lines

---
v1 -> v2:
* fix subject line

  arch/x86/kernel/smpboot.c | 3 +++
  1 file changed, 3 insertions(+)
arch/x86/kernel/smpboot.c +0 -0
index 5492798..0134ec7 100644
@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
 
Quoted 10 lines

arch/x86/kernel/smpboot.c static void remove_siblinginfo(int cpu) · line 1300

Quoted 9 lines
arch/x86/kernel/smpboot.c +3 -0
  	for_each_cpu(sibling, cpu_sibling_mask(cpu))
  		cpumask_clear_cpu(cpu, cpu_sibling_mask(sibling));
+	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
+		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
+	cpumask_clear(cpu_llc_shared_mask(cpu));
  	cpumask_clear(cpu_sibling_mask(cpu));
  	cpumask_clear(cpu_core_mask(cpu));
  	c->phys_proc_id = 0;

when run "xl vcpu-set 0 2", the dom0 only report "broke affinity ..." when run "xl vcpu-set 0 26", the call trace happens.

the dom0 call trace log as following:

[  295.464489] Broke affinity for irq 298
[  295.756205] Broke affinity for irq 299
[  295.767177] Broke affinity for irq 301
[  295.779177] Broke affinity for irq 303
[  366.283682] installing Xen timer for CPU 2
[  366.283749] cpu 2 spinlock event irq 103
[  366.310290] installing Xen timer for CPU 14
[  366.310347] cpu 14 spinlock event irq 110
[  366.312432] divide error: 0000 [#1] SMP
[  366.312449] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4
d
[  366.312583] CPU: 14 PID: 63 Comm: ksoftirqd/14 Not tainted 3.15.6 #2
[  366.312598] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
GRNDSDP4
[  366.312623] task: ffff88017c8d2c10 ti: ffff88017c8f0000 task.ti:
ffff88017c80
[  366.312647] RIP: e030:[<ffffffff810ea5f9>]  [<ffffffff810ea5f9>]
find_busies0
[  366.312681] RSP: e02b:ffff88017c8f3ac8  EFLAGS: 00010046
[  366.312694] RAX: 0000000000000000 RBX: ffff88017c8f3bc8 RCX:
0000000000000000
[  366.312708] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[  366.312724] RBP: ffff88017c8f3c38 R08: ffff880003fb3d00 R09:
0000000000000040
[  366.312742] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000013e00
[  366.312757] R13: ffff88017c8f3cb8 R14: ffff880003fb3ce0 R15:
0000000000000000
[  366.312783] FS:  0000000000000000(0000) GS:ffff880181bc0000(0000)
knlGS:00000
[  366.312803] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[  366.312817] CR2: 00007fad200d5000 CR3: 0000000001c14000 CR4:
0000000000042660
[  366.312836] Stack:
[  366.312843]  0000000000000000 ffff88017c8f3b18 0000000000002e7b
0000000000000
[  366.312868]  ffff880003fb3ce0 0000000000013df8 0000000000000200
0000000000010
[  366.312890]  0000000000000000 ffff880003fb3cf8 0000000000000000
0000000000000
[  366.312911] Call Trace:
[  366.312932]  [<ffffffff810eae37>] load_balance+0x177/0x9d0
[  366.312954]  [<ffffffff810df56b>] ? update_rq_clock+0x2b/0x50
[  366.312976]  [<ffffffff81058ea0>] ? xen_clocksource_read+0x20/0x30
[  366.312997]  [<ffffffff810edb8d>] pick_next_task_fair+0x1ed/0x430
[  366.313019]  [<ffffffff816ff0d3>] __schedule+0x113/0x870
[  366.313039]  [<ffffffff816ff9b4>] ? schedule+0x24/0x70
[  366.313059]  [<ffffffff816ff9b4>] schedule+0x24/0x70
[  366.313095]  [<ffffffff810dcd7c>] smpboot_thread_fn+0xbc/0x190
[  366.313112]  [<ffffffff810dccc0>] ? smpboot_create_threads+0x80/0x80
[  366.313135]  [<ffffffff810d565e>] kthread+0xce/0xf0
[  366.313155]  [<ffffffff810d5590>] ? kthread_freezable_should_stop+0x70/0x70
[  366.313174]  [<ffffffff8170c54c>] ret_from_fork+0x7c/0xb0
[  366.313190]  [<ffffffff810d5590>] ? kthread_freezable_should_stop+0x70/0x70
[  366.313204] Code: 0f 47 d1 eb 95 0f 1f 44 00 00 4d 89 ec 4d 89 f5 4c 8b b5 b
[  366.313372] RIP  [<ffffffff810ea5f9>] find_busiest_group+0x239/0x900
[  366.313391]  RSP <ffff88017c8f3ac8>
[  366.313406] ---[ end trace 42d3248df75182f3 ]---
[  366.313758] divide error: 0000 [#2] SMP
[  366.313776] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4
d
[  366.313883] CPU: 14 PID: 63 Comm: ksoftirqd/14 Tainted: G      D      
3.15.2
[  366.313898] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
GRNDSDP4
[  366.313922] task: ffff88017c8d2c10 ti: ffff88017c8f0000 task.ti:
ffff88017c80
[  366.313940] RIP: e030:[<ffffffff810ea5f9>]  [<ffffffff810ea5f9>]
find_busies0
[  366.313966] RSP: e02b:ffff88017c8f3468  EFLAGS: 00010046
[  366.313979] RAX: 0000000000000000 RBX: ffff88017c8f3568 RCX:
0000000000000000
[  366.313993] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[  366.314008] RBP: ffff88017c8f35d8 R08: ffff880003fb3d00 R09:
0000000000000040
[  366.314023] R10: 0000000000000000 R11: ffff880186148410 R12:
0000000000013e00
[  366.314042] R13: ffff88017c8f3658 R14: ffff880003fb3ce0 R15:
0000000000000000
[  366.314067] FS:  0000000000000000(0000) GS:ffff880181bc0000(0000)
knlGS:00000
[  366.314090] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[  366.314103] CR2: 00007fad200d5000 CR3: 0000000001c14000 CR4:
0000000000042660
[  366.314119] Stack:
[  366.314125]  ffff8801fc8f35c9 ffff88017c8f34b8 0000000000002e7b
00000000812ce
[  366.314150]  ffff880003fb3ce0 0000000000013df8 0000000000000200
0000000000010
[  366.314175]  000000006c106009 ffff880003fb3cf8 0000000000000000
0000000000000
[  366.314196] Call Trace:
[  366.314215]  [<ffffffff810eae37>] load_balance+0x177/0x9d0
[  366.314232]  [<ffffffff810df56b>] ? update_rq_clock+0x2b/0x50
[  366.314252]  [<ffffffff81058ea0>] ? xen_clocksource_read+0x20/0x30
[  366.314269]  [<ffffffff810edb8d>] pick_next_task_fair+0x1ed/0x430
[  366.314288]  [<ffffffff816ff0d3>] __schedule+0x113/0x870
[  366.314307]  [<ffffffff810b3a34>] ? release_task+0x304/0x480
[  366.314324]  [<ffffffff816ff9b4>] schedule+0x24/0x70
[  366.314340]  [<ffffffff810b42ac>] do_exit+0x6fc/0xac0
[  366.314356]  [<ffffffff81705008>] oops_end+0xa8/0x170
[  366.314371]  [<ffffffff810663b6>] die+0x56/0x90
[  366.314385]  [<ffffffff81704a23>] do_trap+0xc3/0x170
[  366.314402]  [<ffffffff8170812d>] ? __atomic_notifier_call_chain+0xd/0x10
[  366.314422]  [<ffffffff8106395b>] do_divide_error+0x9b/0xb0
[  366.314439]  [<ffffffff810ea5f9>] ? find_busiest_group+0x239/0x900
[  366.314456]  [<ffffffff8170dc0e>] divide_error+0x1e/0x30
[  366.314473]  [<ffffffff810ea5f9>] ? find_busiest_group+0x239/0x900
[  366.314491]  [<ffffffff810ea513>] ? find_busiest_group+0x153/0x900
[  366.314511]  [<ffffffff810eae37>] load_balance+0x177/0x9d0
[  366.314526]  [<ffffffff810df56b>] ? update_rq_clock+0x2b/0x50
[  366.314547]  [<ffffffff81058ea0>] ? xen_clocksource_read+0x20/0x30
[  366.314563]  [<ffffffff810edb8d>] pick_next_task_fair+0x1ed/0x430
[  366.314581]  [<ffffffff816ff0d3>] __schedule+0x113/0x870
[  366.314597]  [<ffffffff816ff9b4>] ? schedule+0x24/0x70
[  366.314613]  [<ffffffff816ff9b4>] schedule+0x24/0x70
[  366.314628]  [<ffffffff810dcd7c>] smpboot_thread_fn+0xbc/0x190
[  366.314650]  [<ffffffff810dccc0>] ? smpboot_create_threads+0x80/0x80
[  366.314668]  [<ffffffff810d565e>] kthread+0xce/0xf0
[  366.314684]  [<ffffffff810d5590>] ? kthread_freezable_should_stop+0x70/0x70
[  366.314701]  [<ffffffff8170c54c>] ret_from_fork+0x7c/0xb0
[  366.314717]  [<ffffffff810d5590>] ? kthread_freezable_should_stop+0x70/0x70
[  366.314735] Code: 0f 47 d1 eb 95 0f 1f 44 00 00 4d 89 ec 4d 89 f5 4c 8b b5 b
[  366.314891] RIP  [<ffffffff810ea5f9>] find_busiest_group+0x239/0x900
[  366.314909]  RSP <ffff88017c8f3468>
[  366.314927] ---[ end trace 42d3248df75182f4 ]---
[  366.314932] BUG: unable to handle kernel NULL pointer dereference at
0000000c
[  366.314938] IP: [<ffffffff810e86e7>] select_task_rq_fair+0x337/0x8c0
[  366.314942] PGD 0
[  366.314943] Oops: 0000 [#3] SMP
[  366.314960] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4
d
[  366.314962] CPU: 1 PID: 8225 Comm: udevd Tainted: G      D       3.15.6 #2
[  366.314965] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
GRNDSDP4
[  366.314966] task: ffff8801771634e0 ti: ffff880002598000 task.ti:
ffff88000250
[  366.314972] RIP: e030:[<ffffffff810e86e7>]  [<ffffffff810e86e7>]
select_task0
[  366.314973] RSP: e02b:ffff88000259bd48  EFLAGS: 00010046
[  366.314974] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000019
[  366.314975] RDX: 0000000000000008 RSI: 0000000000000040 RDI:
0000000000000040
[  366.314979] RBP: ffff88000259be28 R08: ffff880003fb33f8 R09:
0000000000000000
[  366.314980] R10: 0000000000000000 R11: ffff88017cfe4338 R12:
0000000000000000
[  366.314981] R13: ffff880003fb33f8 R14: ffff880003fb33e0 R15:
0000000000000000
[  366.314988] FS:  00007fad200bb7a0(0000) GS:ffff880181a20000(0000)
knlGS:00000
[  366.314989] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[  366.314990] CR2: 000000000000000c CR3: 00000000030f2000 CR4:
0000000000042660
[  366.314991] Stack:
[  366.314993]  ffff88017c7de000 00000000ffffff9c ffff88000259be38
ffffffff811d5
[  366.314995]  0000000000013e00 0000000000013e00 ffff8801771637d8
000000000000d
[  366.314997]  ffff880003fb3420 0000000000001ade 0000000000013e00
0000000000018
[  366.314998] Call Trace:
[  366.315003]  [<ffffffff811d6a45>] ? do_filp_open+0x45/0xa0
[  366.315005]  [<ffffffff810dedd7>] sched_exec+0x47/0xc0
[  366.315009]  [<ffffffff811ccaca>] ? do_open_exec+0xaa/0xe0
[  366.315014]  [<ffffffff811cccee>] do_execve_common+0x1be/0x640
[  366.315019]  [<ffffffff811b5b77>] ? kmem_cache_alloc+0x37/0x120
[  366.315021]  [<ffffffff811cd202>] do_execve+0x32/0x40
[  366.315026]  [<ffffffff811cd23a>] SyS_execve+0x2a/0x40
[  366.315029]  [<ffffffff8170cba9>] stub_execve+0x69/0xa0
[  366.315055] Code: 48 8b 55 c0 4d 8b 36 4c 3b 72 10 74 43 48 89 45 b0 e9 6e f
[  366.315058] RIP  [<ffffffff810e86e7>] select_task_rq_fair+0x337/0x8c0
[  366.315058]  RSP <ffff88000259bd48>
[  366.315059] CR2: 000000000000000c
[  366.315060] ---[ end trace 42d3248df75182f5 ]---
[  366.315418] Fixing recursive fault but reboot is needed!
[  366.315538] BUG: unable to handle kernel NULL pointer dereference at
0000000c
[  366.315580] IP: [<ffffffff810e86e7>] select_task_rq_fair+0x337/0x8c0
[  366.315609] PGD 0
[  366.315616] Oops: 0000 [#4] SMP
[  366.315620] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4
d
[  366.315660] CPU: 0 PID: 8220 Comm: udevd Tainted: G      D       3.15.6 #2
[  366.315666] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
GRNDSDP4
[  366.315673] task: ffff88017c680000 ti: ffff88007274c000 task.ti:
ffff88007270
[  366.315678] RIP: e030:[<ffffffff810e86e7>]  [<ffffffff810e86e7>]
select_task0
[  366.315687] RSP: e02b:ffff88007274fd48  EFLAGS: 00010046
[  366.315693] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000019
[  366.315699] RDX: 0000000000000008 RSI: 0000000000000040 RDI:
0000000000000040
[  366.315704] RBP: ffff88007274fe28 R08: ffff880003fb33f8 R09:
0000000000000000
[  366.315710] R10: 0000000000000000 R11: ffff88017cfe4338 R12:
0000000000000000
[  366.315715] R13: ffff880003fb33f8 R14: ffff880003fb33e0 R15:
0000000000000000
[  366.315724] FS:  00007fad200bb7a0(0000) GS:ffff880181a00000(0000)
knlGS:00000
[  366.315730] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[  366.315735] CR2: 000000000000000c CR3: 00000000725d7000 CR4:
0000000000042660
[  366.315740] Stack:
[  366.315743]  ffff88017c6e5000 00000000ffffff9c ffff88007274fe38
ffffffff811d5
[  366.315751]  0000000000013e00 0000000000013e00 ffff88017c6802f8
000000000000d
[  366.315759]  ffff880003fb3420 000000000000163e 0000000000013e00
0000000000018
[  366.315766] Call Trace:
[  366.315772]  [<ffffffff811d6a45>] ? do_filp_open+0x45/0xa0
[  366.315779]  [<ffffffff810dedd7>] sched_exec+0x47/0xc0
[  366.315787]  [<ffffffff811ccaca>] ? do_open_exec+0xaa/0xe0
[  366.315793]  [<ffffffff811cccee>] do_execve_common+0x1be/0x640
[  366.315801]  [<ffffffff811b5b77>] ? kmem_cache_alloc+0x37/0x120
[  366.315808]  [<ffffffff811cd202>] do_execve+0x32/0x40
[  366.315813]  [<ffffffff811cd23a>] SyS_execve+0x2a/0x40
[  366.315819]  [<ffffffff8170cba9>] stub_execve+0x69/0xa0
[  366.315824] Code: 48 8b 55 c0 4d 8b 36 4c 3b 72 10 74 43 48 89 45 b0 e9 6e f
[  366.315882] RIP  [<ffffffff810e86e7>] select_task_rq_fair+0x337/0x8c0
[  366.315890]  RSP <ffff88007274fd48>
[  366.315894] CR2: 000000000000000c
[  366.315899] ---[ end trace 42d3248df75182f6 ]---
[  366.317854] divide error: 0000 [#5] SMP
[  366.317869] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4
d
[  366.317967] CPU: 14 PID: 6370 Comm: rsyslogd Tainted: G      D       3.15.6
2
[  366.317982] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
GRNDSDP4
[  366.318002] task: ffff880003098000 ti: ffff88017c04c000 task.ti:
ffff88017c00
[  366.318017] RIP: e030:[<ffffffff810ea5f9>]  [<ffffffff810ea5f9>]
find_busies0
[  366.318040] RSP: e02b:ffff88017c04fa78  EFLAGS: 00010046
[  366.318052] RAX: 0000000000000000 RBX: ffff88017c04fb78 RCX:
0000000000000000
[  366.318067] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[  366.318082] RBP: ffff88017c04fbe8 R08: ffff880003fb3d00 R09:
0000000000000040
[  366.318096] R10: 0000000000000000 R11: 0000000000000293 R12:
0000000000013e00
[  366.318111] R13: ffff88017c04fc68 R14: ffff880003fb3ce0 R15:
0000000000000000
[  366.318135] FS:  00007f348d764700(0000) GS:ffff880181bc0000(0000)
knlGS:00000
[  366.318151] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[  366.318164] CR2: 00007fad200d5000 CR3: 0000000003271000 CR4:
0000000000042660
[  366.318179] Stack:
[  366.318186]  0000000000000001 ffff88017c04fac8 0000000000002e7b
00000000ffffc
[  366.318212]  ffff880003fb3ce0 0000000000013df8 0000000000000200
0000000000010
[  366.318236]  0000000085f9a800 ffff880003fb3cf8 0000000000000000
0000000000000
[  366.318258] Call Trace:
[  366.318274]  [<ffffffff810eae37>] load_balance+0x177/0x9d0
[  366.318290]  [<ffffffff810df56b>] ? update_rq_clock+0x2b/0x50
[  366.318306]  [<ffffffff81058ea0>] ? xen_clocksource_read+0x20/0x30
[  366.318323]  [<ffffffff810edb8d>] pick_next_task_fair+0x1ed/0x430
[  366.318342]  [<ffffffff816ff0d3>] __schedule+0x113/0x870
[  366.318357]  [<ffffffff81703ace>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0
[  366.318375]  [<ffffffff816ff9b4>] schedule+0x24/0x70
[  366.318391]  [<ffffffff810fbb1a>] do_syslog+0x4ba/0x640
[  366.318406]  [<ffffffff810f2d00>] ? bit_waitqueue+0xe0/0xe0
[  366.318424]  [<ffffffff81235ea2>] kmsg_read+0x32/0x70
[  366.318439]  [<ffffffff812294de>] proc_reg_read+0x3e/0x70
[  366.318454]  [<ffffffff811c7405>] vfs_read+0xa5/0x180
[  366.318469]  [<ffffffff811c75c1>] SyS_read+0x51/0xc0
[  366.318484]  [<ffffffff8170c5f9>] system_call_fastpath+0x16/0x1b
[  366.318496] Code: 0f 47 d1 eb 95 0f 1f 44 00 00 4d 89 ec 4d 89 f5 4c 8b b5 b
[  366.318699] RIP  [<ffffffff810ea5f9>] find_busiest_group+0x239/0x900
[  366.318718]  RSP <ffff88017c04fa78>
[  366.318728] ---[ end trace 42d3248df75182f7 ]---
Reviewed-by
#6 lore raw

Hi Wanpeng,

Quoted 33 lines · Wanpeng Li

Hi Yasuaki,

Quoted 31 lines · Yasuaki Ishimatsu
Quoted 29 lines · Wanpeng Li
[  220.262093] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
[  220.262104] IP: [<ffffffff810e7ac9>] find_busiest_group+0x2b9/0xa30
[  220.262111] PGD 5a9d5067 PUD 13067 PMD 0
[  220.262117] Oops: 0000 [#3] SMP
[...]
[  220.262245] Call Trace:
[  220.262252]  [<ffffffff810e8396>] load_balance+0x156/0x980
[  220.262259]  [<ffffffff816eeffe>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0
[  220.262266]  [<ffffffff810e9aa3>] idle_balance+0xe3/0x150
[  220.262270]  [<ffffffff816ec4e7>] __schedule+0x797/0x8d0
[  220.262277]  [<ffffffff816ec934>] schedule+0x24/0x70
[  220.262283]  [<ffffffff816e9cd9>] schedule_timeout+0x119/0x1f0
[  220.262294]  [<ffffffff810bb6e0>] ? lock_timer_base+0x70/0x70
[  220.262301]  [<ffffffff816e9dc9>] schedule_timeout_uninterruptible+0x19/0x20
[  220.262308]  [<ffffffff810bd3e8>] msleep+0x18/0x20
[  220.262317]  [<ffffffff813aa11a>] lock_device_hotplug_sysfs+0x2a/0x50
[  220.262323]  [<ffffffff813aa16e>] online_store+0x2e/0x80
[  220.262358]  [<ffffffff813a873b>] dev_attr_store+0x1b/0x20
[  220.262366]  [<ffffffff812292fd>] sysfs_write_file+0xdd/0x160
[  220.262377]  [<ffffffff811b7e78>] vfs_write+0xc8/0x170
[  220.262384]  [<ffffffff811b83ca>] SyS_write+0x5a/0xa0
[  220.262388]  [<ffffffff816f76b9>] system_call_fastpath+0x16/0x1b

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 7 lines
Quoted 4 lines

I posted a latest patch as follows:
https://lkml.org/lkml/2014/7/22/1018

Could you confirm the patch fixes your issue?

Sorry for the late, there is still call trace w/ your patch applied. The call trace is in attachment.

Thank you for reporting the result. As Borislav said, your v2 patch is necessary for fixing your issue.

Quoted 9 lines

Regards,
Wanpeng Li

Quoted 5 lines

Thanks,
Yasuaki Ishimatsu

Quoted 1 line

Thanks,
Yasuaki Ishimatsu

Quoted 23 lines
Quoted 23 lines
Quoted 22 lines

---
v1 -> v2:
* fix subject line

   arch/x86/kernel/smpboot.c | 3 +++
   1 file changed, 3 insertions(+)
arch/x86/kernel/smpboot.c +3 -0
index 5492798..0134ec7 100644
@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
 
   	for_each_cpu(sibling, cpu_sibling_mask(cpu))
   		cpumask_clear_cpu(cpu, cpu_sibling_mask(sibling));
+	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
+		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
+	cpumask_clear(cpu_llc_shared_mask(cpu));
   	cpumask_clear(cpu_sibling_mask(cpu));
   	cpumask_clear(cpu_core_mask(cpu));
   	c->phys_proc_id = 0;
maintainer
#7 lore raw
Quoted 10 lines · Yasuaki Ishimatsu
Quoted 7 lines
Quoted 4 lines

I posted a latest patch as follows:
https://lkml.org/lkml/2014/7/22/1018

Could you confirm the patch fixes your issue?

Sorry for the late, there is still call trace w/ your patch applied. The call trace is in attachment.

Thank you for reporting the result. As Borislav said, your v2 patch is necessary for fixing your issue.

..

Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Ok, Wanpeng, please send a cleaner version of your patch with a simplified backtrace in it - the addresses will make no sense anyway so you can do this:

Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
...

so that we can still see how it happens.

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
author
#8 lore raw
Quoted 22 lines · Borislav Petkov
Quoted 10 lines · Yasuaki Ishimatsu
Quoted 7 lines
Quoted 4 lines

I posted a latest patch as follows:
https://lkml.org/lkml/2014/7/22/1018

Could you confirm the patch fixes your issue?

Sorry for the late, there is still call trace w/ your patch applied. The call trace is in attachment.

Thank you for reporting the result. As Borislav said, your v2 patch is necessary for fixing your issue.

..

Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Ok, Wanpeng, please send a cleaner version of your patch with a simplified backtrace in it - the addresses will make no sense anyway so you can do this:

Quoted 6 lines
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
...

Just send out the new version.

Regards,
Wanpeng Li

Quoted 10 lines

so that we can still see how it happens.

Thanks.

Signature · 5 lines
Regards/Gruss,
   Boris.

Sent from a fat crate under my desk. Formatting is fine.
--