x86, hotplug: fix llc shared map unreleased during cpu hotplug
PATCH v2 1/1 x86, hotplug: fix llc shared map unreleased during cpu hotplug
[ 220.262093] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 220.262104] IP: [<ffffffff810e7ac9>] find_busiest_group+0x2b9/0xa30 [ 220.262111] PGD 5a9d5067 PUD 13067 PMD 0 [ 220.262117] Oops: 0000 [#3] SMP [...] [ 220.262245] Call Trace: [ 220.262252] [<ffffffff810e8396>] load_balance+0x156/0x980 [ 220.262259] [<ffffffff816eeffe>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0 [ 220.262266] [<ffffffff810e9aa3>] idle_balance+0xe3/0x150 [ 220.262270] [<ffffffff816ec4e7>] __schedule+0x797/0x8d0 [ 220.262277] [<ffffffff816ec934>] schedule+0x24/0x70 [ 220.262283] [<ffffffff816e9cd9>] schedule_timeout+0x119/0x1f0 [ 220.262294] [<ffffffff810bb6e0>] ? lock_timer_base+0x70/0x70 [ 220.262301] [<ffffffff816e9dc9>] schedule_timeout_uninterruptible+0x19/0x20 [ 220.262308] [<ffffffff810bd3e8>] msleep+0x18/0x20 [ 220.262317] [<ffffffff813aa11a>] lock_device_hotplug_sysfs+0x2a/0x50 [ 220.262323] [<ffffffff813aa16e>] online_store+0x2e/0x80 [ 220.262358] [<ffffffff813a873b>] dev_attr_store+0x1b/0x20 [ 220.262366] [<ffffffff812292fd>] sysfs_write_file+0xdd/0x160 [ 220.262377] [<ffffffff811b7e78>] vfs_write+0xc8/0x170 [ 220.262384] [<ffffffff811b83ca>] SyS_write+0x5a/0xa0 [ 220.262388] [<ffffffff816f76b9>] system_call_fastpath+0x16/0x1b
Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.
---
v1 -> v2:
* fix subject line
arch/x86/kernel/smpboot.c | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/kernel/smpboot.c +3 -0
@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)for_each_cpu (sibling,cpu_sibling_mask (cpu))cpumask_clear_cpu (cpu,cpu_sibling_mask (sibling)); +for_each_cpu (sibling,cpu_llc_shared_mask (cpu)) +cpumask_clear_cpu (cpu,cpu_llc_shared_mask (sibling)); +cpumask_clear (cpu_llc_shared_mask (cpu));cpumask_clear (cpu_sibling_mask (cpu));cpumask_clear (cpu_core_mask (cpu)); c->phys_proc_id =0 ;
Signature · 1 line
1.9.1
Quoted 2 lines · Wanpeng Li
Subject: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu
hotplug
See this link:
https://lkml.org/lkml/2014/7/17/78
Quoted 6 lines · Gong
Quoted 2 lines · Wanpeng Li
Subject: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu
hotplugSee this link:
https://lkml.org/lkml/2014/7/17/78
Interesting, thanks for your pointing out.
Regards,
Wanpeng Li
Borislav Petkov
maintainerQuoted 9 lines · Wanpeng Li
Quoted 6 lines · Gong
Quoted 2 lines · Wanpeng Li
Subject: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu
hotplugSee this link:
https://lkml.org/lkml/2014/7/17/78Interesting, thanks for your pointing out.
Well, we might end up doing both.
Yasuaki's patch makes the core numbering stable which makes a lot of sense in and of itself.
However, if some code somewhere relies implicitly on the fact that a set bit in LLC mask means that core is online and goes and does stuff on it, Wanpeng's reported splat will happen.
So probably we will have to have *stable* core numbers *and* clear the LLC mask in order to be on the safe side of things.
Thanks.
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Quoted 23 lines · Wanpeng Li
[ 220.262093] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 220.262104] IP: [<ffffffff810e7ac9>] find_busiest_group+0x2b9/0xa30 [ 220.262111] PGD 5a9d5067 PUD 13067 PMD 0 [ 220.262117] Oops: 0000 [#3] SMP [...] [ 220.262245] Call Trace: [ 220.262252] [<ffffffff810e8396>] load_balance+0x156/0x980 [ 220.262259] [<ffffffff816eeffe>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0 [ 220.262266] [<ffffffff810e9aa3>] idle_balance+0xe3/0x150 [ 220.262270] [<ffffffff816ec4e7>] __schedule+0x797/0x8d0 [ 220.262277] [<ffffffff816ec934>] schedule+0x24/0x70 [ 220.262283] [<ffffffff816e9cd9>] schedule_timeout+0x119/0x1f0 [ 220.262294] [<ffffffff810bb6e0>] ? lock_timer_base+0x70/0x70 [ 220.262301] [<ffffffff816e9dc9>] schedule_timeout_uninterruptible+0x19/0x20 [ 220.262308] [<ffffffff810bd3e8>] msleep+0x18/0x20 [ 220.262317] [<ffffffff813aa11a>] lock_device_hotplug_sysfs+0x2a/0x50 [ 220.262323] [<ffffffff813aa16e>] online_store+0x2e/0x80 [ 220.262358] [<ffffffff813a873b>] dev_attr_store+0x1b/0x20 [ 220.262366] [<ffffffff812292fd>] sysfs_write_file+0xdd/0x160 [ 220.262377] [<ffffffff811b7e78>] vfs_write+0xc8/0x170 [ 220.262384] [<ffffffff811b83ca>] SyS_write+0x5a/0xa0 [ 220.262388] [<ffffffff816f76b9>] system_call_fastpath+0x16/0x1b
Quoted 6 lines
Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.
I posted a latest patch as follows:
https://lkml.org/lkml/2014/7/22/1018
Could you confirm the patch fixes your issue?
Thanks,
Yasuaki Ishimatsu
Quoted 23 lines
Signed-off-by Wanpeng Li <[email protected]>---
v1 -> v2:
* fix subject linearch/x86/kernel/smpboot.c | 3 +++ 1 file changed, 3 insertions(+)arch/x86/kernel/smpboot.c +3
-0@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)for_each_cpu (sibling,cpu_sibling_mask (cpu))cpumask_clear_cpu (cpu,cpu_sibling_mask (sibling)); +for_each_cpu (sibling,cpu_llc_shared_mask (cpu)) +cpumask_clear_cpu (cpu,cpu_llc_shared_mask (sibling)); +cpumask_clear (cpu_llc_shared_mask (cpu));cpumask_clear (cpu_sibling_mask (cpu));cpumask_clear (cpu_core_mask (cpu)); c->phys_proc_id =0 ;
Hi Yasuaki,
Quoted 31 lines · Yasuaki Ishimatsu
Quoted 23 lines · Wanpeng Li
[ 220.262093] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 220.262104] IP: [<ffffffff810e7ac9>] find_busiest_group+0x2b9/0xa30 [ 220.262111] PGD 5a9d5067 PUD 13067 PMD 0 [ 220.262117] Oops: 0000 [#3] SMP [...] [ 220.262245] Call Trace: [ 220.262252] [<ffffffff810e8396>] load_balance+0x156/0x980 [ 220.262259] [<ffffffff816eeffe>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0 [ 220.262266] [<ffffffff810e9aa3>] idle_balance+0xe3/0x150 [ 220.262270] [<ffffffff816ec4e7>] __schedule+0x797/0x8d0 [ 220.262277] [<ffffffff816ec934>] schedule+0x24/0x70 [ 220.262283] [<ffffffff816e9cd9>] schedule_timeout+0x119/0x1f0 [ 220.262294] [<ffffffff810bb6e0>] ? lock_timer_base+0x70/0x70 [ 220.262301] [<ffffffff816e9dc9>] schedule_timeout_uninterruptible+0x19/0x20 [ 220.262308] [<ffffffff810bd3e8>] msleep+0x18/0x20 [ 220.262317] [<ffffffff813aa11a>] lock_device_hotplug_sysfs+0x2a/0x50 [ 220.262323] [<ffffffff813aa16e>] online_store+0x2e/0x80 [ 220.262358] [<ffffffff813a873b>] dev_attr_store+0x1b/0x20 [ 220.262366] [<ffffffff812292fd>] sysfs_write_file+0xdd/0x160 [ 220.262377] [<ffffffff811b7e78>] vfs_write+0xc8/0x170 [ 220.262384] [<ffffffff811b83ca>] SyS_write+0x5a/0xa0 [ 220.262388] [<ffffffff816f76b9>] system_call_fastpath+0x16/0x1bQuoted 6 lines
Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.
Quoted 4 lines
I posted a latest patch as follows:
https://lkml.org/lkml/2014/7/22/1018Could you confirm the patch fixes your issue?
Sorry for the late, there is still call trace w/ your patch applied. The call trace is in attachment.
Regards,
Wanpeng Li
Quoted 18 lines
Thanks,
Yasuaki IshimatsuQuoted 14 lines
Signed-off-by Wanpeng Li <[email protected]>---
v1 -> v2:
* fix subject linearch/x86/kernel/smpboot.c | 3 +++ 1 file changed, 3 insertions(+)arch/x86/kernel/smpboot.c +0
-0@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
Quoted 10 lines
arch/x86/kernel/smpboot.c static void remove_siblinginfo(int cpu) · line 1300
Quoted 9 lines
arch/x86/kernel/smpboot.c +3
-0for_each_cpu (sibling,cpu_sibling_mask (cpu))cpumask_clear_cpu (cpu,cpu_sibling_mask (sibling)); +for_each_cpu (sibling,cpu_llc_shared_mask (cpu)) +cpumask_clear_cpu (cpu,cpu_llc_shared_mask (sibling)); +cpumask_clear (cpu_llc_shared_mask (cpu));cpumask_clear (cpu_sibling_mask (cpu));cpumask_clear (cpu_core_mask (cpu)); c->phys_proc_id =0 ;
when run "xl vcpu-set 0 2", the dom0 only report "broke affinity ..." when run "xl vcpu-set 0 26", the call trace happens.
the dom0 call trace log as following:
[ 295.464489] Broke affinity for irq 298 [ 295.756205] Broke affinity for irq 299 [ 295.767177] Broke affinity for irq 301 [ 295.779177] Broke affinity for irq 303 [ 366.283682] installing Xen timer for CPU 2 [ 366.283749] cpu 2 spinlock event irq 103 [ 366.310290] installing Xen timer for CPU 14 [ 366.310347] cpu 14 spinlock event irq 110 [ 366.312432] divide error: 0000 [#1] SMP [ 366.312449] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4 d [ 366.312583] CPU: 14 PID: 63 Comm: ksoftirqd/14 Not tainted 3.15.6 #2 [ 366.312598] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRNDSDP4 [ 366.312623] task: ffff88017c8d2c10 ti: ffff88017c8f0000 task.ti: ffff88017c80 [ 366.312647] RIP: e030:[<ffffffff810ea5f9>] [<ffffffff810ea5f9>] find_busies0 [ 366.312681] RSP: e02b:ffff88017c8f3ac8 EFLAGS: 00010046 [ 366.312694] RAX: 0000000000000000 RBX: ffff88017c8f3bc8 RCX: 0000000000000000 [ 366.312708] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 366.312724] RBP: ffff88017c8f3c38 R08: ffff880003fb3d00 R09: 0000000000000040 [ 366.312742] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000013e00 [ 366.312757] R13: ffff88017c8f3cb8 R14: ffff880003fb3ce0 R15: 0000000000000000 [ 366.312783] FS: 0000000000000000(0000) GS:ffff880181bc0000(0000) knlGS:00000 [ 366.312803] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 366.312817] CR2: 00007fad200d5000 CR3: 0000000001c14000 CR4: 0000000000042660 [ 366.312836] Stack: [ 366.312843] 0000000000000000 ffff88017c8f3b18 0000000000002e7b 0000000000000 [ 366.312868] ffff880003fb3ce0 0000000000013df8 0000000000000200 0000000000010 [ 366.312890] 0000000000000000 ffff880003fb3cf8 0000000000000000 0000000000000 [ 366.312911] Call Trace: [ 366.312932] [<ffffffff810eae37>] load_balance+0x177/0x9d0 [ 366.312954] [<ffffffff810df56b>] ? update_rq_clock+0x2b/0x50 [ 366.312976] [<ffffffff81058ea0>] ? xen_clocksource_read+0x20/0x30 [ 366.312997] [<ffffffff810edb8d>] pick_next_task_fair+0x1ed/0x430 [ 366.313019] [<ffffffff816ff0d3>] __schedule+0x113/0x870 [ 366.313039] [<ffffffff816ff9b4>] ? schedule+0x24/0x70 [ 366.313059] [<ffffffff816ff9b4>] schedule+0x24/0x70 [ 366.313095] [<ffffffff810dcd7c>] smpboot_thread_fn+0xbc/0x190 [ 366.313112] [<ffffffff810dccc0>] ? smpboot_create_threads+0x80/0x80 [ 366.313135] [<ffffffff810d565e>] kthread+0xce/0xf0 [ 366.313155] [<ffffffff810d5590>] ? kthread_freezable_should_stop+0x70/0x70 [ 366.313174] [<ffffffff8170c54c>] ret_from_fork+0x7c/0xb0 [ 366.313190] [<ffffffff810d5590>] ? kthread_freezable_should_stop+0x70/0x70 [ 366.313204] Code: 0f 47 d1 eb 95 0f 1f 44 00 00 4d 89 ec 4d 89 f5 4c 8b b5 b [ 366.313372] RIP [<ffffffff810ea5f9>] find_busiest_group+0x239/0x900 [ 366.313391] RSP <ffff88017c8f3ac8> [ 366.313406] ---[ end trace 42d3248df75182f3 ]--- [ 366.313758] divide error: 0000 [#2] SMP [ 366.313776] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4 d [ 366.313883] CPU: 14 PID: 63 Comm: ksoftirqd/14 Tainted: G D 3.15.2 [ 366.313898] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRNDSDP4 [ 366.313922] task: ffff88017c8d2c10 ti: ffff88017c8f0000 task.ti: ffff88017c80 [ 366.313940] RIP: e030:[<ffffffff810ea5f9>] [<ffffffff810ea5f9>] find_busies0 [ 366.313966] RSP: e02b:ffff88017c8f3468 EFLAGS: 00010046 [ 366.313979] RAX: 0000000000000000 RBX: ffff88017c8f3568 RCX: 0000000000000000 [ 366.313993] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 366.314008] RBP: ffff88017c8f35d8 R08: ffff880003fb3d00 R09: 0000000000000040 [ 366.314023] R10: 0000000000000000 R11: ffff880186148410 R12: 0000000000013e00 [ 366.314042] R13: ffff88017c8f3658 R14: ffff880003fb3ce0 R15: 0000000000000000 [ 366.314067] FS: 0000000000000000(0000) GS:ffff880181bc0000(0000) knlGS:00000 [ 366.314090] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 366.314103] CR2: 00007fad200d5000 CR3: 0000000001c14000 CR4: 0000000000042660 [ 366.314119] Stack: [ 366.314125] ffff8801fc8f35c9 ffff88017c8f34b8 0000000000002e7b 00000000812ce [ 366.314150] ffff880003fb3ce0 0000000000013df8 0000000000000200 0000000000010 [ 366.314175] 000000006c106009 ffff880003fb3cf8 0000000000000000 0000000000000 [ 366.314196] Call Trace: [ 366.314215] [<ffffffff810eae37>] load_balance+0x177/0x9d0 [ 366.314232] [<ffffffff810df56b>] ? update_rq_clock+0x2b/0x50 [ 366.314252] [<ffffffff81058ea0>] ? xen_clocksource_read+0x20/0x30 [ 366.314269] [<ffffffff810edb8d>] pick_next_task_fair+0x1ed/0x430 [ 366.314288] [<ffffffff816ff0d3>] __schedule+0x113/0x870 [ 366.314307] [<ffffffff810b3a34>] ? release_task+0x304/0x480 [ 366.314324] [<ffffffff816ff9b4>] schedule+0x24/0x70 [ 366.314340] [<ffffffff810b42ac>] do_exit+0x6fc/0xac0 [ 366.314356] [<ffffffff81705008>] oops_end+0xa8/0x170 [ 366.314371] [<ffffffff810663b6>] die+0x56/0x90 [ 366.314385] [<ffffffff81704a23>] do_trap+0xc3/0x170 [ 366.314402] [<ffffffff8170812d>] ? __atomic_notifier_call_chain+0xd/0x10 [ 366.314422] [<ffffffff8106395b>] do_divide_error+0x9b/0xb0 [ 366.314439] [<ffffffff810ea5f9>] ? find_busiest_group+0x239/0x900 [ 366.314456] [<ffffffff8170dc0e>] divide_error+0x1e/0x30 [ 366.314473] [<ffffffff810ea5f9>] ? find_busiest_group+0x239/0x900 [ 366.314491] [<ffffffff810ea513>] ? find_busiest_group+0x153/0x900 [ 366.314511] [<ffffffff810eae37>] load_balance+0x177/0x9d0 [ 366.314526] [<ffffffff810df56b>] ? update_rq_clock+0x2b/0x50 [ 366.314547] [<ffffffff81058ea0>] ? xen_clocksource_read+0x20/0x30 [ 366.314563] [<ffffffff810edb8d>] pick_next_task_fair+0x1ed/0x430 [ 366.314581] [<ffffffff816ff0d3>] __schedule+0x113/0x870 [ 366.314597] [<ffffffff816ff9b4>] ? schedule+0x24/0x70 [ 366.314613] [<ffffffff816ff9b4>] schedule+0x24/0x70 [ 366.314628] [<ffffffff810dcd7c>] smpboot_thread_fn+0xbc/0x190 [ 366.314650] [<ffffffff810dccc0>] ? smpboot_create_threads+0x80/0x80 [ 366.314668] [<ffffffff810d565e>] kthread+0xce/0xf0 [ 366.314684] [<ffffffff810d5590>] ? kthread_freezable_should_stop+0x70/0x70 [ 366.314701] [<ffffffff8170c54c>] ret_from_fork+0x7c/0xb0 [ 366.314717] [<ffffffff810d5590>] ? kthread_freezable_should_stop+0x70/0x70 [ 366.314735] Code: 0f 47 d1 eb 95 0f 1f 44 00 00 4d 89 ec 4d 89 f5 4c 8b b5 b [ 366.314891] RIP [<ffffffff810ea5f9>] find_busiest_group+0x239/0x900 [ 366.314909] RSP <ffff88017c8f3468> [ 366.314927] ---[ end trace 42d3248df75182f4 ]--- [ 366.314932] BUG: unable to handle kernel NULL pointer dereference at 0000000c [ 366.314938] IP: [<ffffffff810e86e7>] select_task_rq_fair+0x337/0x8c0 [ 366.314942] PGD 0 [ 366.314943] Oops: 0000 [#3] SMP [ 366.314960] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4 d [ 366.314962] CPU: 1 PID: 8225 Comm: udevd Tainted: G D 3.15.6 #2 [ 366.314965] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRNDSDP4 [ 366.314966] task: ffff8801771634e0 ti: ffff880002598000 task.ti: ffff88000250 [ 366.314972] RIP: e030:[<ffffffff810e86e7>] [<ffffffff810e86e7>] select_task0 [ 366.314973] RSP: e02b:ffff88000259bd48 EFLAGS: 00010046 [ 366.314974] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000019 [ 366.314975] RDX: 0000000000000008 RSI: 0000000000000040 RDI: 0000000000000040 [ 366.314979] RBP: ffff88000259be28 R08: ffff880003fb33f8 R09: 0000000000000000 [ 366.314980] R10: 0000000000000000 R11: ffff88017cfe4338 R12: 0000000000000000 [ 366.314981] R13: ffff880003fb33f8 R14: ffff880003fb33e0 R15: 0000000000000000 [ 366.314988] FS: 00007fad200bb7a0(0000) GS:ffff880181a20000(0000) knlGS:00000 [ 366.314989] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 366.314990] CR2: 000000000000000c CR3: 00000000030f2000 CR4: 0000000000042660 [ 366.314991] Stack: [ 366.314993] ffff88017c7de000 00000000ffffff9c ffff88000259be38 ffffffff811d5 [ 366.314995] 0000000000013e00 0000000000013e00 ffff8801771637d8 000000000000d [ 366.314997] ffff880003fb3420 0000000000001ade 0000000000013e00 0000000000018 [ 366.314998] Call Trace: [ 366.315003] [<ffffffff811d6a45>] ? do_filp_open+0x45/0xa0 [ 366.315005] [<ffffffff810dedd7>] sched_exec+0x47/0xc0 [ 366.315009] [<ffffffff811ccaca>] ? do_open_exec+0xaa/0xe0 [ 366.315014] [<ffffffff811cccee>] do_execve_common+0x1be/0x640 [ 366.315019] [<ffffffff811b5b77>] ? kmem_cache_alloc+0x37/0x120 [ 366.315021] [<ffffffff811cd202>] do_execve+0x32/0x40 [ 366.315026] [<ffffffff811cd23a>] SyS_execve+0x2a/0x40 [ 366.315029] [<ffffffff8170cba9>] stub_execve+0x69/0xa0 [ 366.315055] Code: 48 8b 55 c0 4d 8b 36 4c 3b 72 10 74 43 48 89 45 b0 e9 6e f [ 366.315058] RIP [<ffffffff810e86e7>] select_task_rq_fair+0x337/0x8c0 [ 366.315058] RSP <ffff88000259bd48> [ 366.315059] CR2: 000000000000000c [ 366.315060] ---[ end trace 42d3248df75182f5 ]--- [ 366.315418] Fixing recursive fault but reboot is needed! [ 366.315538] BUG: unable to handle kernel NULL pointer dereference at 0000000c [ 366.315580] IP: [<ffffffff810e86e7>] select_task_rq_fair+0x337/0x8c0 [ 366.315609] PGD 0 [ 366.315616] Oops: 0000 [#4] SMP [ 366.315620] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4 d [ 366.315660] CPU: 0 PID: 8220 Comm: udevd Tainted: G D 3.15.6 #2 [ 366.315666] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRNDSDP4 [ 366.315673] task: ffff88017c680000 ti: ffff88007274c000 task.ti: ffff88007270 [ 366.315678] RIP: e030:[<ffffffff810e86e7>] [<ffffffff810e86e7>] select_task0 [ 366.315687] RSP: e02b:ffff88007274fd48 EFLAGS: 00010046 [ 366.315693] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000019 [ 366.315699] RDX: 0000000000000008 RSI: 0000000000000040 RDI: 0000000000000040 [ 366.315704] RBP: ffff88007274fe28 R08: ffff880003fb33f8 R09: 0000000000000000 [ 366.315710] R10: 0000000000000000 R11: ffff88017cfe4338 R12: 0000000000000000 [ 366.315715] R13: ffff880003fb33f8 R14: ffff880003fb33e0 R15: 0000000000000000 [ 366.315724] FS: 00007fad200bb7a0(0000) GS:ffff880181a00000(0000) knlGS:00000 [ 366.315730] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 366.315735] CR2: 000000000000000c CR3: 00000000725d7000 CR4: 0000000000042660 [ 366.315740] Stack: [ 366.315743] ffff88017c6e5000 00000000ffffff9c ffff88007274fe38 ffffffff811d5 [ 366.315751] 0000000000013e00 0000000000013e00 ffff88017c6802f8 000000000000d [ 366.315759] ffff880003fb3420 000000000000163e 0000000000013e00 0000000000018 [ 366.315766] Call Trace: [ 366.315772] [<ffffffff811d6a45>] ? do_filp_open+0x45/0xa0 [ 366.315779] [<ffffffff810dedd7>] sched_exec+0x47/0xc0 [ 366.315787] [<ffffffff811ccaca>] ? do_open_exec+0xaa/0xe0 [ 366.315793] [<ffffffff811cccee>] do_execve_common+0x1be/0x640 [ 366.315801] [<ffffffff811b5b77>] ? kmem_cache_alloc+0x37/0x120 [ 366.315808] [<ffffffff811cd202>] do_execve+0x32/0x40 [ 366.315813] [<ffffffff811cd23a>] SyS_execve+0x2a/0x40 [ 366.315819] [<ffffffff8170cba9>] stub_execve+0x69/0xa0 [ 366.315824] Code: 48 8b 55 c0 4d 8b 36 4c 3b 72 10 74 43 48 89 45 b0 e9 6e f [ 366.315882] RIP [<ffffffff810e86e7>] select_task_rq_fair+0x337/0x8c0 [ 366.315890] RSP <ffff88007274fd48> [ 366.315894] CR2: 000000000000000c [ 366.315899] ---[ end trace 42d3248df75182f6 ]--- [ 366.317854] divide error: 0000 [#5] SMP [ 366.317869] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4 d [ 366.317967] CPU: 14 PID: 6370 Comm: rsyslogd Tainted: G D 3.15.6 2 [ 366.317982] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRNDSDP4 [ 366.318002] task: ffff880003098000 ti: ffff88017c04c000 task.ti: ffff88017c00 [ 366.318017] RIP: e030:[<ffffffff810ea5f9>] [<ffffffff810ea5f9>] find_busies0 [ 366.318040] RSP: e02b:ffff88017c04fa78 EFLAGS: 00010046 [ 366.318052] RAX: 0000000000000000 RBX: ffff88017c04fb78 RCX: 0000000000000000 [ 366.318067] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 366.318082] RBP: ffff88017c04fbe8 R08: ffff880003fb3d00 R09: 0000000000000040 [ 366.318096] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000013e00 [ 366.318111] R13: ffff88017c04fc68 R14: ffff880003fb3ce0 R15: 0000000000000000 [ 366.318135] FS: 00007f348d764700(0000) GS:ffff880181bc0000(0000) knlGS:00000 [ 366.318151] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 366.318164] CR2: 00007fad200d5000 CR3: 0000000003271000 CR4: 0000000000042660 [ 366.318179] Stack: [ 366.318186] 0000000000000001 ffff88017c04fac8 0000000000002e7b 00000000ffffc [ 366.318212] ffff880003fb3ce0 0000000000013df8 0000000000000200 0000000000010 [ 366.318236] 0000000085f9a800 ffff880003fb3cf8 0000000000000000 0000000000000 [ 366.318258] Call Trace: [ 366.318274] [<ffffffff810eae37>] load_balance+0x177/0x9d0 [ 366.318290] [<ffffffff810df56b>] ? update_rq_clock+0x2b/0x50 [ 366.318306] [<ffffffff81058ea0>] ? xen_clocksource_read+0x20/0x30 [ 366.318323] [<ffffffff810edb8d>] pick_next_task_fair+0x1ed/0x430 [ 366.318342] [<ffffffff816ff0d3>] __schedule+0x113/0x870 [ 366.318357] [<ffffffff81703ace>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0 [ 366.318375] [<ffffffff816ff9b4>] schedule+0x24/0x70 [ 366.318391] [<ffffffff810fbb1a>] do_syslog+0x4ba/0x640 [ 366.318406] [<ffffffff810f2d00>] ? bit_waitqueue+0xe0/0xe0 [ 366.318424] [<ffffffff81235ea2>] kmsg_read+0x32/0x70 [ 366.318439] [<ffffffff812294de>] proc_reg_read+0x3e/0x70 [ 366.318454] [<ffffffff811c7405>] vfs_read+0xa5/0x180 [ 366.318469] [<ffffffff811c75c1>] SyS_read+0x51/0xc0 [ 366.318484] [<ffffffff8170c5f9>] system_call_fastpath+0x16/0x1b [ 366.318496] Code: 0f 47 d1 eb 95 0f 1f 44 00 00 4d 89 ec 4d 89 f5 4c 8b b5 b [ 366.318699] RIP [<ffffffff810ea5f9>] find_busiest_group+0x239/0x900 [ 366.318718] RSP <ffff88017c04fa78> [ 366.318728] ---[ end trace 42d3248df75182f7 ]---
Yasuaki Ishimatsu
Reviewed-byHi Wanpeng,
Quoted 33 lines · Wanpeng Li
Hi Yasuaki,
Quoted 31 lines · Yasuaki Ishimatsu
Quoted 29 lines · Wanpeng Li
[ 220.262093] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 220.262104] IP: [<ffffffff810e7ac9>] find_busiest_group+0x2b9/0xa30 [ 220.262111] PGD 5a9d5067 PUD 13067 PMD 0 [ 220.262117] Oops: 0000 [#3] SMP [...] [ 220.262245] Call Trace: [ 220.262252] [<ffffffff810e8396>] load_balance+0x156/0x980 [ 220.262259] [<ffffffff816eeffe>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0 [ 220.262266] [<ffffffff810e9aa3>] idle_balance+0xe3/0x150 [ 220.262270] [<ffffffff816ec4e7>] __schedule+0x797/0x8d0 [ 220.262277] [<ffffffff816ec934>] schedule+0x24/0x70 [ 220.262283] [<ffffffff816e9cd9>] schedule_timeout+0x119/0x1f0 [ 220.262294] [<ffffffff810bb6e0>] ? lock_timer_base+0x70/0x70 [ 220.262301] [<ffffffff816e9dc9>] schedule_timeout_uninterruptible+0x19/0x20 [ 220.262308] [<ffffffff810bd3e8>] msleep+0x18/0x20 [ 220.262317] [<ffffffff813aa11a>] lock_device_hotplug_sysfs+0x2a/0x50 [ 220.262323] [<ffffffff813aa16e>] online_store+0x2e/0x80 [ 220.262358] [<ffffffff813a873b>] dev_attr_store+0x1b/0x20 [ 220.262366] [<ffffffff812292fd>] sysfs_write_file+0xdd/0x160 [ 220.262377] [<ffffffff811b7e78>] vfs_write+0xc8/0x170 [ 220.262384] [<ffffffff811b83ca>] SyS_write+0x5a/0xa0 [ 220.262388] [<ffffffff816f76b9>] system_call_fastpath+0x16/0x1bLast level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.
Quoted 7 lines
Quoted 4 lines
I posted a latest patch as follows:
https://lkml.org/lkml/2014/7/22/1018Could you confirm the patch fixes your issue?
Sorry for the late, there is still call trace w/ your patch applied. The call trace is in attachment.
Thank you for reporting the result. As Borislav said, your v2 patch is necessary for fixing your issue.
Quoted 9 lines
Regards,
Wanpeng LiQuoted 5 lines
Thanks,
Yasuaki IshimatsuQuoted 1 line
Signed-off-by Wanpeng Li <[email protected]>
Thanks,
Yasuaki Ishimatsu
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
---
v1 -> v2:
* fix subject linearch/x86/kernel/smpboot.c | 3 +++ 1 file changed, 3 insertions(+)arch/x86/kernel/smpboot.c +3
-0@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)for_each_cpu (sibling,cpu_sibling_mask (cpu))cpumask_clear_cpu (cpu,cpu_sibling_mask (sibling)); +for_each_cpu (sibling,cpu_llc_shared_mask (cpu)) +cpumask_clear_cpu (cpu,cpu_llc_shared_mask (sibling)); +cpumask_clear (cpu_llc_shared_mask (cpu));cpumask_clear (cpu_sibling_mask (cpu));cpumask_clear (cpu_core_mask (cpu)); c->phys_proc_id =0 ;
Borislav Petkov
maintainerQuoted 10 lines · Yasuaki Ishimatsu
Quoted 7 lines
Quoted 4 lines
I posted a latest patch as follows:
https://lkml.org/lkml/2014/7/22/1018Could you confirm the patch fixes your issue?
Sorry for the late, there is still call trace w/ your patch applied. The call trace is in attachment.
Thank you for reporting the result. As Borislav said, your v2 patch is necessary for fixing your issue.
..
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Signed-off-by Wanpeng Li <[email protected]>Reviewed-by Yasuaki Ishimatsu <[email protected]>
Ok, Wanpeng, please send a cleaner version of your patch with a simplified backtrace in it - the addresses will make no sense anyway so you can do this:
Call Trace:
load_balance
? _raw_spin_unlock_irqrestoreidle_balance
...
so that we can still see how it happens.
Thanks.
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Quoted 22 lines · Borislav Petkov
Quoted 10 lines · Yasuaki Ishimatsu
Quoted 7 lines
Quoted 4 lines
I posted a latest patch as follows:
https://lkml.org/lkml/2014/7/22/1018Could you confirm the patch fixes your issue?
Sorry for the late, there is still call trace w/ your patch applied. The call trace is in attachment.
Thank you for reporting the result. As Borislav said, your v2 patch is necessary for fixing your issue.
..
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Signed-off-by Wanpeng Li <[email protected]>Reviewed-by Yasuaki Ishimatsu <[email protected]>Ok, Wanpeng, please send a cleaner version of your patch with a simplified backtrace in it - the addresses will make no sense anyway so you can do this:
Quoted 6 lines
Call Trace:
load_balance
?_raw_spin_unlock_irqrestoreidle_balance
...
Just send out the new version.
Regards,
Wanpeng Li
Quoted 10 lines
so that we can still see how it happens.
Thanks.
Signature · 5 lines
Regards/Gruss, Boris. Sent from a fat crate under my desk. Formatting is fine. --