Skip to contentSkip to conversation

x86, cpu-hotplug: fix llc shared map unreleased during cpu hotplug

PATCH Superseded Wanpeng Li 5 participants 9 messages Sep 17, 2014 – Sep 24, 2014 x86

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Wanpeng Li

author
#0 lore raw

PATCH v5 1/1 x86, cpu-hotplug: fix llc shared map unreleased during cpu hotplug

BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeatedly.

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

---
v4 -> v5:
* add the description when the bug can occur
v3 -> v4:
* simplify backtrace
v2 -> v3:
* simplify backtrace
v1 -> v2:
* fix subject line

 arch/x86/kernel/smpboot.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kernel/smpboot.c +3 -0
index 5492798..0134ec7 100644
@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
 
 	for_each_cpu(sibling, cpu_sibling_mask(cpu))
 		cpumask_clear_cpu(cpu, cpu_sibling_mask(sibling));
+	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
+		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
+	cpumask_clear(cpu_llc_shared_mask(cpu));
 	cpumask_clear(cpu_sibling_mask(cpu));
 	cpumask_clear(cpu_core_mask(cpu));
 	c->phys_proc_id = 0;
Signature · 1 line
1.9.1
#1 lore raw

Ping Ingo, Peter Z, HPA,

Quoted 63 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeatedly.

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

---
v4 -> v5:
* add the description when the bug can occur
v3 -> v4:
* simplify backtrace
v2 -> v3:
* simplify backtrace
v1 -> v2:
* fix subject line

 arch/x86/kernel/smpboot.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kernel/smpboot.c +3 -0
index 5492798..0134ec7 100644
@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
 
 	for_each_cpu(sibling, cpu_sibling_mask(cpu))
 		cpumask_clear_cpu(cpu, cpu_sibling_mask(sibling));
+	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
+		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
+	cpumask_clear(cpu_llc_shared_mask(cpu));
 	cpumask_clear(cpu_sibling_mask(cpu));
 	cpumask_clear(cpu_core_mask(cpu));
 	c->phys_proc_id = 0;
#2 lore raw
In reply to Wanpeng Li
Quoted 32 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeatedly.

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 4 lines

Yasuaki reported this can happen on our real hardware.
https://lkml.org/lkml/2014/7/22/1018

Our case is here.
==
Here is a example on my system.
My system has 4 sockets and each socket has 15 cores and HT is enabled. In this case, each core of sockes is numbered as follows:

          | CPU#
Socket#0 | 0-14 , 60-74
Socket#1 | 15-29, 75-89
Socket#2 | 30-44, 90-104
Socket#3 | 45-59, 105-119

Then llc_shared_mask of CPU#30 has 0x3fff80000001fffc0000000. It means that last level cache of Socket#2 is shared with CPU#30-44 and 90-104.
When hot-removing socket#2 and #3, each core of sockets is numbered as follows:

          | CPU#

Socket#0 | 0-14 , 60-74
Socket#1 | 15-29, 75-89
But llc_shared_mask is not cleared. So llc_shared_mask of CPU#30 remains having 0x3fff80000001fffc0000000.
After that, when hot-adding socket#2 and #3, each core of sockets is numbered as follows:

          | CPU#
Socket#0 | 0-14 , 60-74
Socket#1 | 15-29, 75-89
Socket#2 | 30-59
Socket#3 | 90-119

Then llc_shared_mask of CPU#30 becomes 0x3fff8000fffffffc0000000. It means that last level cache of Socket#2 is shared with CPU#30-59 and 90-104. So the mask has wrong value.
At first, I cleared hot-removed CPU number's bit from llc_shared_map when hot removing CPU. But Borislav suggested that the problem will disappear if readded CPU is assigned same CPU number. And llc_shared_map must not be changed.
==

So, please.

Thanks,
-Kame

#3 lore raw

Hi Kamezawa,

Quoted 79 lines · Kamezawa Hiroyuki
Quoted 32 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeatedly.

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 4 lines

Yasuaki reported this can happen on our real hardware.
https://lkml.org/lkml/2014/7/22/1018

Our case is here.
==
Here is a example on my system.
My system has 4 sockets and each socket has 15 cores and HT is enabled. In this case, each core of sockes is numbered as follows:

          | CPU#
Socket#0 | 0-14 , 60-74
Socket#1 | 15-29, 75-89
Socket#2 | 30-44, 90-104
Socket#3 | 45-59, 105-119

Then llc_shared_mask of CPU#30 has 0x3fff80000001fffc0000000. It means that last level cache of Socket#2 is shared with CPU#30-44 and 90-104.
When hot-removing socket#2 and #3, each core of sockets is numbered as follows:

          | CPU#

Socket#0 | 0-14 , 60-74
Socket#1 | 15-29, 75-89
But llc_shared_mask is not cleared. So llc_shared_mask of CPU#30 remains having 0x3fff80000001fffc0000000.
After that, when hot-adding socket#2 and #3, each core of sockets is numbered as follows:

          | CPU#
Socket#0 | 0-14 , 60-74
Socket#1 | 15-29, 75-89
Socket#2 | 30-59
Socket#3 | 90-119

Then llc_shared_mask of CPU#30 becomes 0x3fff8000fffffffc0000000. It means that last level cache of Socket#2 is shared with CPU#30-59 and 90-104. So the mask has wrong value.
At first, I cleared hot-removed CPU number's bit from llc_shared_map when hot removing CPU. But Borislav suggested that the problem will disappear if readded CPU is assigned same CPU number. And llc_shared_map must not be changed.
==

Quoted 1 line

So, please.

As I mentioned before, we still observe calltrace after Yasuaki's patch applied.
https://lkml.org/lkml/2014/7/29/40

Actually I prefer to merge both patches, one for fix llc shared map unreleased during hotplug and the other one for assign same CPU number to readded CPU.

Regards,
Wanpeng Li

Quoted 14 lines

Thanks,
-Kame

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#4 lore raw
Quoted 81 lines · Wanpeng Li

Hi Kamezawa,

Quoted 79 lines · Kamezawa Hiroyuki
Quoted 32 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeatedly.

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 4 lines

Yasuaki reported this can happen on our real hardware.
https://lkml.org/lkml/2014/7/22/1018

Our case is here.
==
Here is a example on my system.
My system has 4 sockets and each socket has 15 cores and HT is enabled. In this case, each core of sockes is numbered as follows:

           | CPU#
Socket#0 | 0-14 , 60-74
Socket#1 | 15-29, 75-89
Socket#2 | 30-44, 90-104
Socket#3 | 45-59, 105-119

Then llc_shared_mask of CPU#30 has 0x3fff80000001fffc0000000. It means that last level cache of Socket#2 is shared with CPU#30-44 and 90-104.
When hot-removing socket#2 and #3, each core of sockets is numbered as follows:

           | CPU#

Socket#0 | 0-14 , 60-74
Socket#1 | 15-29, 75-89
But llc_shared_mask is not cleared. So llc_shared_mask of CPU#30 remains having 0x3fff80000001fffc0000000.
After that, when hot-adding socket#2 and #3, each core of sockets is numbered as follows:

           | CPU#
Socket#0 | 0-14 , 60-74
Socket#1 | 15-29, 75-89
Socket#2 | 30-59
Socket#3 | 90-119

Then llc_shared_mask of CPU#30 becomes 0x3fff8000fffffffc0000000. It means that last level cache of Socket#2 is shared with CPU#30-59 and 90-104. So the mask has wrong value.
At first, I cleared hot-removed CPU number's bit from llc_shared_map when hot removing CPU. But Borislav suggested that the problem will disappear if readded CPU is assigned same CPU number. And llc_shared_map must not be changed.
==

Quoted 6 lines
Quoted 1 line

So, please.

As I mentioned before, we still observe calltrace after Yasuaki's patch applied.
https://lkml.org/lkml/2014/7/29/40

Yes.
I just wanted to say we need your patch by showing real hardware case. Sorry for confusion I just reused his explanation of the problem.

I know Yasuaki's original trial was clearing llc_shared map as you do.

Quoted 4 lines

Actually I prefer to merge both patches, one for fix llc shared map unreleased during hotplug and the other one for assign same CPU number to readded CPU.

I agree.

Thanks,
-Kame

maintainer Reviewed-by on v5 1/1
#5 lore raw
In reply to Wanpeng Li
Quoted 32 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeatedly.

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 4 lines
Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#6 lore raw
Quoted 33 lines · Borislav Petkov
Quoted 32 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeatedly.

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 6 lines
Quoted 4 lines

Thanks.

Ingo, Peter Z, HPA,

Could this patch catch up with 3.18 merge window?

Regards,
Wanpeng Li

#7 lore raw
Quoted 42 lines · Wanpeng Li
Quoted 33 lines · Borislav Petkov
Quoted 32 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeatedly.

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 6 lines
Quoted 4 lines
Quoted 7 lines

Thanks.

Ingo, Peter Z, HPA,

Could this patch catch up with 3.18 merge window?

Please also add the real-hardware reports to the changelog.

Thanks,

Ingo

#8 lore raw
Quoted 42 lines · Ingo Molnar
Quoted 40 lines · Wanpeng Li
Quoted 39 lines · Borislav Petkov
Quoted 32 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeatedly.

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 4 lines
Quoted 8 lines
Quoted 7 lines

Thanks.

Ingo, Peter Z, HPA,

Could this patch catch up with 3.18 merge window?

Please also add the real-hardware reports to the changelog.

Just send out the latest version.

Regards,
Wanpeng Li

Quoted 4 lines

Thanks,

Ingo