Skip to contentSkip to conversation

x86, hotplug: fix llc shared map unreleased during cpu hotplug

PATCH Superseded Wanpeng Li 7 participants 16 messages Jul 29, 2014 – Sep 16, 2014 1 message not in the archive x86

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Wanpeng Li

author
#0 lore raw

PATCH v4 1/1 x86, hotplug: fix llc shared map unreleased during cpu hotplug

BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

---
v1 -> v2:
* fix subject line
v2 -> v3:
* simplify backtrace

 arch/x86/kernel/smpboot.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kernel/smpboot.c +3 -0
index 5492798..0134ec7 100644
@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
 
 	for_each_cpu(sibling, cpu_sibling_mask(cpu))
 		cpumask_clear_cpu(cpu, cpu_sibling_mask(sibling));
+	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
+		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
+	cpumask_clear(cpu_llc_shared_mask(cpu));
 	cpumask_clear(cpu_sibling_mask(cpu));
 	cpumask_clear(cpu_core_mask(cpu));
 	c->phys_proc_id = 0;
Signature · 1 line
1.9.1
Reviewed-by on v4 1/1
#1 lore raw
Quoted 23 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

Quoted 8 lines

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

The change looks good to me.

Thanks,
-Toshi

#2 lore raw
In reply to Wanpeng Li

Ping

Quoted 54 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

---
v1 -> v2:
* fix subject line
v2 -> v3:
* simplify backtrace

 arch/x86/kernel/smpboot.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kernel/smpboot.c +3 -0
index 5492798..0134ec7 100644
@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
 
 	for_each_cpu(sibling, cpu_sibling_mask(cpu))
 		cpumask_clear_cpu(cpu, cpu_sibling_mask(sibling));
+	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
+		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
+	cpumask_clear(cpu_llc_shared_mask(cpu));
 	cpumask_clear(cpu_sibling_mask(cpu));
 	cpumask_clear(cpu_core_mask(cpu));
 	c->phys_proc_id = 0;

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#4 lore raw

Hi Peter,
On Fri, Aug 08, 2014 at 04:40:57PM -0600, Linn Crosetto wrote:
[...]

Quoted 5 lines

Tested with a CPU hotplug stress test, run on a large system with 240 CPUs. Thanks.

Is it ok for you to apply this patch or still need update?

Regards,
Wanpeng Li

maintainer
#5 lore raw
Quoted 1 line · Wanpeng Li

Is it ok for you to apply this patch or still need update?

Just be patient: we have the merge window still open and after that kernel summit coming up first.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#6 lore raw
Quoted 4 lines · Borislav Petkov
Quoted 1 line · Wanpeng Li

Is it ok for you to apply this patch or still need update?

Just be patient: we have the merge window still open and after that kernel summit coming up first.

Thanks for pointing out.

Regards,
Wanpeng Li

Quoted 1 line
author
#7 lore raw

Ping Ingo, Peter,

Quoted 13 lines · Wanpeng Li
Quoted 4 lines · Borislav Petkov
Quoted 1 line · Wanpeng Li

Is it ok for you to apply this patch or still need update?

Just be patient: we have the merge window still open and after that kernel summit coming up first.

Thanks for pointing out.

Regards,
Wanpeng Li

Quoted 1 line
#8 lore raw
In reply to Wanpeng Li
Quoted 23 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

Quoted 5 lines

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Very little is said in this changelog about how the bug was found, how likely it is to occur for others, what systems are affected, etc.

Thanks,

Ingo

#9 lore raw
Quoted 32 lines · Ingo Molnar
Quoted 23 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

Quoted 5 lines

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 3 lines

Very little is said in this changelog about how the bug was found, how likely it is to occur for others, what systems are affected, etc.

Hi Wanpeng,

In my understanding, the panic occurs by just onlining CPU as follows:
echo 1 > /sys/devices/system/cpu/cpuX/online

So, how about add the information?

Thanks,
Yasuaki Ishimatsu

Quoted 5 lines

Thanks,

Ingo

#10 lore raw
Quoted 37 lines · Yasuaki Ishimatsu
Quoted 32 lines · Ingo Molnar
Quoted 23 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

Quoted 5 lines

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 3 lines

Very little is said in this changelog about how the bug was found, how likely it is to occur for others, what systems are affected, etc.

Quoted 8 lines

Hi Wanpeng,

In my understanding, the panic occurs by just onlining CPU as follows:
echo 1 > /sys/devices/system/cpu/cpuX/online

So, how about add the information?

From what I remember you need a special kind of hardware too, one that

doesn't preserve cpu numbers across hotplug. Most systems do; just not this magic special one.

We want to fix that, but the only reason for this patch is consistency with the rest of the code, we do indeed clear and set these bits in all other masks, but not this one.

But yes, the Changelog needs help.

author
#11 lore raw
In reply to Yasuaki Ishimatsu
Quoted 33 lines · Yasuaki Ishimatsu
Quoted 32 lines · Ingo Molnar
Quoted 28 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 5 lines
Quoted 3 lines

Very little is said in this changelog about how the bug was found, how likely it is to occur for others, what systems are affected, etc.

Hi Wanpeng,

Hi Yasuaki,

Quoted 4 lines

In my understanding, the panic occurs by just onlining CPU as follows:
echo 1 > /sys/devices/system/cpu/cpuX/online

See my reply to Ingo.

Regards,
Wanpeng Li

Quoted 12 lines

So, how about add the information?

Thanks,
Yasuaki Ishimatsu

Quoted 5 lines

Thanks,

Ingo

author
#12 lore raw
In reply to Ingo Molnar
Quoted 32 lines · Ingo Molnar
Quoted 23 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

Quoted 5 lines

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 3 lines

Very little is said in this changelog about how the bug was found, how likely it is to occur for others, what systems are affected, etc.

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeated.

Regards,
Wanpeng Li

Quoted 4 lines

Thanks,

Ingo

author
#13 lore raw

Hi Ingo,

Quoted 33 lines · Wanpeng Li
Quoted 32 lines · Ingo Molnar
Quoted 28 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 7 lines
Quoted 3 lines

Very little is said in this changelog about how the bug was found, how likely it is to occur for others, what systems are affected, etc.

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeated.

If I need to send a new version of the patch or you can pick the patch w/ the updated changelog for me? ;-)

Regards,
Wanpeng Li

Quoted 7 lines

Regards,
Wanpeng Li

Quoted 4 lines

Thanks,

Ingo

#14 lore raw

Ping Ingo, HPA, PeterZ

Quoted 56 lines · Wanpeng Li

Hi Ingo,

Quoted 31 lines · Wanpeng Li
Quoted 30 lines · Ingo Molnar
Quoted 28 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Quoted 6 lines
Quoted 3 lines

Very little is said in this changelog about how the bug was found, how likely it is to occur for others, what systems are affected, etc.

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeated.

If I need to send a new version of the patch or you can pick the patch w/ the updated changelog for me? ;-)

Regards,
Wanpeng Li

Quoted 6 lines

Regards,
Wanpeng Li

Quoted 3 lines

Thanks,

Ingo

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#15 lore raw
In reply to Wanpeng Li
Quoted 39 lines · Wanpeng Li

Hi Ingo,

Quoted 37 lines · Wanpeng Li
Quoted 35 lines · Ingo Molnar
Quoted 23 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

Quoted 5 lines

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Very little is said in this changelog about how the bug was found, how likely it is to occur for others, what systems are affected, etc.

Quoted 6 lines
Quoted 3 lines

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeated.

If I need to send a new version of the patch or you can pick the patch w/ the updated changelog for me? ;-)

Please send a fresh new version, maintainers trying to splice & dice patches and changelogs is an unrobust approach prone to mistakes.

Thanks,

Ingo

#16 lore raw
Quoted 38 lines · Ingo Molnar
Quoted 36 lines · Wanpeng Li

Hi Ingo,

Quoted 34 lines · Wanpeng Li
Quoted 33 lines · Ingo Molnar
Quoted 23 lines · Wanpeng Li
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
IP: [..] find_busiest_group
PGD 5a9d5067 PUD 13067 PMD 0
Oops: 0000 [#3] SMP
[...]
Call Trace:

load_balance
? _raw_spin_unlock_irqrestore
idle_balance
__schedule
schedule
schedule_timeout
? lock_timer_base
schedule_timeout_uninterruptible
msleep
lock_device_hotplug_sysfs
online_store
dev_attr_store
sysfs_write_file
vfs_write
SyS_write
system_call_fastpath

Quoted 5 lines

Last level cache shared map is built during cpu up and build sched domain routine takes advantage of it to setup sched domain cpu topology, however, llc shared map is unreleased during cpu disable which lead to invalid sched domain cpu topology. This patch fix it by release llc shared map correctly during cpu disable.

Very little is said in this changelog about how the bug was found, how likely it is to occur for others, what systems are affected, etc.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

This bug can be triggered by hot add and remove large number of xen domain0's vcpus repeated.

If I need to send a new version of the patch or you can pick the patch w/ the updated changelog for me? ;-)

Please send a fresh new version, maintainers trying to splice & dice patches and changelogs is an unrobust approach prone to mistakes.

Ok, I will send out a new version tomorrow. ;-)

Regards,
Wanpeng Li

Quoted 9 lines

Thanks,

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/