s390/smp: Move rcu_cpu_starting() earlier
PATCH 1/1 s390/smp: Move rcu_cpu_starting() earlier
The call to rcu_cpu_starting() in smp_init_secondary() is not early enough in the CPU-hotplug onlining process, which results in lockdep splats as follows:
WARNING: suspicious RCU usage
-----------------------------
kernel/locking/lockdep.c:3497 RCU-list traversed in non-reader section!!
other info that might help us debug this:
RCU used illegally from offline CPU!
rcu_scheduler_active = 1, debug_locks = 1
no locks held by swapper/1/0.
Call Trace: show_stack+0x158/0x1f0 dump_stack+0x1f2/0x238 __lock_acquire+0x2640/0x4dd0 lock_acquire+0x3a8/0xd08 _raw_spin_lock_irqsave+0xc0/0xf0 clockevents_register_device+0xa8/0x528 init_cpu_timer+0x33e/0x468 smp_init_secondary+0x11a/0x328 smp_start_secondary+0x82/0x88
This is avoided by moving the call to rcu_cpu_starting up near the beginning of the smp_init_secondary() function. Note that the raw_smp_processor_id() is required in order to avoid calling into lockdep before RCU has declared the CPU to be watched for readers.
--- arch/s390/kernel/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/s390/kernel/smp.c +2 -1
@@ -855,13 +855,14 @@ void __init smp_detect_cpus(void)static void smp_init_secondary (void ) { -int cpu =smp_processor_id (); +int cpu =raw_smp_processor_id (); S390_lowcore.last_update_clock =get_tod_clock ();restore_access_regs (S390_lowcore.access_regs_save_area);set_cpu_flag (CIF_ASCE_PRIMARY );set_cpu_flag (CIF_ASCE_SECONDARY );cpu_init (); +rcu_cpu_starting (cpu);preempt_disable ();init_cpu_timer ();vtime_init ();
Signature · 1 line
2.28.0
Quoted 25 lines · Qian Cai
The call to
rcu_cpu_starting()insmp_init_secondary()is not early enough in the CPU-hotplug onlining process, which results in lockdep splats as follows:WARNING: suspicious RCU usage
-----------------------------
kernel/locking/lockdep.c:3497RCU-list traversed in non-reader section!!other info that might help us debug this:
RCU used illegally from offline CPU!
rcu_scheduler_active= 1,debug_locks= 1
no locks held by swapper/1/0.Call Trace: show_stack+0x158/0x1f0 dump_stack+0x1f2/0x238 __lock_acquire+0x2640/0x4dd0 lock_acquire+0x3a8/0xd08 _raw_spin_lock_irqsave+0xc0/0xf0 clockevents_register_device+0xa8/0x528 init_cpu_timer+0x33e/0x468 smp_init_secondary+0x11a/0x328 smp_start_secondary+0x82/0x88
Quoted 7 lines
This is avoided by moving the call to
rcu_cpu_startingup near the beginning of thesmp_init_secondary()function. Note that theraw_smp_processor_id()is required in order to avoid calling into lockdep before RCU has declared the CPU to be watched for readers.
Quoted 27 lines
--- arch/s390/kernel/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)arch/s390/kernel/smp.c +2
-1@@ -855,13 +855,14 @@ void __init smp_detect_cpus(void)static void smp_init_secondary (void ) { -int cpu =smp_processor_id (); +int cpu =raw_smp_processor_id (); S390_lowcore.last_update_clock =get_tod_clock ();restore_access_regs (S390_lowcore.access_regs_save_area);set_cpu_flag (CIF_ASCE_PRIMARY );set_cpu_flag (CIF_ASCE_SECONDARY );cpu_init (); +rcu_cpu_starting (cpu);preempt_disable ();init_cpu_timer ();vtime_init ();Signature · 1 line
2.28.0
Heiko Carstens
maintainerQuoted 30 lines · Qian Cai
The call to
rcu_cpu_starting()insmp_init_secondary()is not early enough in the CPU-hotplug onlining process, which results in lockdep splats as follows:WARNING: suspicious RCU usage
-----------------------------
kernel/locking/lockdep.c:3497RCU-list traversed in non-reader section!!other info that might help us debug this:
RCU used illegally from offline CPU!
rcu_scheduler_active= 1,debug_locks= 1
no locks held by swapper/1/0.Call Trace: show_stack+0x158/0x1f0 dump_stack+0x1f2/0x238 __lock_acquire+0x2640/0x4dd0 lock_acquire+0x3a8/0xd08 _raw_spin_lock_irqsave+0xc0/0xf0 clockevents_register_device+0xa8/0x528 init_cpu_timer+0x33e/0x468 smp_init_secondary+0x11a/0x328 smp_start_secondary+0x82/0x88This is avoided by moving the call to
rcu_cpu_startingup near the beginning of thesmp_init_secondary()function. Note that theraw_smp_processor_id()is required in order to avoid calling into lockdep before RCU has declared the CPU to be watched for readers.
Quoted 5 lines
Link https://lore.kernel.org/lkml/160223032121.7002.1269740091547117869.tip-bot2@tip-bot2/ open hereSigned-off-by Qian Cai <[email protected]>--- arch/s390/kernel/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Could you provide the config you used? I'm wondering why I can't reproduce this even though I have lot's of debug options enabled.
I will apply it anyway after rc2 has been released, just curious.
Quoted 31 lines · Heiko Carstens
Quoted 30 lines · Qian Cai
The call to
rcu_cpu_starting()insmp_init_secondary()is not early enough in the CPU-hotplug onlining process, which results in lockdep splats as follows:WARNING: suspicious RCU usage
-----------------------------
kernel/locking/lockdep.c:3497RCU-list traversed in non-reader section!!other info that might help us debug this:
RCU used illegally from offline CPU!
rcu_scheduler_active= 1,debug_locks= 1
no locks held by swapper/1/0.Call Trace: show_stack+0x158/0x1f0 dump_stack+0x1f2/0x238 __lock_acquire+0x2640/0x4dd0 lock_acquire+0x3a8/0xd08 _raw_spin_lock_irqsave+0xc0/0xf0 clockevents_register_device+0xa8/0x528 init_cpu_timer+0x33e/0x468 smp_init_secondary+0x11a/0x328 smp_start_secondary+0x82/0x88This is avoided by moving the call to
rcu_cpu_startingup near the beginning of thesmp_init_secondary()function. Note that theraw_smp_processor_id()is required in order to avoid calling into lockdep before RCU has declared the CPU to be watched for readers.
Quoted 8 lines
Quoted 5 lines
Link https://lore.kernel.org/lkml/160223032121.7002.1269740091547117869.tip-bot2@tip-bot2/ open hereSigned-off-by Qian Cai <[email protected]>--- arch/s390/kernel/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)Could you provide the config you used? I'm wondering why I can't reproduce this even though I have lot's of debug options enabled.
https://cailca.coding.net/public/linux/mm/git/files/master/s390.config
Essentially, I believe it requires CONFIG_PROVE_RCU_LIST=y. Also, it occurs to me that this only starts to happen after the commit mentioned in the above link.
Heiko Carstens
maintainerQuoted 11 lines · Qian Cai
Quoted 11 lines
Quoted 10 lines
This is avoided by moving the call to
rcu_cpu_startingup near the beginning of thesmp_init_secondary()function. Note that theraw_smp_processor_id()is required in order to avoid calling into lockdep before RCU has declared the CPU to be watched for readers.Link https://lore.kernel.org/lkml/160223032121.7002.1269740091547117869.tip-bot2@tip-bot2/ open hereSigned-off-by Qian Cai <[email protected]>--- arch/s390/kernel/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 6 lines
Quoted 2 lines
Could you provide the config you used? I'm wondering why I can't reproduce this even though I have lot's of debug options enabled.
https://cailca.coding.net/public/linux/mm/git/files/master/s390.config
Essentially, I believe it requires
CONFIG_PROVE_RCU_LIST=y. Also, it occurs to me that this only starts to happen after the commit mentioned in the above link.
Yes, with that enabled I can reprocuce it. Thanks! It depends on CONFIG_RCU_EXPERT. I can't image why I didn't had that enabled.. :)