powerpc/smp: Move rcu_cpu_starting() earlier
PATCH 1/1 powerpc/smp: Move rcu_cpu_starting() earlier
The call to rcu_cpu_starting() in start_secondary() is not early enough in the CPU-hotplug onlining process, which results in lockdep splats as follows:
WARNING: suspicious RCU usage
-----------------------------
kernel/locking/lockdep.c:3497 RCU-list traversed in non-reader section!!
other info that might help us debug this:
RCU used illegally from offline CPU!
rcu_scheduler_active = 1, debug_locks = 1
no locks held by swapper/1/0.
Call Trace: dump_stack+0xec/0x144 (unreliable) lockdep_rcu_suspicious+0x128/0x14c __lock_acquire+0x1060/0x1c60 lock_acquire+0x140/0x5f0 _raw_spin_lock_irqsave+0x64/0xb0 clockevents_register_device+0x74/0x270 register_decrementer_clockevent+0x94/0x110 start_secondary+0x134/0x800 start_secondary_prolog+0x10/0x14
This is avoided by moving the call to rcu_cpu_starting up near the beginning of the start_secondary() function. Note that the raw_smp_processor_id() is required in order to avoid calling into lockdep before RCU has declared the CPU to be watched for readers.
--- arch/powerpc/kernel/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/powerpc/kernel/smp.c +2 -1
@@ -1393,13 +1393,14 @@ static void add_cpu_to_masks(int cpu)/* Activate a secondary processor. */ void start_secondary (void *unused) { -unsigned int cpu =smp_processor_id (); +unsigned int cpu =raw_smp_processor_id ();mmgrab (&init_mm); current->active_mm = &init_mm;smp_store_cpu_info (cpu);set_dec (tb_ticks_per_jiffy); +rcu_cpu_starting (cpu);preempt_disable (); cpu_callin_map[cpu] =1 ;
Signature · 1 line
2.28.0
Quoted 25 lines · Qian Cai
The call to
rcu_cpu_starting()instart_secondary()is not early enough in the CPU-hotplug onlining process, which results in lockdep splats as follows:WARNING: suspicious RCU usage
-----------------------------
kernel/locking/lockdep.c:3497RCU-list traversed in non-reader section!!other info that might help us debug this:
RCU used illegally from offline CPU!
rcu_scheduler_active= 1,debug_locks= 1
no locks held by swapper/1/0.Call Trace: dump_stack+0xec/0x144 (unreliable) lockdep_rcu_suspicious+0x128/0x14c __lock_acquire+0x1060/0x1c60 lock_acquire+0x140/0x5f0 _raw_spin_lock_irqsave+0x64/0xb0 clockevents_register_device+0x74/0x270 register_decrementer_clockevent+0x94/0x110 start_secondary+0x134/0x800 start_secondary_prolog+0x10/0x14
Quoted 7 lines
This is avoided by moving the call to
rcu_cpu_startingup near the beginning of thestart_secondary()function. Note that theraw_smp_processor_id()is required in order to avoid calling into lockdep before RCU has declared the CPU to be watched for readers.
Quoted 27 lines
--- arch/powerpc/kernel/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)arch/powerpc/kernel/smp.c +2
-1@@ -1393,13 +1393,14 @@ static void add_cpu_to_masks(int cpu)/* Activate a secondary processor. */ void start_secondary (void *unused) { -unsigned int cpu =smp_processor_id (); +unsigned int cpu =raw_smp_processor_id ();mmgrab (&init_mm); current->active_mm = &init_mm;smp_store_cpu_info (cpu);set_dec (tb_ticks_per_jiffy); +rcu_cpu_starting (cpu);preempt_disable (); cpu_callin_map[cpu] =1 ;Signature · 1 line
2.28.0
Michael Ellerman
reviewerQuoted 3 lines · Qian Cai
The call to
rcu_cpu_starting()instart_secondary()is not early enough in the CPU-hotplug onlining process, which results in lockdep splats as follows:
Since when?
What kernel version?
I haven't seen this running CPU hotplug tests with PROVE_LOCKING=y on v5.10-rc1. Am I missing a CONFIG?
cheers
Quoted 54 lines
WARNING: suspicious RCU usage
-----------------------------
kernel/locking/lockdep.c:3497RCU-list traversed in non-reader section!!other info that might help us debug this:
RCU used illegally from offline CPU!
rcu_scheduler_active= 1,debug_locks= 1
no locks held by swapper/1/0.Call Trace: dump_stack+0xec/0x144 (unreliable) lockdep_rcu_suspicious+0x128/0x14c __lock_acquire+0x1060/0x1c60 lock_acquire+0x140/0x5f0 _raw_spin_lock_irqsave+0x64/0xb0 clockevents_register_device+0x74/0x270 register_decrementer_clockevent+0x94/0x110 start_secondary+0x134/0x800 start_secondary_prolog+0x10/0x14This is avoided by moving the call to
rcu_cpu_startingup near the beginning of thestart_secondary()function. Note that theraw_smp_processor_id()is required in order to avoid calling into lockdep before RCU has declared the CPU to be watched for readers.Link https://lore.kernel.org/lkml/160223032121.7002.1269740091547117869.tip-bot2@tip-bot2/ open hereSigned-off-by Qian Cai <[email protected]>--- arch/powerpc/kernel/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)arch/powerpc/kernel/smp.c +2
-1@@ -1393,13 +1393,14 @@ static void add_cpu_to_masks(int cpu)/* Activate a secondary processor. */ void start_secondary (void *unused) { -unsigned int cpu =smp_processor_id (); +unsigned int cpu =raw_smp_processor_id ();mmgrab (&init_mm); current->active_mm = &init_mm;smp_store_cpu_info (cpu);set_dec (tb_ticks_per_jiffy); +rcu_cpu_starting (cpu);preempt_disable (); cpu_callin_map[cpu] =1 ;Signature · 1 line
2.28.0
Quoted 10 lines · Michael Ellerman
Quoted 3 lines · Qian Cai
The call to
rcu_cpu_starting()instart_secondary()is not early enough in the CPU-hotplug onlining process, which results in lockdep splats as follows:Since when?
What kernel version?I haven't seen this running CPU hotplug tests with PROVE_LOCKING=y on v5.10-rc1. Am I missing a CONFIG?
My guess would be that adding CONFIG_PROVE_RAW_LOCK_NESTING=y will get you some splats.
Thanx, Paul
Quoted 57 lines
cheers
Quoted 54 lines
WARNING: suspicious RCU usage
-----------------------------
kernel/locking/lockdep.c:3497RCU-list traversed in non-reader section!!other info that might help us debug this:
RCU used illegally from offline CPU!
rcu_scheduler_active= 1,debug_locks= 1
no locks held by swapper/1/0.Call Trace: dump_stack+0xec/0x144 (unreliable) lockdep_rcu_suspicious+0x128/0x14c __lock_acquire+0x1060/0x1c60 lock_acquire+0x140/0x5f0 _raw_spin_lock_irqsave+0x64/0xb0 clockevents_register_device+0x74/0x270 register_decrementer_clockevent+0x94/0x110 start_secondary+0x134/0x800 start_secondary_prolog+0x10/0x14This is avoided by moving the call to
rcu_cpu_startingup near the beginning of thestart_secondary()function. Note that theraw_smp_processor_id()is required in order to avoid calling into lockdep before RCU has declared the CPU to be watched for readers.Link https://lore.kernel.org/lkml/160223032121.7002.1269740091547117869.tip-bot2@tip-bot2/ open hereSigned-off-by Qian Cai <[email protected]>--- arch/powerpc/kernel/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)arch/powerpc/kernel/smp.c +2
-1@@ -1393,13 +1393,14 @@ static void add_cpu_to_masks(int cpu)/* Activate a secondary processor. */ void start_secondary (void *unused) { -unsigned int cpu =smp_processor_id (); +unsigned int cpu =raw_smp_processor_id ();mmgrab (&init_mm); current->active_mm = &init_mm;smp_store_cpu_info (cpu);set_dec (tb_ticks_per_jiffy); +rcu_cpu_starting (cpu);preempt_disable (); cpu_callin_map[cpu] =1 ;Signature · 1 line
2.28.0
Quoted 6 lines · Paul E. McKenney
Quoted 5 lines · Michael Ellerman
Quoted 3 lines · Qian Cai
The call to
rcu_cpu_starting()instart_secondary()is not early enough in the CPU-hotplug onlining process, which results in lockdep splats as follows:
Quoted 8 lines
Quoted 5 lines
Since when?
What kernel version?I haven't seen this running CPU hotplug tests with PROVE_LOCKING=y on v5.10-rc1. Am I missing a CONFIG?
My guess would be that adding
CONFIG_PROVE_RAW_LOCK_NESTING=ywill get you some splats.
Well, I don't have that set, so it should be CONFIG_PROVE_RCU_LIST=y. Anyway, this is .config to reproduce on Power9 NV:
https://cailca.coding.net/public/linux/mm/git/files/master/powerpc.config
Quoted 6 lines · Michael Ellerman
Quoted 3 lines · Qian Cai
The call to
rcu_cpu_starting()instart_secondary()is not early enough in the CPU-hotplug onlining process, which results in lockdep splats as follows:Since when?
For me, it is since the commit in the link which looks now merged into v5.10-rc1. Then, it needs CONFIG_PROVE_RCU_LIST=y.
Quoted 63 lines
What kernel version?
I haven't seen this running CPU hotplug tests with PROVE_LOCKING=y on v5.10-rc1. Am I missing a CONFIG?
cheers
Quoted 55 lines
WARNING: suspicious RCU usage
-----------------------------
kernel/locking/lockdep.c:3497RCU-list traversed in non-reader section!!other info that might help us debug this:
RCU used illegally from offline CPU!
rcu_scheduler_active= 1,debug_locks= 1
no locks held by swapper/1/0.Call Trace: dump_stack+0xec/0x144 (unreliable) lockdep_rcu_suspicious+0x128/0x14c __lock_acquire+0x1060/0x1c60 lock_acquire+0x140/0x5f0 _raw_spin_lock_irqsave+0x64/0xb0 clockevents_register_device+0x74/0x270 register_decrementer_clockevent+0x94/0x110 start_secondary+0x134/0x800 start_secondary_prolog+0x10/0x14This is avoided by moving the call to
rcu_cpu_startingup near the beginning of thestart_secondary()function. Note that theraw_smp_processor_id()is required in order to avoid calling into lockdep before RCU has declared the CPU to be watched for readers.Link:
https://lore.kernel.org/lkml/160223032121.7002.1269740091547117869.tip-bot2@tip-bot2/Signed-off-by Qian Cai <[email protected]>--- arch/powerpc/kernel/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)arch/powerpc/kernel/smp.c +2
-1@@ -1393,13 +1393,14 @@ static void add_cpu_to_masks(int cpu)/* Activate a secondary processor. */ void start_secondary (void *unused) { -unsigned int cpu =smp_processor_id (); +unsigned int cpu =raw_smp_processor_id ();mmgrab (&init_mm); current->active_mm = &init_mm;smp_store_cpu_info (cpu);set_dec (tb_ticks_per_jiffy); +rcu_cpu_starting (cpu);preempt_disable (); cpu_callin_map[cpu] =1 ;Signature · 1 line
2.28.0
Michael Ellerman
AppliedQuoted 9 lines · Qian Cai
The call to
rcu_cpu_starting()instart_secondary()is not early enough in the CPU-hotplug onlining process, which results in lockdep splats as follows:WARNING: suspicious RCU usage
-----------------------------
kernel/locking/lockdep.c:3497RCU-list traversed in non-reader section!![...]
Applied to powerpc/fixes.
[1/1] powerpc/smp: Call rcu_cpu_starting() earlier
https://git.kernel.org/powerpc/c/99f070b62322a4b8c1252952735806d09eb44b68
cheers