notifier: Make atomic_notifiers use raw_spinlock
PATCH 1/1 notifier: Make atomic_notifiers use raw_spinlock
From: Valentin Schneider <[email protected]>
Date: Sun, 22 Nov 2020 20:19:04 +0000
Booting a recent PREEMPT_RT kernel (v5.10-rc3-rt7-rebase) on my arm64 Juno leads to the idle task blocking on an RT sleeping spinlock down some notifier path:
| BUG: scheduling while atomic: swapper/5/0/0x00000002 …
| atomic_notifier_call_chain_robust (kernel/notifier.c:71 kernel/notifier.c:118 kernel/notifier.c:186) | cpu_pm_enter (kernel/cpu_pm.c:39 kernel/cpu_pm.c:93) | psci_enter_idle_state (drivers/cpuidle/cpuidle-psci.c:52 drivers/cpuidle/cpuidle-psci.c:129) | cpuidle_enter_state (drivers/cpuidle/cpuidle.c:238) | cpuidle_enter (drivers/cpuidle/cpuidle.c:353) | do_idle (kernel/sched/idle.c:132 kernel/sched/idle.c:213 kernel/sched/idle.c:273) | cpu_startup_entry (kernel/sched/idle.c:368 (discriminator 1)) | secondary_start_kernel (arch/arm64/kernel/smp.c:273)
Two points worth noting:
1) That this is conceptually the same issue as pointed out in:
313c8c16ee62 ("PM / CPU: replace raw_notifier with atomic_notifier")
2) Only the _robust() variant of atomic_notifier callchains suffer from this
AFAICT only the cpu_pm_notifier_chain really needs to be changed, but singling it out would mean introducing a new (truly) non-blocking API. At the same time, callers that are fine with any blocking within the call chain should use blocking notifiers, so patching up all atomic_notifier's doesn't seem *too* crazy to me.
---
What do we do with this?
Do we merge this as-is, add another "robust atomic notifier" using only raw_spinlock_t for registration and notification (for only cpu_pm_notifier_chain) instead of switching to raw_spinlock_t for all atomic notifier in -tree?
include/linux/notifier.h | 6 +++--- kernel/notifier.c | 12 ++++++------ 2 files changed, 9 insertions(+), 9 deletions(-)
include/linux/notifier.h +3 -3
@@ -58,7 +58,7 @@ struct notifier_block { };struct atomic_notifier_head { -spinlock_t lock; +raw_spinlock_t lock;struct notifier_block __rcu *head; };
@@ -78,7 +78,7 @@ struct srcu_notifier_head { };#define ATOMIC_INIT_NOTIFIER_HEAD (name) do { \ - spin_lock_init(&(name)->lock); \ + raw_spin_lock_init(&(name)->lock); \ (name)->head = NULL; \ } while (0)#define BLOCKING_INIT_NOTIFIER_HEAD (name) do { \
@@ -95,7 +95,7 @@ extern void srcu_init_notifier_head(struct srcu_notifier_head *nh);cleanup_srcu_struct (&(name)->srcu);#define ATOMIC_NOTIFIER_INIT (name) { \ - .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ + .lock = __RAW_SPIN_LOCK_UNLOCKED(name.lock), \ .head = NULL }#define BLOCKING_NOTIFIER_INIT (name) { \ .rwsem = __RWSEM_INITIALIZER((name).rwsem), \
kernel/notifier.c +6 -6
@@ -142,9 +142,9 @@ int atomic_notifier_chain_register(struct atomic_notifier_head *nh,unsigned long flags;int ret; -spin_lock_irqsave (&nh->lock, flags); +raw_spin_lock_irqsave (&nh->lock, flags); ret =notifier_chain_register (&nh->head, n); -spin_unlock_irqrestore (&nh->lock, flags); +raw_spin_unlock_irqrestore (&nh->lock, flags);return ret; }EXPORT_SYMBOL_GPL (atomic_notifier_chain_register);
@@ -164,9 +164,9 @@ int atomic_notifier_chain_unregister(struct atomic_notifier_head *nh,unsigned long flags;int ret; -spin_lock_irqsave (&nh->lock, flags); +raw_spin_lock_irqsave (&nh->lock, flags); ret =notifier_chain_unregister (&nh->head, n); -spin_unlock_irqrestore (&nh->lock, flags); +raw_spin_unlock_irqrestore (&nh->lock, flags);synchronize_rcu ();return ret; }
@@ -182,9 +182,9 @@ int atomic_notifier_call_chain_robust(struct atomic_notifier_head *nh,* Musn't use RCU; because then the notifier list can * change between the up and down traversal. */ -spin_lock_irqsave (&nh->lock, flags); +raw_spin_lock_irqsave (&nh->lock, flags); ret =notifier_call_chain_robust (&nh->head, val_up, val_down, v); -spin_unlock_irqrestore (&nh->lock, flags); +raw_spin_unlock_irqrestore (&nh->lock, flags);return ret; }
Signature · 1 line
2.32.0
Quoted 5 lines · Sebastian Andrzej Siewior
What do we do with this?
Do we merge this as-is, add another "robust atomic notifier" using onlyraw_spinlock_tfor registration and notification (for onlycpu_pm_notifier_chain) instead of switching toraw_spinlock_tfor all atomic notifier in -tree?
Right, so the problem I see with this is that notifier_chain_{,un}register() are O(n). Hardly something we should be putting under raw_spin_lock :/
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Sebastian Andrzej Siewior
What do we do with this?
Do we merge this as-is, add another "robust atomic notifier" using onlyraw_spinlock_tfor registration and notification (for onlycpu_pm_notifier_chain) instead of switching toraw_spinlock_tfor all atomic notifier in -tree?Right, so the problem I see with this is that notifier_chain_{,un}register() are O(n). Hardly something we should be putting under
raw_spin_lock:/
Yup, pretty much. So we make one robust notifier for cpu_pm_notifier_chain?
Sebastian
Quoted 8 lines · Sebastian Andrzej Siewior
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Sebastian Andrzej Siewior
What do we do with this?
Do we merge this as-is, add another "robust atomic notifier" using onlyraw_spinlock_tfor registration and notification (for onlycpu_pm_notifier_chain) instead of switching toraw_spinlock_tfor all atomic notifier in -tree?
Quoted 6 lines
Quoted 3 lines
Right, so the problem I see with this is that notifier_chain_{,un}register() are O(n). Hardly something we should be putting under
raw_spin_lock:/Yup, pretty much. So we make one robust notifier for
cpu_pm_notifier_chain?
Yeah, I suppose so :-( Ideally that whole pm notifier thing goes, but that's *far* more work and I really don't want to be responsible for the brain damange resulting from looking at all that 'special' idle code.