PM: cpu: Restore the RCU grace period to cpu_pm_unregister_notifier()
cover PM: cpu: Restore the RCU grace period to cpu_pm_unregister_notifier()
Since commit b2f6662ac08d ("PM: cpu: Make notifier chain use a raw_spinlock_t") cpu_pm_unregister_notifier() no longer waits for an RCU grace period, but cpu_pm_notify() still walks the chain with only rcu_read_lock() held. A driver that frees its notifier block right after unregistering can race with a concurrent walker.
Patch 1 moves the one caller that unregisters from atomic context out from under its spinlock, patch 2 restores the grace period.
Bradley Morgan (2):
clocksource/drivers/timer-ti-dm: Unregister CPU PM notifier outside of
the timer lock
PM: cpu: Restore synchronize_rcu() to cpu_pm_unregister_notifier()
drivers/clocksource/timer-ti-dm.c | 9 ++++++--- kernel/cpu_pm.c | 9 ++++++++- 2 files changed, 14 insertions(+), 4 deletions(-)
Signature · 1 line
2.47.3
PATCH 1/2 clocksource/drivers/timer-ti-dm: Unregister CPU PM notifier outside of the timer lock
omap_dm_timer_remove() calls cpu_pm_unregister_notifier() with dm_timer_lock held and interrupts disabled. Nothing sleeps in there today, but it pins the helper into a context where it can never be allowed to sleep, which is in the way of restoring the RCU grace period on the cpu_pm notifier chain.
Do the list lookup under the lock and move the unregister after the unlock. Nothing can race it at that point: remove() owns the device, and once the timer is off the list nobody can reach it anymore.
--- drivers/clocksource/timer-ti-dm.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
drivers/clocksource/timer-ti-dm.c +6 -3
@@ -1530,7 +1530,7 @@ static int omap_dm_timer_probe(struct platform_device *pdev)*/ static void omap_dm_timer_remove (struct platform_device *pdev) { -struct dmtimer *timer; +struct dmtimer *timer, *found =NULL ;unsigned long flags;int ret = -EINVAL ;
@@ -1538,14 +1538,17 @@ static void omap_dm_timer_remove(struct platform_device *pdev)list_for_each_entry (timer, &omap_timer_list, node)if (!strcmp (dev_name (&timer->pdev->dev),dev_name (&pdev->dev))) { -if (!(timer->capability &OMAP_TIMER_ALWON )) -cpu_pm_unregister_notifier (&timer->nb);list_del (&timer->node); + found = timer; ret =0 ;break ; }spin_unlock_irqrestore (&dm_timer_lock, flags); +/* Unregister outside the lock: cpu_pm_unregister_notifier() may sleep. */ +if (found && !(found->capability &OMAP_TIMER_ALWON )) +cpu_pm_unregister_notifier (&found->nb); +pm_runtime_disable (&pdev->dev);if (ret)
Signature · 1 line
2.47.3
Re: [PATCH 1/2] clocksource/drivers/timer-ti-dm: Unregister CPU PM notifier outside of the timer lock
Quoted 40 lines · Bradley Morgan
omap_dm_timer_remove()callscpu_pm_unregister_notifier()withdm_timer_lockheld and interrupts disabled. Nothing sleeps in there today, but it pins the helper into a context where it can never be allowed to sleep, which is in the way of restoring the RCU grace period on thecpu_pmnotifier chain.Do the list lookup under the lock and move the unregister after the unlock. Nothing can race it at that point:
remove()owns the device, and once the timer is off the list nobody can reach it anymore.Signed-off-by Bradley Morgan <[email protected]>--- drivers/clocksource/timer-ti-dm.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)drivers/clocksource/timer-ti-dm.c +2
-3@@ -1530,7 +1530,7 @@ static int omap_dm_timer_probe(struct platform_device *pdev)*/ static void omap_dm_timer_remove (struct platform_device *pdev) { -struct dmtimer *timer; +struct dmtimer *timer, *found =NULL ;unsigned long flags;int ret = -EINVAL ;@@ -1538,14 +1538,17 @@ static void omap_dm_timer_remove(struct platform_device *pdev)list_for_each_entry (timer, &omap_timer_list, node)if (!strcmp (dev_name (&timer->pdev->dev),dev_name (&pdev->dev))) { -if (!(timer->capability &OMAP_TIMER_ALWON )) -cpu_pm_unregister_notifier (&timer->nb);list_del (&timer->node); + found = timer; ret =0 ;break ; }
drivers/clocksource/timer-ti-dm.c static void omap_dm_timer_remove(struct platform_device *pdev)
Quoted 10 lines
drivers/clocksource/timer-ti-dm.c +4
-1spin_unlock_irqrestore (&dm_timer_lock, flags); +/* Unregister outside the lock: cpu_pm_unregister_notifier() may sleep. */ +if (found && !(found->capability &OMAP_TIMER_ALWON )) +cpu_pm_unregister_notifier (&found->nb); +pm_runtime_disable (&pdev->dev);if (ret) --
Nobody seems to be interested in this patch, but Sashiko only sees pre-existing issues related to it:
https://sashiko.dev/#/patchset/20260813074343.26460-1-include%40grrlz.net
and the second patch in the series depends on it, so in the absence of any objections or concerns, I'm going to apply them both.
Thanks!
Re: [PATCH 1/2] clocksource/drivers/timer-ti-dm: Unregister CPU PM notifier outside of the timer lock
Quoted 59 lines · Rafael J. Wysocki (Intel)
Quoted 11 lines · Bradley Morgan
omap_dm_timer_remove()callscpu_pm_unregister_notifier()withdm_timer_lockheld and interrupts disabled. Nothing sleeps in there today, but it pins the helper into a context where it can never be allowed to sleep, which is in the way of restoring the RCU grace period on thecpu_pmnotifier chain.Do the list lookup under the lock and move the unregister after the unlock. Nothing can race it at that point:
remove()owns the device, and once the timer is off the list nobody can reach it anymore.Review
Quoted 6 lines
Signed-off-by Bradley Morgan <[email protected]>--- drivers/clocksource/timer-ti-dm.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)(quoted diff) +0
-0
b/drivers/clocksource/timer-ti-dm.cdrivers/clocksource/timer-ti-dm.c static int omap_dm_timer_probe(struct
Quoted 4 lines
index
bd06afb7d522..032e102bdd0a100644drivers/clocksource/timer-ti-dm.c +0
-0@@ -1530,7 +1530,7 @@ static int omap_dm_timer_probe(struct
platform_device*pdev)drivers/clocksource/timer-ti-dm.c static void omap_dm_timer_remove(struct
Quoted 9 lines
drivers/clocksource/timer-ti-dm.c +1
-1*/ static void omap_dm_timer_remove (struct platform_device *pdev) { -struct dmtimer *timer; +struct dmtimer *timer, *found =NULL ;unsigned long flags;int ret = -EINVAL ;@@ -1538,14 +1538,17 @@ static void omap_dm_timer_remove(struct
platform_device*pdev)drivers/clocksource/timer-ti-dm.c static void omap_dm_timer_remove(struct platform_device *pdev)
Quoted 13 lines
drivers/clocksource/timer-ti-dm.c +2
-2list_for_each_entry (timer, &omap_timer_list, node)if (!strcmp (dev_name (&timer->pdev->dev),dev_name (&pdev->dev))) { -if (!(timer->capability &OMAP_TIMER_ALWON )) -cpu_pm_unregister_notifier (&timer->nb);list_del (&timer->node); + found = timer; ret =0 ;break ; }spin_unlock_irqrestore (&dm_timer_lock, flags); +/* Unregister outside the lock: cpu_pm_unregister_notifier() may sleep. */
drivers/clocksource/timer-ti-dm.c static void omap_dm_timer_remove(struct platform_device *pdev)
Quoted 7 lines
drivers/clocksource/timer-ti-dm.c +3
-1+if (found && !(found->capability &OMAP_TIMER_ALWON )) +cpu_pm_unregister_notifier (&found->nb); +pm_runtime_disable (&pdev->dev);if (ret) --Nobody seems to be interested in this patch, but Sashiko only sees pre-existing issues related to it:
Quoted 8 lines
https://sashiko.dev/#/patchset/20260813074343.26460-1-include%40grrlz.net
and the second patch in the series depends on it, so in the absence of any objections or concerns, I'm going to apply them both.
Thanks!
Ugh, I'll check / fix the pre existing issues now
Id like to wonder why Google didn't use their 3.8 flash, considering it's like 10x more capable
--- Thanks!
https://lore.kernel.org/all/[email protected]/
Re: [PATCH 1/2] clocksource/drivers/timer-ti-dm: Unregister CPU PM notifier outside of the timer lock
In reply to Rafael J. Wysocki (Intel)Quoted 56 lines · Rafael J. Wysocki (Intel)
Quoted 40 lines · Bradley Morgan
omap_dm_timer_remove()callscpu_pm_unregister_notifier()withdm_timer_lockheld and interrupts disabled. Nothing sleeps in there today, but it pins the helper into a context where it can never be allowed to sleep, which is in the way of restoring the RCU grace period on thecpu_pmnotifier chain.Do the list lookup under the lock and move the unregister after the unlock. Nothing can race it at that point:
remove()owns the device, and once the timer is off the list nobody can reach it anymore.Signed-off-by Bradley Morgan <[email protected]>--- drivers/clocksource/timer-ti-dm.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)drivers/clocksource/timer-ti-dm.c +2
-3@@ -1530,7 +1530,7 @@ static int omap_dm_timer_probe(struct platform_device *pdev)*/ static void omap_dm_timer_remove (struct platform_device *pdev) { -struct dmtimer *timer; +struct dmtimer *timer, *found =NULL ;unsigned long flags;int ret = -EINVAL ;@@ -1538,14 +1538,17 @@ static void omap_dm_timer_remove(struct platform_device *pdev)list_for_each_entry (timer, &omap_timer_list, node)if (!strcmp (dev_name (&timer->pdev->dev),dev_name (&pdev->dev))) { -if (!(timer->capability &OMAP_TIMER_ALWON )) -cpu_pm_unregister_notifier (&timer->nb);list_del (&timer->node); + found = timer; ret =0 ;break ; }drivers/clocksource/timer-ti-dm.c static void omap_dm_timer_remove(struct platform_device *pdev)
Quoted 10 lines
drivers/clocksource/timer-ti-dm.c +4
-1spin_unlock_irqrestore (&dm_timer_lock, flags); +/* Unregister outside the lock: cpu_pm_unregister_notifier() may sleep. */ +if (found && !(found->capability &OMAP_TIMER_ALWON )) +cpu_pm_unregister_notifier (&found->nb); +pm_runtime_disable (&pdev->dev);if (ret) --Nobody seems to be interested in this patch, but Sashiko only sees pre-existing issues related to it:
Quoted 4 lines
https://sashiko.dev/#/patchset/20260813074343.26460-1-include%40grrlz.net
and the second patch in the series depends on it, so in the absence of any objections or concerns, I'm going to apply them both.
And so applied as 7.4 material, thanks!
PATCH 2/2 PM: cpu: Restore synchronize_rcu() to cpu_pm_unregister_notifier()
In reply to Bradley Morgancpu_pm_notify() walks the notifier chain lockless under only rcu_read_lock(). That only works if a removed notifier block is not freed until every concurrent walker is done with it.
The chain used to be an atomic_notifier, whose unregister ends in synchronize_rcu(). Commit b2f6662ac08d ("PM: cpu: Make notifier chain use a raw_spinlock_t") switched it over to a raw_notifier, and in doing so replaced that with raw_notifier_chain_unregister(), which does not synchronize. The grace period quietly went away, so a driver that frees the memory holding its notifier block right after cpu_pm_unregister_notifier() returns can race with a concurrent walker:
cpu1 (idle exit) cpu2 (driver remove)
---------------- --------------------
cpu_pm_notify(CPU_PM_EXIT)
rcu_read_lock()
nb = rcu_dereference_raw(*nl)
cpu_pm_unregister_notifier(&od->nb)
unlink, no grace period
remove() frees od (devm)
nb->notifier_call(nb, ...)
/* use after free */The rcu_read_lock() on cpu1 does not stop cpu2 from freeing the block, and nothing else does. The read side is the idle exit path, so this can hit on any system where a driver with a cpu_pm notifier gets unbound.
Add the grace period back. It is only needed when a notifier was actually removed, so wait on success and return -ENOENT without waiting otherwise. The kerneldoc gets back the "may sleep" note that the same commit dropped.
--- kernel/cpu_pm.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/cpu_pm.c +8 -1
@@ -10,6 +10,7 @@#include <linux/cpu_pm.h> #include <linux/module.h> #include <linux/notifier.h> +#include <linux/rcupdate.h> #include <linux/spinlock.h> #include <linux/syscore_ops.h>
@@ -76,7 +77,8 @@ EXPORT_SYMBOL_GPL(cpu_pm_register_notifier);* * Remove a driver from the CPU PM notifier list. * -* This function has the same return conditions as raw_notifier_chain_unregister. +* This function may sleep, and has the same return conditions as +* raw_notifier_chain_unregister. */ int cpu_pm_unregister_notifier (struct notifier_block *nb) {
@@ -86,6 +88,11 @@ int cpu_pm_unregister_notifier(struct notifier_block *nb)raw_spin_lock_irqsave (&cpu_pm_notifier.lock, flags); ret =raw_notifier_chain_unregister (&cpu_pm_notifier.chain, nb);raw_spin_unlock_irqrestore (&cpu_pm_notifier.lock, flags); + +/* Wait for the rcu_read_lock() walkers in cpu_pm_notify(). */ +if (!ret) +synchronize_rcu (); +return ret; }EXPORT_SYMBOL_GPL (cpu_pm_unregister_notifier);
Signature · 1 line
2.47.3
Re: [PATCH 2/2] PM: cpu: Restore synchronize_rcu() to cpu_pm_unregister_notifier()
Quoted 30 lines · Bradley Morgan
cpu_pm_notify()walks the notifier chain lockless under onlyrcu_read_lock(). That only works if a removed notifier block is not freed until every concurrent walker is done with it.The chain used to be an
atomic_notifier, whose unregister ends insynchronize_rcu(). Commitb2f6662ac08d("PM: cpu: Make notifier chain use araw_spinlock_t") switched it over to araw_notifier, and in doing so replaced that withraw_notifier_chain_unregister(), which does not synchronize. The grace period quietly went away, so a driver that frees the memory holding its notifier block right aftercpu_pm_unregister_notifier()returns can race with a concurrent walker:cpu1 (idle exit) cpu2 (driver remove)
---------------- --------------------
cpu_pm_notify(CPU_PM_EXIT)rcu_read_lock() nb = rcu_dereference_raw(*nl) cpu_pm_unregister_notifier(&od->nb) unlink, no grace period remove() frees od (devm) nb->notifier_call(nb, ...) /* use after free */The
rcu_read_lock()on cpu1 does not stop cpu2 from freeing the block, and nothing else does. The read side is the idle exit path, so this can hit on any system where a driver with acpu_pmnotifier gets unbound.
Quoted 8 lines
Add the grace period back. It is only needed when a notifier was actually removed, so wait on success and return
-ENOENTwithout waiting otherwise. The kerneldoc gets back the "may sleep" note that the same commit dropped.Fixes b2f6662ac08d ("PM: cpu: Make notifier chain use a raw_spinlock_t")Signed-off-by Bradley Morgan <[email protected]>
or if you want me to pick up this series, I need an ACK on the first patch.
Quoted 41 lines
--- kernel/cpu_pm.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)kernel/cpu_pm.c +8
-2@@ -10,6 +10,7 @@#include <linux/cpu_pm.h> #include <linux/module.h> #include <linux/notifier.h> +#include <linux/rcupdate.h> #include <linux/spinlock.h> #include <linux/syscore_ops.h> @@ -76,7 +77,8 @@ EXPORT_SYMBOL_GPL(cpu_pm_register_notifier);* * Remove a driver from the CPU PM notifier list. * -* This function has the same return conditions as raw_notifier_chain_unregister. +* This function may sleep, and has the same return conditions as +* raw_notifier_chain_unregister. */ int cpu_pm_unregister_notifier (struct notifier_block *nb) {@@ -86,6 +88,11 @@ int cpu_pm_unregister_notifier(struct notifier_block *nb)raw_spin_lock_irqsave (&cpu_pm_notifier.lock, flags); ret =raw_notifier_chain_unregister (&cpu_pm_notifier.chain, nb);raw_spin_unlock_irqrestore (&cpu_pm_notifier.lock, flags); + +/* Wait for the rcu_read_lock() walkers in cpu_pm_notify(). */ +if (!ret) +synchronize_rcu (); +return ret; }EXPORT_SYMBOL_GPL (cpu_pm_unregister_notifier); --2.47.3
Re: [PATCH 2/2] PM: cpu: Restore synchronize_rcu() to cpu_pm_unregister_notifier()
Quoted 37 lines · Rafael J. Wysocki (Intel)
Quoted 30 lines · Bradley Morgan
cpu_pm_notify()walks the notifier chain lockless under onlyrcu_read_lock(). That only works if a removed notifier block is not freed until every concurrent walker is done with it.The chain used to be an
atomic_notifier, whose unregister ends insynchronize_rcu(). Commitb2f6662ac08d("PM: cpu: Make notifier chain use araw_spinlock_t") switched it over to araw_notifier, and in doing so replaced that withraw_notifier_chain_unregister(), which does not synchronize. The grace period quietly went away, so a driver that frees the memory holding its notifier block right aftercpu_pm_unregister_notifier()returns can race with a concurrent walker:cpu1 (idle exit) cpu2 (driver remove)
---------------- --------------------
cpu_pm_notify(CPU_PM_EXIT)rcu_read_lock() nb = rcu_dereference_raw(*nl) cpu_pm_unregister_notifier(&od->nb) unlink, no grace period remove() frees od (devm) nb->notifier_call(nb, ...) /* use after free */The
rcu_read_lock()on cpu1 does not stop cpu2 from freeing the block, and nothing else does. The read side is the idle exit path, so this can hit on any system where a driver with acpu_pmnotifier gets unbound.Quoted 6 lines
Add the grace period back. It is only needed when a notifier was actually removed, so wait on success and return
-ENOENTwithout waiting otherwise. The kerneldoc gets back the "may sleep" note that the same commit dropped.Fixes b2f6662ac08d ("PM: cpu: Make notifier chain use a
Quoted 9 lines
raw_spinlock_t")Quoted 2 lines
Signed-off-by Bradley Morgan <[email protected]>or if you want me to pick up this series, I need an ACK on the first patch.
Waiting game... (And I can't take it! I'm not maintainer of anything sadly, one day, one day, I'm manifesting it.. one day)
Quoted 29 lines
Quoted 17 lines
--- kernel/cpu_pm.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)kernel/cpu_pm.c +1
-0@@ -10,6 +10,7 @@#include <linux/cpu_pm.h> #include <linux/module.h> #include <linux/notifier.h> +#include <linux/rcupdate.h> #include <linux/spinlock.h> #include <linux/syscore_ops.h> kernel/cpu_pm.c EXPORT_SYMBOL_GPL(cpu_pm_register_notifier); · line 79
Quoted 4 lines
kernel/cpu_pm.c +0
-1@@ -76,7 +77,8 @@ EXPORT_SYMBOL_GPL(cpu_pm_register_notifier);* * Remove a driver from the CPU PM notifier list. * -* This function has the same return conditions as
raw_notifier_chain_unregister.kernel/cpu_pm.c int cpu_pm_unregister_notifier(struct notifier_block
Quoted 6 lines
kernel/cpu_pm.c +2
-0+* This function may sleep, and has the same return conditions as +* raw_notifier_chain_unregister. */ int cpu_pm_unregister_notifier (struct notifier_block *nb) {@@ -86,6 +88,11 @@ int cpu_pm_unregister_notifier(struct notifier_block*nb)
Quoted 15 lines
Quoted 12 lines
kernel/cpu_pm.c +5
-1raw_spin_lock_irqsave (&cpu_pm_notifier.lock, flags); ret =raw_notifier_chain_unregister (&cpu_pm_notifier.chain, nb);raw_spin_unlock_irqrestore (&cpu_pm_notifier.lock, flags); + +/* Wait for the rcu_read_lock() walkers in cpu_pm_notify(). */ +if (!ret) +synchronize_rcu (); +return ret; }EXPORT_SYMBOL_GPL (cpu_pm_unregister_notifier); --Quoted 2 lines
2.47.3
Thanks!