Skip to contentSkip to conversation

PM: cpu: Restore the RCU grace period to cpu_pm_unregister_notifier()

PATCH In review Bradley Morgan 3 participants 8 messages Aug 13 – Sep 11 Clocksource, clockevent drivers

lore
Share & tools
lore
#0 lore raw

cover PM: cpu: Restore the RCU grace period to cpu_pm_unregister_notifier()

Since commit b2f6662ac08d ("PM: cpu: Make notifier chain use a raw_spinlock_t") cpu_pm_unregister_notifier() no longer waits for an RCU grace period, but cpu_pm_notify() still walks the chain with only rcu_read_lock() held. A driver that frees its notifier block right after unregistering can race with a concurrent walker.

Patch 1 moves the one caller that unregisters from atomic context out from under its spinlock, patch 2 restores the grace period.

Bradley Morgan (2):
clocksource/drivers/timer-ti-dm: Unregister CPU PM notifier outside of
the timer lock
PM: cpu: Restore synchronize_rcu() to cpu_pm_unregister_notifier()

 drivers/clocksource/timer-ti-dm.c | 9 ++++++---
 kernel/cpu_pm.c                   | 9 ++++++++-
 2 files changed, 14 insertions(+), 4 deletions(-)
Signature · 1 line
2.47.3
#1 lore raw

PATCH 1/2 clocksource/drivers/timer-ti-dm: Unregister CPU PM notifier outside of the timer lock

omap_dm_timer_remove() calls cpu_pm_unregister_notifier() with dm_timer_lock held and interrupts disabled. Nothing sleeps in there today, but it pins the helper into a context where it can never be allowed to sleep, which is in the way of restoring the RCU grace period on the cpu_pm notifier chain.

Do the list lookup under the lock and move the unregister after the unlock. Nothing can race it at that point: remove() owns the device, and once the timer is off the list nobody can reach it anymore.

---
 drivers/clocksource/timer-ti-dm.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
drivers/clocksource/timer-ti-dm.c +6 -3
index bd06afb7d522..032e102bdd0a 100644
@@ -1530,7 +1530,7 @@ static int omap_dm_timer_probe(struct platform_device *pdev)
  */
 static void omap_dm_timer_remove(struct platform_device *pdev)
 {
-	struct dmtimer *timer;
+	struct dmtimer *timer, *found = NULL;
 	unsigned long flags;
 	int ret = -EINVAL;
 
@@ -1538,14 +1538,17 @@ static void omap_dm_timer_remove(struct platform_device *pdev)
 	list_for_each_entry(timer, &omap_timer_list, node)
 		if (!strcmp(dev_name(&timer->pdev->dev),
 			    dev_name(&pdev->dev))) {
-			if (!(timer->capability & OMAP_TIMER_ALWON))
-				cpu_pm_unregister_notifier(&timer->nb);
 			list_del(&timer->node);
+			found = timer;
 			ret = 0;
 			break;
 		}
 	spin_unlock_irqrestore(&dm_timer_lock, flags);
 
+	/* Unregister outside the lock: cpu_pm_unregister_notifier() may sleep. */
+	if (found && !(found->capability & OMAP_TIMER_ALWON))
+		cpu_pm_unregister_notifier(&found->nb);
+
 	pm_runtime_disable(&pdev->dev);
 
 	if (ret)
Signature · 1 line
2.47.3
#2 lore raw

Re: [PATCH 1/2] clocksource/drivers/timer-ti-dm: Unregister CPU PM notifier outside of the timer lock

Quoted 40 lines · Bradley Morgan

omap_dm_timer_remove() calls cpu_pm_unregister_notifier() with dm_timer_lock held and interrupts disabled. Nothing sleeps in there today, but it pins the helper into a context where it can never be allowed to sleep, which is in the way of restoring the RCU grace period on the cpu_pm notifier chain.

Do the list lookup under the lock and move the unregister after the unlock. Nothing can race it at that point: remove() owns the device, and once the timer is off the list nobody can reach it anymore.

---
 drivers/clocksource/timer-ti-dm.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
drivers/clocksource/timer-ti-dm.c +2 -3
index bd06afb7d522..032e102bdd0a 100644
@@ -1530,7 +1530,7 @@ static int omap_dm_timer_probe(struct platform_device *pdev)
  */
 static void omap_dm_timer_remove(struct platform_device *pdev)
 {
-       struct dmtimer *timer;
+       struct dmtimer *timer, *found = NULL;
        unsigned long flags;
        int ret = -EINVAL;
 
@@ -1538,14 +1538,17 @@ static void omap_dm_timer_remove(struct platform_device *pdev)
        list_for_each_entry(timer, &omap_timer_list, node)
                if (!strcmp(dev_name(&timer->pdev->dev),
                            dev_name(&pdev->dev))) {
-                       if (!(timer->capability & OMAP_TIMER_ALWON))
-                               cpu_pm_unregister_notifier(&timer->nb);
                        list_del(&timer->node);
+                       found = timer;
                        ret = 0;
                        break;
                }

drivers/clocksource/timer-ti-dm.c static void omap_dm_timer_remove(struct platform_device *pdev)

Quoted 10 lines
drivers/clocksource/timer-ti-dm.c +4 -1
        spin_unlock_irqrestore(&dm_timer_lock, flags);
 
+       /* Unregister outside the lock: cpu_pm_unregister_notifier() may sleep. */
+       if (found && !(found->capability & OMAP_TIMER_ALWON))
+               cpu_pm_unregister_notifier(&found->nb);
+
        pm_runtime_disable(&pdev->dev);
 
        if (ret)
--

Nobody seems to be interested in this patch, but Sashiko only sees pre-existing issues related to it:

https://sashiko.dev/#/patchset/20260813074343.26460-1-include%40grrlz.net

and the second patch in the series depends on it, so in the absence of any objections or concerns, I'm going to apply them both.

Thanks!

#3 lore raw

Re: [PATCH 1/2] clocksource/drivers/timer-ti-dm: Unregister CPU PM notifier outside of the timer lock

Quoted 59 lines · Rafael J. Wysocki (Intel)
Quoted 11 lines · Bradley Morgan

omap_dm_timer_remove() calls cpu_pm_unregister_notifier() with dm_timer_lock held and interrupts disabled. Nothing sleeps in there today, but it pins the helper into a context where it can never be allowed to sleep, which is in the way of restoring the RCU grace period on the cpu_pm notifier chain.

Do the list lookup under the lock and move the unregister after the unlock. Nothing can race it at that point: remove() owns the device, and once the timer is off the list nobody can reach it anymore.

Review

Quoted 6 lines
---
 drivers/clocksource/timer-ti-dm.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
(quoted diff) +0 -0

b/drivers/clocksource/timer-ti-dm.c

drivers/clocksource/timer-ti-dm.c static int omap_dm_timer_probe(struct

Quoted 4 lines

index bd06afb7d522..032e102bdd0a 100644

drivers/clocksource/timer-ti-dm.c +0 -0
@@ -1530,7 +1530,7 @@ static int omap_dm_timer_probe(struct

platform_device *pdev)

drivers/clocksource/timer-ti-dm.c static void omap_dm_timer_remove(struct

Quoted 9 lines
drivers/clocksource/timer-ti-dm.c +1 -1
  */
 static void omap_dm_timer_remove(struct platform_device *pdev)
 {
-       struct dmtimer *timer;
+       struct dmtimer *timer, *found = NULL;
        unsigned long flags;
        int ret = -EINVAL;
 
@@ -1538,14 +1538,17 @@ static void omap_dm_timer_remove(struct

platform_device *pdev)

drivers/clocksource/timer-ti-dm.c static void omap_dm_timer_remove(struct platform_device *pdev)

Quoted 13 lines
drivers/clocksource/timer-ti-dm.c +2 -2
        list_for_each_entry(timer, &omap_timer_list, node)
                if (!strcmp(dev_name(&timer->pdev->dev),
                            dev_name(&pdev->dev))) {
-                       if (!(timer->capability & OMAP_TIMER_ALWON))
-                               cpu_pm_unregister_notifier(&timer->nb);
                        list_del(&timer->node);
+                       found = timer;
                        ret = 0;
                        break;
                }
        spin_unlock_irqrestore(&dm_timer_lock, flags);
 
+       /* Unregister outside the lock: cpu_pm_unregister_notifier() may

sleep. */

drivers/clocksource/timer-ti-dm.c static void omap_dm_timer_remove(struct platform_device *pdev)

Quoted 7 lines
drivers/clocksource/timer-ti-dm.c +3 -1
+       if (found && !(found->capability & OMAP_TIMER_ALWON))
+               cpu_pm_unregister_notifier(&found->nb);
+
        pm_runtime_disable(&pdev->dev);
 
        if (ret)
--

Nobody seems to be interested in this patch, but Sashiko only sees pre-existing issues related to it:

Quoted 8 lines

https://sashiko.dev/#/patchset/20260813074343.26460-1-include%40grrlz.net

and the second patch in the series depends on it, so in the absence of any objections or concerns, I'm going to apply them both.

Thanks!

Ugh, I'll check / fix the pre existing issues now

Id like to wonder why Google didn't use their 3.8 flash, considering it's like 10x more capable
--- Thanks!
https://lore.kernel.org/all/[email protected]/

#4 lore raw

Re: [PATCH 1/2] clocksource/drivers/timer-ti-dm: Unregister CPU PM notifier outside of the timer lock

In reply to Rafael J. Wysocki (Intel)
Quoted 56 lines · Rafael J. Wysocki (Intel)
Quoted 40 lines · Bradley Morgan

omap_dm_timer_remove() calls cpu_pm_unregister_notifier() with dm_timer_lock held and interrupts disabled. Nothing sleeps in there today, but it pins the helper into a context where it can never be allowed to sleep, which is in the way of restoring the RCU grace period on the cpu_pm notifier chain.

Do the list lookup under the lock and move the unregister after the unlock. Nothing can race it at that point: remove() owns the device, and once the timer is off the list nobody can reach it anymore.

---
 drivers/clocksource/timer-ti-dm.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
drivers/clocksource/timer-ti-dm.c +2 -3
index bd06afb7d522..032e102bdd0a 100644
@@ -1530,7 +1530,7 @@ static int omap_dm_timer_probe(struct platform_device *pdev)
  */
 static void omap_dm_timer_remove(struct platform_device *pdev)
 {
-       struct dmtimer *timer;
+       struct dmtimer *timer, *found = NULL;
        unsigned long flags;
        int ret = -EINVAL;
 
@@ -1538,14 +1538,17 @@ static void omap_dm_timer_remove(struct platform_device *pdev)
        list_for_each_entry(timer, &omap_timer_list, node)
                if (!strcmp(dev_name(&timer->pdev->dev),
                            dev_name(&pdev->dev))) {
-                       if (!(timer->capability & OMAP_TIMER_ALWON))
-                               cpu_pm_unregister_notifier(&timer->nb);
                        list_del(&timer->node);
+                       found = timer;
                        ret = 0;
                        break;
                }

drivers/clocksource/timer-ti-dm.c static void omap_dm_timer_remove(struct platform_device *pdev)

Quoted 10 lines
drivers/clocksource/timer-ti-dm.c +4 -1
        spin_unlock_irqrestore(&dm_timer_lock, flags);
 
+       /* Unregister outside the lock: cpu_pm_unregister_notifier() may sleep. */
+       if (found && !(found->capability & OMAP_TIMER_ALWON))
+               cpu_pm_unregister_notifier(&found->nb);
+
        pm_runtime_disable(&pdev->dev);
 
        if (ret)
--

Nobody seems to be interested in this patch, but Sashiko only sees pre-existing issues related to it:

Quoted 4 lines

https://sashiko.dev/#/patchset/20260813074343.26460-1-include%40grrlz.net

and the second patch in the series depends on it, so in the absence of any objections or concerns, I'm going to apply them both.

And so applied as 7.4 material, thanks!

#5 lore raw

PATCH 2/2 PM: cpu: Restore synchronize_rcu() to cpu_pm_unregister_notifier()

In reply to Bradley Morgan

cpu_pm_notify() walks the notifier chain lockless under only rcu_read_lock(). That only works if a removed notifier block is not freed until every concurrent walker is done with it.

The chain used to be an atomic_notifier, whose unregister ends in synchronize_rcu(). Commit b2f6662ac08d ("PM: cpu: Make notifier chain use a raw_spinlock_t") switched it over to a raw_notifier, and in doing so replaced that with raw_notifier_chain_unregister(), which does not synchronize. The grace period quietly went away, so a driver that frees the memory holding its notifier block right after cpu_pm_unregister_notifier() returns can race with a concurrent walker:

cpu1 (idle exit) cpu2 (driver remove)
---------------- --------------------
cpu_pm_notify(CPU_PM_EXIT)

    rcu_read_lock()
    nb = rcu_dereference_raw(*nl)
                                    cpu_pm_unregister_notifier(&od->nb)
                                      unlink, no grace period
                                      remove() frees od (devm)
    nb->notifier_call(nb, ...)
      /* use after free */

The rcu_read_lock() on cpu1 does not stop cpu2 from freeing the block, and nothing else does. The read side is the idle exit path, so this can hit on any system where a driver with a cpu_pm notifier gets unbound.

Add the grace period back. It is only needed when a notifier was actually removed, so wait on success and return -ENOENT without waiting otherwise. The kerneldoc gets back the "may sleep" note that the same commit dropped.

Fixes b2f6662ac08d ("PM: cpu: Make notifier chain use a raw_spinlock_t")
---
 kernel/cpu_pm.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/cpu_pm.c +8 -1
index 7481fbb947d3..a2a598ad6e6d 100644
@@ -10,6 +10,7 @@
 #include <linux/cpu_pm.h>
 #include <linux/module.h>
 #include <linux/notifier.h>
+#include <linux/rcupdate.h>
 #include <linux/spinlock.h>
 #include <linux/syscore_ops.h>
 
@@ -76,7 +77,8 @@ EXPORT_SYMBOL_GPL(cpu_pm_register_notifier);
  *
  * Remove a driver from the CPU PM notifier list.
  *
- * This function has the same return conditions as raw_notifier_chain_unregister.
+ * This function may sleep, and has the same return conditions as
+ * raw_notifier_chain_unregister.
  */
 int cpu_pm_unregister_notifier(struct notifier_block *nb)
 {
@@ -86,6 +88,11 @@ int cpu_pm_unregister_notifier(struct notifier_block *nb)
 	raw_spin_lock_irqsave(&cpu_pm_notifier.lock, flags);
 	ret = raw_notifier_chain_unregister(&cpu_pm_notifier.chain, nb);
 	raw_spin_unlock_irqrestore(&cpu_pm_notifier.lock, flags);
+
+	/* Wait for the rcu_read_lock() walkers in cpu_pm_notify(). */
+	if (!ret)
+		synchronize_rcu();
+
 	return ret;
 }
 EXPORT_SYMBOL_GPL(cpu_pm_unregister_notifier);
Signature · 1 line
2.47.3
Acked-by on v1 2/2
#6 lore raw

Re: [PATCH 2/2] PM: cpu: Restore synchronize_rcu() to cpu_pm_unregister_notifier()

Quoted 30 lines · Bradley Morgan

cpu_pm_notify() walks the notifier chain lockless under only rcu_read_lock(). That only works if a removed notifier block is not freed until every concurrent walker is done with it.

The chain used to be an atomic_notifier, whose unregister ends in synchronize_rcu(). Commit b2f6662ac08d ("PM: cpu: Make notifier chain use a raw_spinlock_t") switched it over to a raw_notifier, and in doing so replaced that with raw_notifier_chain_unregister(), which does not synchronize. The grace period quietly went away, so a driver that frees the memory holding its notifier block right after cpu_pm_unregister_notifier() returns can race with a concurrent walker:

cpu1 (idle exit) cpu2 (driver remove)
---------------- --------------------
cpu_pm_notify(CPU_PM_EXIT)

    rcu_read_lock()
    nb = rcu_dereference_raw(*nl)
                                    cpu_pm_unregister_notifier(&od->nb)
                                      unlink, no grace period
                                      remove() frees od (devm)
    nb->notifier_call(nb, ...)
      /* use after free */

The rcu_read_lock() on cpu1 does not stop cpu2 from freeing the block, and nothing else does. The read side is the idle exit path, so this can hit on any system where a driver with a cpu_pm notifier gets unbound.

Quoted 8 lines

Add the grace period back. It is only needed when a notifier was actually removed, so wait on success and return -ENOENT without waiting otherwise. The kerneldoc gets back the "may sleep" note that the same commit dropped.

Fixes b2f6662ac08d ("PM: cpu: Make notifier chain use a raw_spinlock_t")

or if you want me to pick up this series, I need an ACK on the first patch.

Quoted 41 lines
---
 kernel/cpu_pm.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/cpu_pm.c +8 -2
index 7481fbb947d3..a2a598ad6e6d 100644
@@ -10,6 +10,7 @@
 #include <linux/cpu_pm.h>
 #include <linux/module.h>
 #include <linux/notifier.h>
+#include <linux/rcupdate.h>
 #include <linux/spinlock.h>
 #include <linux/syscore_ops.h>
 
@@ -76,7 +77,8 @@ EXPORT_SYMBOL_GPL(cpu_pm_register_notifier);
  *
  * Remove a driver from the CPU PM notifier list.
  *
- * This function has the same return conditions as raw_notifier_chain_unregister.
+ * This function may sleep, and has the same return conditions as
+ * raw_notifier_chain_unregister.
  */
 int cpu_pm_unregister_notifier(struct notifier_block *nb)
 {
@@ -86,6 +88,11 @@ int cpu_pm_unregister_notifier(struct notifier_block *nb)
        raw_spin_lock_irqsave(&cpu_pm_notifier.lock, flags);
        ret = raw_notifier_chain_unregister(&cpu_pm_notifier.chain, nb);
        raw_spin_unlock_irqrestore(&cpu_pm_notifier.lock, flags);
+
+       /* Wait for the rcu_read_lock() walkers in cpu_pm_notify(). */
+       if (!ret)
+               synchronize_rcu();
+
        return ret;
 }
 EXPORT_SYMBOL_GPL(cpu_pm_unregister_notifier);
--

2.47.3

#7 lore raw

Re: [PATCH 2/2] PM: cpu: Restore synchronize_rcu() to cpu_pm_unregister_notifier()

Quoted 37 lines · Rafael J. Wysocki (Intel)
Quoted 30 lines · Bradley Morgan

cpu_pm_notify() walks the notifier chain lockless under only rcu_read_lock(). That only works if a removed notifier block is not freed until every concurrent walker is done with it.

The chain used to be an atomic_notifier, whose unregister ends in synchronize_rcu(). Commit b2f6662ac08d ("PM: cpu: Make notifier chain use a raw_spinlock_t") switched it over to a raw_notifier, and in doing so replaced that with raw_notifier_chain_unregister(), which does not synchronize. The grace period quietly went away, so a driver that frees the memory holding its notifier block right after cpu_pm_unregister_notifier() returns can race with a concurrent walker:

cpu1 (idle exit) cpu2 (driver remove)
---------------- --------------------
cpu_pm_notify(CPU_PM_EXIT)

    rcu_read_lock()
    nb = rcu_dereference_raw(*nl)
                                    cpu_pm_unregister_notifier(&od->nb)
                                      unlink, no grace period
                                      remove() frees od (devm)
    nb->notifier_call(nb, ...)
      /* use after free */

The rcu_read_lock() on cpu1 does not stop cpu2 from freeing the block, and nothing else does. The read side is the idle exit path, so this can hit on any system where a driver with a cpu_pm notifier gets unbound.

Quoted 6 lines

Add the grace period back. It is only needed when a notifier was actually removed, so wait on success and return -ENOENT without waiting otherwise. The kerneldoc gets back the "may sleep" note that the same commit dropped.

Fixes b2f6662ac08d ("PM: cpu: Make notifier chain use a
Quoted 9 lines

raw_spinlock_t")

Quoted 2 lines

or if you want me to pick up this series, I need an ACK on the first patch.

Waiting game... (And I can't take it! I'm not maintainer of anything sadly, one day, one day, I'm manifesting it.. one day)

Quoted 29 lines
Quoted 17 lines
---
 kernel/cpu_pm.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/cpu_pm.c +1 -0
index 7481fbb947d3..a2a598ad6e6d 100644
@@ -10,6 +10,7 @@
 #include <linux/cpu_pm.h>
 #include <linux/module.h>
 #include <linux/notifier.h>
+#include <linux/rcupdate.h>
 #include <linux/spinlock.h>
 #include <linux/syscore_ops.h>
 

kernel/cpu_pm.c EXPORT_SYMBOL_GPL(cpu_pm_register_notifier); · line 79

Quoted 4 lines
kernel/cpu_pm.c +0 -1
@@ -76,7 +77,8 @@ EXPORT_SYMBOL_GPL(cpu_pm_register_notifier);
  *
  * Remove a driver from the CPU PM notifier list.
  *
- * This function has the same return conditions as

raw_notifier_chain_unregister.

kernel/cpu_pm.c int cpu_pm_unregister_notifier(struct notifier_block

Quoted 6 lines
kernel/cpu_pm.c +2 -0
+ * This function may sleep, and has the same return conditions as
+ * raw_notifier_chain_unregister.
  */
 int cpu_pm_unregister_notifier(struct notifier_block *nb)
 {
@@ -86,6 +88,11 @@ int cpu_pm_unregister_notifier(struct notifier_block

*nb)

Quoted 15 lines
Quoted 12 lines
kernel/cpu_pm.c +5 -1
        raw_spin_lock_irqsave(&cpu_pm_notifier.lock, flags);
        ret = raw_notifier_chain_unregister(&cpu_pm_notifier.chain, nb);
        raw_spin_unlock_irqrestore(&cpu_pm_notifier.lock, flags);
+
+       /* Wait for the rcu_read_lock() walkers in cpu_pm_notify(). */
+       if (!ret)
+               synchronize_rcu();
+
        return ret;
 }
 EXPORT_SYMBOL_GPL(cpu_pm_unregister_notifier);
--
Quoted 2 lines

2.47.3

Thanks!