Skip to contentSkip to conversation

cpufreq: Fix RCU reboot regression on x86 PIC machines

PATCH In review Ville Syrjala 4 participants 6 messages Oct 3, 2019 – Oct 10, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cpufreq: Fix RCU reboot regression on x86 PIC machines

From: Ville Syrjälä <[email protected]>

Since 4.20-rc1 my PIC machines no longer reboot/shutdown. I bisected this down to commit 45975c7d21a1 ("rcu: Define RCU-sched API in terms of RCU for Tree RCU PREEMPT builds").

I traced the hang into
-> cpufreq_suspend()
-> cpufreq_stop_governor()
-> cpufreq_dbs_governor_stop()
-> gov_clear_update_util()

    -> synchronize_sched()
     -> synchronize_rcu()

Only PREEMPT=y is affected for obvious reasons. The problem is limited to PIC machines since they mask off interrupts in i8259A_shutdown() (syscore_ops.shutdown() registered from device_initcall()).

I reported this long ago but no better fix has surfaced, hence sending out my initial workaround which I've been carrying around ever since. I just move cpufreq_core_init() to late_initcall() so the syscore_ops get registered in the oppsite order and thus the .shutdown() hooks get executed in the opposite order as well. Not 100% convinced this is safe (especially moving the cpufreq_global_kobject creation to late_initcall()) but I've not had any problems with it at least.

Here's the resulting change in initcall_debug:
+ PM: Calling cpufreq_suspend+0x0/0x100
PM: Calling mce_syscore_shutdown+0x0/0x10
PM: Calling i8259A_shutdown+0x0/0x10
- PM: Calling cpufreq_suspend+0x0/0x100
+ reboot: Restarting system
+ reboot: machine restart

Cc Paul E. McKenney <[email protected]>
Cc Andi Kleen <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
Cc Viresh Kumar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Fixes 45975c7d21a1 ("rcu: Define RCU-sched API in terms of RCU for Tree RCU PREEMPT builds")
---
 drivers/cpufreq/cpufreq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/cpufreq/cpufreq.c +1 -1
index c52d6fa32aac..6a8fb9b08e33 100644
@@ -2761,4 +2761,4 @@ static int __init cpufreq_core_init(void)
 	return 0;
 }
 module_param(off, int, 0444);
-core_initcall(cpufreq_core_init);
+late_initcall(cpufreq_core_init);
Signature · 1 line
2.21.0
#1 lore raw
Quoted 11 lines · CEST Ville Syrjala

From: Ville Syrjälä <[email protected]>

Since 4.20-rc1 my PIC machines no longer reboot/shutdown. I bisected this down to commit 45975c7d21a1 ("rcu: Define RCU-sched API in terms of RCU for Tree RCU PREEMPT builds").

I traced the hang into
-> cpufreq_suspend()
-> cpufreq_stop_governor()
-> cpufreq_dbs_governor_stop()
-> gov_clear_update_util()

Quoted 7 lines
    -> synchronize_sched()
     -> synchronize_rcu()

Only PREEMPT=y is affected for obvious reasons. The problem is limited to PIC machines since they mask off interrupts in i8259A_shutdown() (syscore_ops.shutdown() registered from device_initcall()).

Let me treat this as a fresh bug report. :-)

Quoted 1 line

I reported this long ago but no better fix has surfaced,

So I don't recall seeing the original report or if I did, I had not understood the problem then.

Quoted 8 lines

hence sending out my initial workaround which I've been carrying around ever since. I just move cpufreq_core_init() to late_initcall() so the syscore_ops get registered in the oppsite order and thus the .shutdown() hooks get executed in the opposite order as well. Not 100% convinced this is safe (especially moving the cpufreq_global_kobject creation to late_initcall()) but I've not had any problems with it at least.

The problem is a bug in cpufreq that shouldn't point its syscore shutdown callback pointer to cpufreq_suspend(), because the syscore stage is generally too lat to call that function and I'm not sure why this has not been causing any other issues to trigger (or maybe it did, but they were not reported).

Does the patch below work for you?

---
 drivers/base/core.c       |    3 +++
 drivers/cpufreq/cpufreq.c |   10 ----------
 2 files changed, 3 insertions(+), 10 deletions(-)
drivers/cpufreq/cpufreq.c +0 -10
Index: linux-pm/drivers/cpufreq/cpufreq.c
===================================================================
@@ -2737,14 +2737,6 @@ int cpufreq_unregister_driver(struct cpu
 }
 EXPORT_SYMBOL_GPL(cpufreq_unregister_driver);
 
-/*
- * Stop cpufreq at shutdown to make sure it isn't holding any locks
- * or mutexes when secondary CPUs are halted.
- */
-static struct syscore_ops cpufreq_syscore_ops = {
-	.shutdown = cpufreq_suspend,
-};
-
 struct kobject *cpufreq_global_kobject;
 EXPORT_SYMBOL(cpufreq_global_kobject);
 
@@ -2756,8 +2748,6 @@ static int __init cpufreq_core_init(void
 	cpufreq_global_kobject = kobject_create_and_add("cpufreq", &cpu_subsys.dev_root->kobj);
 	BUG_ON(!cpufreq_global_kobject);
 
-	register_syscore_ops(&cpufreq_syscore_ops);
-
 	return 0;
 }
 module_param(off, int, 0444);
drivers/base/core.c +3 -0
Index: linux-pm/drivers/base/core.c
===================================================================
@@ -9,6 +9,7 @@
  */
 
 #include <linux/acpi.h>
+#include <linux/cpufreq.h>
 #include <linux/device.h>
 #include <linux/err.h>
 #include <linux/fwnode.h>
@@ -3179,6 +3180,8 @@ void device_shutdown(void)
 	wait_for_device_probe();
 	device_block_probing();
 
+	cpufreq_suspend();
+
 	spin_lock(&devices_kset->list_lock);
 	/*
 	 * Walk the devices list backward, shutting down each in turn.
author Tested-by
#2 lore raw
Quoted 36 lines · Rafael J. Wysocki
Quoted 11 lines · CEST Ville Syrjala

From: Ville Syrjälä <[email protected]>

Since 4.20-rc1 my PIC machines no longer reboot/shutdown. I bisected this down to commit 45975c7d21a1 ("rcu: Define RCU-sched API in terms of RCU for Tree RCU PREEMPT builds").

I traced the hang into
-> cpufreq_suspend()
-> cpufreq_stop_governor()
-> cpufreq_dbs_governor_stop()
-> gov_clear_update_util()

Quoted 7 lines
    -> synchronize_sched()
     -> synchronize_rcu()

Only PREEMPT=y is affected for obvious reasons. The problem is limited to PIC machines since they mask off interrupts in i8259A_shutdown() (syscore_ops.shutdown() registered from device_initcall()).

Let me treat this as a fresh bug report. :-)

Quoted 1 line

I reported this long ago but no better fix has surfaced,

So I don't recall seeing the original report or if I did, I had not understood the problem then.

Quoted 8 lines

hence sending out my initial workaround which I've been carrying around ever since. I just move cpufreq_core_init() to late_initcall() so the syscore_ops get registered in the oppsite order and thus the .shutdown() hooks get executed in the opposite order as well. Not 100% convinced this is safe (especially moving the cpufreq_global_kobject creation to late_initcall()) but I've not had any problems with it at least.

Quoted 6 lines

The problem is a bug in cpufreq that shouldn't point its syscore shutdown callback pointer to cpufreq_suspend(), because the syscore stage is generally too lat to call that function and I'm not sure why this has not been causing any other issues to trigger (or maybe it did, but they were not reported).

Does the patch below work for you?

It does. Thanks.

Feel free to slap on

if you want to go with that.

Quoted 57 lines
---
 drivers/base/core.c       |    3 +++
 drivers/cpufreq/cpufreq.c |   10 ----------
 2 files changed, 3 insertions(+), 10 deletions(-)

Index: linux-pm/drivers/cpufreq/cpufreq.c
===================================================================

drivers/cpufreq/cpufreq.c +0 -10
@@ -2737,14 +2737,6 @@ int cpufreq_unregister_driver(struct cpu
 }
 EXPORT_SYMBOL_GPL(cpufreq_unregister_driver);
 
-/*
- * Stop cpufreq at shutdown to make sure it isn't holding any locks
- * or mutexes when secondary CPUs are halted.
- */
-static struct syscore_ops cpufreq_syscore_ops = {
-	.shutdown = cpufreq_suspend,
-};
-
 struct kobject *cpufreq_global_kobject;
 EXPORT_SYMBOL(cpufreq_global_kobject);
 
@@ -2756,8 +2748,6 @@ static int __init cpufreq_core_init(void
 	cpufreq_global_kobject = kobject_create_and_add("cpufreq", &cpu_subsys.dev_root->kobj);
 	BUG_ON(!cpufreq_global_kobject);
 
-	register_syscore_ops(&cpufreq_syscore_ops);
-
 	return 0;
 }
 module_param(off, int, 0444);

Index: linux-pm/drivers/base/core.c
===================================================================

drivers/base/core.c +3 -0
@@ -9,6 +9,7 @@
  */
 
 #include <linux/acpi.h>
+#include <linux/cpufreq.h>
 #include <linux/device.h>
 #include <linux/err.h>
 #include <linux/fwnode.h>
@@ -3179,6 +3180,8 @@ void device_shutdown(void)
 	wait_for_device_probe();
 	device_block_probing();
 
+	cpufreq_suspend();
+
 	spin_lock(&devices_kset->list_lock);
 	/*
 	 * Walk the devices list backward, shutting down each in turn.
Signature · 2 lines
Ville Syrjälä
Intel
#3 lore raw
Quoted 43 lines · Ville Syrjälä
Quoted 41 lines · Rafael J. Wysocki
Quoted 11 lines · CEST Ville Syrjala

From: Ville Syrjälä <[email protected]>

Since 4.20-rc1 my PIC machines no longer reboot/shutdown. I bisected this down to commit 45975c7d21a1 ("rcu: Define RCU-sched API in terms of RCU for Tree RCU PREEMPT builds").

I traced the hang into
-> cpufreq_suspend()
-> cpufreq_stop_governor()
-> cpufreq_dbs_governor_stop()
-> gov_clear_update_util()

Quoted 7 lines
    -> synchronize_sched()
     -> synchronize_rcu()

Only PREEMPT=y is affected for obvious reasons. The problem is limited to PIC machines since they mask off interrupts in i8259A_shutdown() (syscore_ops.shutdown() registered from device_initcall()).

Let me treat this as a fresh bug report. :-)

Quoted 1 line

I reported this long ago but no better fix has surfaced,

So I don't recall seeing the original report or if I did, I had not understood the problem then.

Quoted 8 lines

hence sending out my initial workaround which I've been carrying around ever since. I just move cpufreq_core_init() to late_initcall() so the syscore_ops get registered in the oppsite order and thus the .shutdown() hooks get executed in the opposite order as well. Not 100% convinced this is safe (especially moving the cpufreq_global_kobject creation to late_initcall()) but I've not had any problems with it at least.

The problem is a bug in cpufreq that shouldn't point its syscore shutdown callback pointer to cpufreq_suspend(), because the syscore stage is generally too lat to call that function and I'm not sure why this has not been causing any other issues to trigger (or maybe it did, but they were not reported).

Quoted 8 lines
Quoted 1 line

Does the patch below work for you?

It does. Thanks.

Feel free to slap on

if you want to go with that.

I will, thank you!

#4 lore raw

PATCH 1/1 cpufreq: Avoid cpufreq_suspend() deadlock on system shutdown

From: Rafael J. Wysocki <[email protected]>

It is incorrect to set the cpufreq syscore shutdown callback pointer to cpufreq_suspend(), because that function cannot be run in the syscore stage of system shutdown for two reasons: (a) it may attempt to carry out actions depending on devices that have already been shut down at that point and (b) the RCU synchronization carried out by it may not be able to make progress then.

The latter issue has been present since commit 45975c7d21a1 ("rcu:
Define RCU-sched API in terms of RCU for Tree RCU PREEMPT builds"), but the former one has always been there regardless.

Fix that by dropping cpufreq_syscore_ops altogether and making device_shutdown() call cpufreq_suspend() directly before shutting down devices, which is along the lines of what system-wide power management does.

Fixes 45975c7d21a1 ("rcu: Define RCU-sched API in terms of RCU for Tree RCU PREEMPT builds")
---
 drivers/base/core.c       |    3 +++
 drivers/cpufreq/cpufreq.c |   10 ----------
 2 files changed, 3 insertions(+), 10 deletions(-)
drivers/cpufreq/cpufreq.c +0 -10
Index: linux-pm/drivers/cpufreq/cpufreq.c
===================================================================
@@ -2737,14 +2737,6 @@ int cpufreq_unregister_driver(struct cpu
 }
 EXPORT_SYMBOL_GPL(cpufreq_unregister_driver);
 
-/*
- * Stop cpufreq at shutdown to make sure it isn't holding any locks
- * or mutexes when secondary CPUs are halted.
- */
-static struct syscore_ops cpufreq_syscore_ops = {
-	.shutdown = cpufreq_suspend,
-};
-
 struct kobject *cpufreq_global_kobject;
 EXPORT_SYMBOL(cpufreq_global_kobject);
 
@@ -2756,8 +2748,6 @@ static int __init cpufreq_core_init(void
 	cpufreq_global_kobject = kobject_create_and_add("cpufreq", &cpu_subsys.dev_root->kobj);
 	BUG_ON(!cpufreq_global_kobject);
 
-	register_syscore_ops(&cpufreq_syscore_ops);
-
 	return 0;
 }
 module_param(off, int, 0444);
drivers/base/core.c +3 -0
Index: linux-pm/drivers/base/core.c
===================================================================
@@ -9,6 +9,7 @@
  */
 
 #include <linux/acpi.h>
+#include <linux/cpufreq.h>
 #include <linux/device.h>
 #include <linux/err.h>
 #include <linux/fwnode.h>
@@ -3179,6 +3180,8 @@ void device_shutdown(void)
 	wait_for_device_probe();
 	device_block_probing();
 
+	cpufreq_suspend();
+
 	spin_lock(&devices_kset->list_lock);
 	/*
 	 * Walk the devices list backward, shutting down each in turn.
maintainer Acked-by on v1 1/1
#5 lore raw

Re: [PATCH] cpufreq: Avoid cpufreq_suspend() deadlock on system shutdown

Quoted 18 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

It is incorrect to set the cpufreq syscore shutdown callback pointer to cpufreq_suspend(), because that function cannot be run in the syscore stage of system shutdown for two reasons: (a) it may attempt to carry out actions depending on devices that have already been shut down at that point and (b) the RCU synchronization carried out by it may not be able to make progress then.

The latter issue has been present since commit 45975c7d21a1 ("rcu:
Define RCU-sched API in terms of RCU for Tree RCU PREEMPT builds"), but the former one has always been there regardless.

Fix that by dropping cpufreq_syscore_ops altogether and making device_shutdown() call cpufreq_suspend() directly before shutting down devices, which is along the lines of what system-wide power management does.

Quoted 8 lines
Fixes 45975c7d21a1 ("rcu: Define RCU-sched API in terms of RCU for Tree RCU PREEMPT builds")
---
 drivers/base/core.c       |    3 +++
 drivers/cpufreq/cpufreq.c |   10 ----------
 2 files changed, 3 insertions(+), 10 deletions(-)
Signature · 1 line
viresh