Skip to contentSkip to conversation

Rework disabling of interrupts during suspend-resume

RFC PATCH In review Rafael J. Wysocki 13 participants 344 messages Feb 22, 2009 – Mar 14, 2009 Page 1 of 3

Subsystems (10)
lore
Share & tools
lore
#0 lore raw

cover Rework disabling of interrupts during suspend-resume

Hi,

The following two patches modify the way in which we handle disabling interrupts during suspend and enabling them during resume. Namely, currently interrupts are disabled on the boot CPU as soon as the nonboot CPUs have been disabled, which doesn't allow device drivers' "late" suspend and "early" resume callbacks to sleep. Among other things this means they cannot execute ACPI AML routines, which leads to problems with suspend-resume of PCI devices, as recently discussed on this list.

1/2 is based on an earlier patch from Linus and it only splits up sysdev_[suspend|resume] from the ["late suspend|"early" resume'] of devices.

2/2 actually modifies the [suspend|hibernation] and resume code, as well as the other code using the device PM framework.

The patches have been initially tested and they don't appear to break suspend on my boxes, but this is the first approximation only. In particular, I'm not sure if I did the XEN, kexec and APM parts right, so people with experience in these areas are gently requested to have a look and tell me if there's anything to fix in there.

Moreover, the real purpose of these changes is to be able to execute the "late" suspend and "early" resume device callbacks with timer interrupts enabled, so that they can use mutexes etc. However, x86 currently doesn't set the IRQF_TIMER flag and I need to make it do so before going further in this direction and changing the PCI PM framework to take advantage of the $subject changes, for example. So, I need to know how to modify x86 timer code so that the IRQF_TIMER flag is set by it.

Comments welcome.

Thanks,
Rafael

#1 lore raw

RFC PATCH 1/2 PM: Split up sysdev_[suspend|resume] from device_power_[down|up]

From: Rafael J. Wysocki <[email protected]>

Move the sysdev_suspend/resume from the callee to the callers, with no real change in semantics, so that we can rework the disabling of interrupts during suspend/hibernation.

This is based on an earlier patch from Linus.

---
 arch/x86/kernel/apm_32.c  |    4 ++++
 drivers/base/base.h       |    2 --
 drivers/base/power/main.c |    3 ---
 drivers/xen/manage.c      |    8 ++++++++
 include/linux/pm.h        |    2 ++
 kernel/kexec.c            |    7 +++++++
 kernel/power/disk.c       |   11 +++++++++++
 kernel/power/main.c       |    8 ++++++--
 8 files changed, 38 insertions(+), 7 deletions(-)
arch/x86/kernel/apm_32.c +4 -0
Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================
@@ -1192,6 +1192,7 @@ static int suspend(int vetoable)
 	device_suspend(PMSG_SUSPEND);
 	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+	sysdev_suspend(PMSG_SUSPEND);
 
 	local_irq_enable();
 
@@ -1208,6 +1209,7 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 	local_irq_enable();
 	device_resume(PMSG_RESUME);
@@ -1228,6 +1230,7 @@ static void standby(void)
 
 	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+	sysdev_suspend(PMSG_SUSPEND);
 	local_irq_enable();
 
 	err = set_system_power_state(APM_STATE_STANDBY);
@@ -1235,6 +1238,7 @@ static void standby(void)
 		apm_error("standby", err);
 
 	local_irq_disable();
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 	local_irq_enable();
 }
drivers/base/power/main.c +0 -3
Index: linux-2.6/drivers/base/power/main.c
===================================================================
@@ -333,7 +333,6 @@ static void dpm_power_up(pm_message_t st
  */
 void device_power_up(pm_message_t state)
 {
-	sysdev_resume();
 	dpm_power_up(state);
 }
 EXPORT_SYMBOL_GPL(device_power_up);
@@ -577,8 +576,6 @@ int device_power_down(pm_message_t state
 		}
 		dev->power.status = DPM_OFF_IRQ;
 	}
-	if (!error)
-		error = sysdev_suspend(state);
 	if (error)
 		dpm_power_up(resume_event(state));
 	return error;
drivers/xen/manage.c +8 -0
Index: linux-2.6/drivers/xen/manage.c
===================================================================
@@ -45,6 +45,13 @@ static int xen_suspend(void *data)
 		       err);
 		return err;
 	}
+	err = sysdev_suspend(PMSG_SUSPEND);
+	if (err) {
+		printk(KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n",
+			err);
+		device_power_up(PMSG_RESUME);
+		return err;
+	}
 
 	xen_mm_pin_all();
 	gnttab_suspend();
@@ -61,6 +68,7 @@ static int xen_suspend(void *data)
 	gnttab_resume();
 	xen_mm_unpin_all();
 
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 
 	if (!*cancelled) {
kernel/kexec.c +7 -0
Index: linux-2.6/kernel/kexec.c
===================================================================
@@ -1465,6 +1465,11 @@ int kernel_kexec(void)
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
 			goto Enable_irqs;
+
+		/* Suspend system devices */
+		error = sysdev_suspend(PMSG_FREEZE);
+		if (error)
+			goto Power_up_devices;
 	} else
 #endif
 	{
@@ -1477,6 +1482,8 @@ int kernel_kexec(void)
 
 #ifdef CONFIG_KEXEC_JUMP
 	if (kexec_image->preserve_context) {
+		sysdev_resume();
+ Power_up_devices:
 		device_power_up(PMSG_RESTORE);
  Enable_irqs:
 		local_irq_enable();
kernel/power/disk.c +11 -0
Index: linux-2.6/kernel/power/disk.c
===================================================================
@@ -227,6 +227,12 @@ static int create_image(int platform_mod
 			"aborting hibernation\n");
 		goto Enable_irqs;
 	}
+	sysdev_suspend(PMSG_FREEZE);
+	if (error) {
+		printk(KERN_ERR "PM: Some devices failed to power down, "
+			"aborting hibernation\n");
+		goto Power_up_devices;
+	}
 
 	if (hibernation_test(TEST_CORE))
 		goto Power_up;
@@ -242,9 +248,11 @@ static int create_image(int platform_mod
 	if (!in_suspend)
 		platform_leave(platform_mode);
  Power_up:
+	sysdev_resume();
 	/* NOTE:  device_power_up() is just a resume() for devices
 	 * that suspended with irqs off ... no overall powerup.
 	 */
+ Power_up_devices:
 	device_power_up(in_suspend ?
 		(error ? PMSG_RECOVER : PMSG_THAW) : PMSG_RESTORE);
  Enable_irqs:
@@ -335,6 +343,7 @@ static int resume_target_kernel(void)
 			"aborting resume\n");
 		goto Enable_irqs;
 	}
+	sysdev_suspend(PMSG_QUIESCE);
 	/* We'll ignore saved state, but this gets preempt count (etc) right */
 	save_processor_state();
 	error = restore_highmem();
@@ -357,6 +366,7 @@ static int resume_target_kernel(void)
 	swsusp_free();
 	restore_processor_state();
 	touch_softlockup_watchdog();
+	sysdev_resume();
 	device_power_up(PMSG_RECOVER);
  Enable_irqs:
 	local_irq_enable();
@@ -440,6 +450,7 @@ int hibernation_platform_enter(void)
 	local_irq_disable();
 	error = device_power_down(PMSG_HIBERNATE);
 	if (!error) {
+		sysdev_suspend(PMSG_HIBERNATE);
 		hibernation_ops->enter();
 		/* We should never get here */
 		while (1);
kernel/power/main.c +6 -2
Index: linux-2.6/kernel/power/main.c
===================================================================
@@ -298,8 +298,12 @@ static int suspend_enter(suspend_state_t
 		goto Done;
 	}
 
-	if (!suspend_test(TEST_CORE))
-		error = suspend_ops->enter(state);
+	error = sysdev_suspend(PMSG_SUSPEND);
+	if (!error) {
+		if (!suspend_test(TEST_CORE))
+			error = suspend_ops->enter(state);
+		sysdev_resume();
+	}
 
 	device_power_up(PMSG_RESUME);
  Done:
drivers/base/base.h +0 -2
Index: linux-2.6/drivers/base/base.h
===================================================================
@@ -88,8 +88,6 @@ extern void driver_detach(struct device_
 extern int driver_probe_device(struct device_driver *drv, struct device *dev);
 
 extern void sysdev_shutdown(void);
-extern int sysdev_suspend(pm_message_t state);
-extern int sysdev_resume(void);
 
 extern char *make_class_name(const char *name, struct kobject *kobj);
 
include/linux/pm.h +2 -0
Index: linux-2.6/include/linux/pm.h
===================================================================
@@ -381,10 +381,12 @@ struct dev_pm_info {
 
 #ifdef CONFIG_PM_SLEEP
 extern void device_pm_lock(void);
+extern int sysdev_resume(void);
 extern void device_power_up(pm_message_t state);
 extern void device_resume(pm_message_t state);
 
 extern void device_pm_unlock(void);
+extern int sysdev_suspend(pm_message_t state);
 extern int device_power_down(pm_message_t state);
 extern int device_suspend(pm_message_t state);
 extern int device_prepare_suspend(pm_message_t state);
#2 lore raw

RFC PATCH 1/2 PM: Split up sysdev_[suspend|resume] from device_power_[down|up]

In reply to Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Move the sysdev_suspend/resume from the callee to the callers, with no real change in semantics, so that we can rework the disabling of interrupts during suspend/hibernation.

This is based on an earlier patch from Linus.

---
 arch/x86/kernel/apm_32.c  |    4 ++++
 drivers/base/base.h       |    2 --
 drivers/base/power/main.c |    3 ---
 drivers/xen/manage.c      |    8 ++++++++
 include/linux/pm.h        |    2 ++
 kernel/kexec.c            |    7 +++++++
 kernel/power/disk.c       |   11 +++++++++++
 kernel/power/main.c       |    8 ++++++--
 8 files changed, 38 insertions(+), 7 deletions(-)
arch/x86/kernel/apm_32.c +4 -0
Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================
@@ -1192,6 +1192,7 @@ static int suspend(int vetoable)
 	device_suspend(PMSG_SUSPEND);
 	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+	sysdev_suspend(PMSG_SUSPEND);
 
 	local_irq_enable();
 
@@ -1208,6 +1209,7 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 	local_irq_enable();
 	device_resume(PMSG_RESUME);
@@ -1228,6 +1230,7 @@ static void standby(void)
 
 	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+	sysdev_suspend(PMSG_SUSPEND);
 	local_irq_enable();
 
 	err = set_system_power_state(APM_STATE_STANDBY);
@@ -1235,6 +1238,7 @@ static void standby(void)
 		apm_error("standby", err);
 
 	local_irq_disable();
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 	local_irq_enable();
 }
drivers/base/power/main.c +0 -3
Index: linux-2.6/drivers/base/power/main.c
===================================================================
@@ -333,7 +333,6 @@ static void dpm_power_up(pm_message_t st
  */
 void device_power_up(pm_message_t state)
 {
-	sysdev_resume();
 	dpm_power_up(state);
 }
 EXPORT_SYMBOL_GPL(device_power_up);
@@ -577,8 +576,6 @@ int device_power_down(pm_message_t state
 		}
 		dev->power.status = DPM_OFF_IRQ;
 	}
-	if (!error)
-		error = sysdev_suspend(state);
 	if (error)
 		dpm_power_up(resume_event(state));
 	return error;
drivers/xen/manage.c +8 -0
Index: linux-2.6/drivers/xen/manage.c
===================================================================
@@ -45,6 +45,13 @@ static int xen_suspend(void *data)
 		       err);
 		return err;
 	}
+	err = sysdev_suspend(PMSG_SUSPEND);
+	if (err) {
+		printk(KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n",
+			err);
+		device_power_up(PMSG_RESUME);
+		return err;
+	}
 
 	xen_mm_pin_all();
 	gnttab_suspend();
@@ -61,6 +68,7 @@ static int xen_suspend(void *data)
 	gnttab_resume();
 	xen_mm_unpin_all();
 
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 
 	if (!*cancelled) {
kernel/kexec.c +7 -0
Index: linux-2.6/kernel/kexec.c
===================================================================
@@ -1465,6 +1465,11 @@ int kernel_kexec(void)
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
 			goto Enable_irqs;
+
+		/* Suspend system devices */
+		error = sysdev_suspend(PMSG_FREEZE);
+		if (error)
+			goto Power_up_devices;
 	} else
 #endif
 	{
@@ -1477,6 +1482,8 @@ int kernel_kexec(void)
 
 #ifdef CONFIG_KEXEC_JUMP
 	if (kexec_image->preserve_context) {
+		sysdev_resume();
+ Power_up_devices:
 		device_power_up(PMSG_RESTORE);
  Enable_irqs:
 		local_irq_enable();
kernel/power/disk.c +11 -0
Index: linux-2.6/kernel/power/disk.c
===================================================================
@@ -227,6 +227,12 @@ static int create_image(int platform_mod
 			"aborting hibernation\n");
 		goto Enable_irqs;
 	}
+	sysdev_suspend(PMSG_FREEZE);
+	if (error) {
+		printk(KERN_ERR "PM: Some devices failed to power down, "
+			"aborting hibernation\n");
+		goto Power_up_devices;
+	}
 
 	if (hibernation_test(TEST_CORE))
 		goto Power_up;
@@ -242,9 +248,11 @@ static int create_image(int platform_mod
 	if (!in_suspend)
 		platform_leave(platform_mode);
  Power_up:
+	sysdev_resume();
 	/* NOTE:  device_power_up() is just a resume() for devices
 	 * that suspended with irqs off ... no overall powerup.
 	 */
+ Power_up_devices:
 	device_power_up(in_suspend ?
 		(error ? PMSG_RECOVER : PMSG_THAW) : PMSG_RESTORE);
  Enable_irqs:
@@ -335,6 +343,7 @@ static int resume_target_kernel(void)
 			"aborting resume\n");
 		goto Enable_irqs;
 	}
+	sysdev_suspend(PMSG_QUIESCE);
 	/* We'll ignore saved state, but this gets preempt count (etc) right */
 	save_processor_state();
 	error = restore_highmem();
@@ -357,6 +366,7 @@ static int resume_target_kernel(void)
 	swsusp_free();
 	restore_processor_state();
 	touch_softlockup_watchdog();
+	sysdev_resume();
 	device_power_up(PMSG_RECOVER);
  Enable_irqs:
 	local_irq_enable();
@@ -440,6 +450,7 @@ int hibernation_platform_enter(void)
 	local_irq_disable();
 	error = device_power_down(PMSG_HIBERNATE);
 	if (!error) {
+		sysdev_suspend(PMSG_HIBERNATE);
 		hibernation_ops->enter();
 		/* We should never get here */
 		while (1);
kernel/power/main.c +6 -2
Index: linux-2.6/kernel/power/main.c
===================================================================
@@ -298,8 +298,12 @@ static int suspend_enter(suspend_state_t
 		goto Done;
 	}
 
-	if (!suspend_test(TEST_CORE))
-		error = suspend_ops->enter(state);
+	error = sysdev_suspend(PMSG_SUSPEND);
+	if (!error) {
+		if (!suspend_test(TEST_CORE))
+			error = suspend_ops->enter(state);
+		sysdev_resume();
+	}
 
 	device_power_up(PMSG_RESUME);
  Done:
drivers/base/base.h +0 -2
Index: linux-2.6/drivers/base/base.h
===================================================================
@@ -88,8 +88,6 @@ extern void driver_detach(struct device_
 extern int driver_probe_device(struct device_driver *drv, struct device *dev);
 
 extern void sysdev_shutdown(void);
-extern int sysdev_suspend(pm_message_t state);
-extern int sysdev_resume(void);
 
 extern char *make_class_name(const char *name, struct kobject *kobj);
 
include/linux/pm.h +2 -0
Index: linux-2.6/include/linux/pm.h
===================================================================
@@ -381,10 +381,12 @@ struct dev_pm_info {
 
 #ifdef CONFIG_PM_SLEEP
 extern void device_pm_lock(void);
+extern int sysdev_resume(void);
 extern void device_power_up(pm_message_t state);
 extern void device_resume(pm_message_t state);
 
 extern void device_pm_unlock(void);
+extern int sysdev_suspend(pm_message_t state);
 extern int device_power_down(pm_message_t state);
 extern int device_suspend(pm_message_t state);
 extern int device_prepare_suspend(pm_message_t state);
#3 lore raw

Re: [RFC][PATCH 1/2] PM: Split up sysdev_[suspend|resume] from device_power_[down|up]

Quoted 37 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Move the sysdev_suspend/resume from the callee to the callers, with no real change in semantics, so that we can rework the disabling of interrupts during suspend/hibernation.

This is based on an earlier patch from Linus.

...

arch/x86/kernel/apm_32.c +3 -0
@@ -1192,6 +1192,7 @@ static int suspend(int vetoable)
 	device_suspend(PMSG_SUSPEND);
 	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+	sysdev_suspend(PMSG_SUSPEND);
 
 	local_irq_enable();
 
@@ -1208,6 +1209,7 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 	local_irq_enable();
 	device_resume(PMSG_RESUME);
@@ -1228,6 +1230,7 @@ static void standby(void)
 
 	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+	sysdev_suspend(PMSG_SUSPEND);
 	local_irq_enable();
 
 	err = set_system_power_state(APM_STATE_STANDBY);

arch/x86/kernel/apm_32.c static void standby(void) · line 1244

Quoted 8 lines
arch/x86/kernel/apm_32.c +1 -0
@@ -1235,6 +1238,7 @@ static void standby(void)
 		apm_error("standby", err);
 
 	local_irq_disable();
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 	local_irq_enable();
 }
...

This causes the following build error with CONFIG_APM=m:

<-- snip -->

...
MODPOST 2586 modules
ERROR: "sysdev_resume" [arch/x86/kernel/apm.ko] undefined!
ERROR: "sysdev_suspend" [arch/x86/kernel/apm.ko] undefined! make[2]: *** [__modpost] Error 1

<-- snip -->

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#4 lore raw

Re: [RFC][PATCH 1/2] PM: Split up sysdev_[suspend|resume] from device_power_[down|up]

Quoted 5 lines · Adrian Bunk

...
MODPOST 2586 modules
ERROR: "sysdev_resume" [arch/x86/kernel/apm.ko] undefined!
ERROR: "sysdev_suspend" [arch/x86/kernel/apm.ko] undefined! make[2]: *** [__modpost] Error 1

Ahh. device_power_[down|up] were EXPORT_SYMBOL_GPL, so now that we've split them, so must sysdev_[suspend|resume] be.

Does this fix it?

Linus

---
 drivers/base/sys.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
drivers/base/sys.c +2 -0
index c98c31e..ef2055e 100644
@@ -432,6 +432,7 @@ aux_driver:
 	}
 	return ret;
 }
+EXPORT_SYMBOL_GPL(sysdev_suspend);
 
 
 /**
@@ -463,6 +464,7 @@ int sysdev_resume(void)
 	}
 	return 0;
 }
+EXPORT_SYMBOL_GPL(sysdev_resume);
 
 
 int __init system_bus_init(void)
#5 lore raw

Re: [RFC][PATCH 1/2] PM: Split up sysdev_[suspend|resume] from device_power_[down|up]

Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Adrian Bunk

...
MODPOST 2586 modules
ERROR: "sysdev_resume" [arch/x86/kernel/apm.ko] undefined!
ERROR: "sysdev_suspend" [arch/x86/kernel/apm.ko] undefined! make[2]: *** [__modpost] Error 1

Quoted 4 lines

Ahh. device_power_[down|up] were EXPORT_SYMBOL_GPL, so now that we've split them, so must sysdev_[suspend|resume] be.

Does this fix it?

I just hit the same issue in -tip testing and did the same fix:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git core/urgent

Ingo

------------------>
Ingo Molnar (1):
PM: Split up sysdev_[suspend|resume] from device_power_[down|up], fix

 drivers/base/sys.c |    7 ++-----
 1 files changed, 2 insertions(+), 5 deletions(-)
drivers/base/sys.c +2 -5
index c98c31e..b428c8c 100644
@@ -303,7 +303,6 @@ void sysdev_unregister(struct sys_device * sysdev)
  *	is guaranteed by virtue of the fact that child devices are registered
  *	after their parents.
  */
-
 void sysdev_shutdown(void)
 {
 	struct sysdev_class * cls;
@@ -363,7 +362,6 @@ static void __sysdev_resume(struct sys_device *dev)
  *	This is only called by the device PM core, so we let them handle
  *	all synchronization.
  */
-
 int sysdev_suspend(pm_message_t state)
 {
 	struct sysdev_class * cls;
@@ -432,7 +430,7 @@ aux_driver:
 	}
 	return ret;
 }
-
+EXPORT_SYMBOL_GPL(sysdev_suspend);
 
 /**
  *	sysdev_resume - Bring system devices back to life.
@@ -442,7 +440,6 @@ aux_driver:
  *
  *	Note: Interrupts are disabled when called.
  */
-
 int sysdev_resume(void)
 {
 	struct sysdev_class * cls;
@@ -463,7 +460,7 @@ int sysdev_resume(void)
 	}
 	return 0;
 }
-
+EXPORT_SYMBOL_GPL(sysdev_resume);
 
 int __init system_bus_init(void)
 {
#6 lore raw

Re: [RFC][PATCH 1/2] PM: Split up sysdev_[suspend|resume] from device_power_[down|up]

In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Adrian Bunk

...
MODPOST 2586 modules
ERROR: "sysdev_resume" [arch/x86/kernel/apm.ko] undefined!
ERROR: "sysdev_suspend" [arch/x86/kernel/apm.ko] undefined! make[2]: *** [__modpost] Error 1

Quoted 4 lines

Ahh. device_power_[down|up] were EXPORT_SYMBOL_GPL, so now that we've split them, so must sysdev_[suspend|resume] be.

Does this fix it?

Thanks, works fine.

Quoted 18 lines

Linus

---
 drivers/base/sys.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
drivers/base/sys.c +1 -0
index c98c31e..ef2055e 100644
@@ -432,6 +432,7 @@ aux_driver:
 	}
 	return ret;
 }
+EXPORT_SYMBOL_GPL(sysdev_suspend);
 
 
 /**

drivers/base/sys.c int sysdev_resume(void) · line 470

Quoted 7 lines
drivers/base/sys.c +1 -0
@@ -463,6 +464,7 @@ int sysdev_resume(void)
 	}
 	return 0;
 }
+EXPORT_SYMBOL_GPL(sysdev_resume);
 
 
 int __init system_bus_init(void)

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#7 lore raw

Re: [RFC][PATCH 1/2] PM: Split up sysdev_[suspend|resume] from device_power_[down|up]

In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Adrian Bunk

...
MODPOST 2586 modules
ERROR: "sysdev_resume" [arch/x86/kernel/apm.ko] undefined!
ERROR: "sysdev_suspend" [arch/x86/kernel/apm.ko] undefined! make[2]: *** [__modpost] Error 1

Quoted 4 lines

Ahh. device_power_[down|up] were EXPORT_SYMBOL_GPL, so now that we've split them, so must sysdev_[suspend|resume] be.

Does this fix it?

Thanks, works fine.

Quoted 18 lines

Linus

---
 drivers/base/sys.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
drivers/base/sys.c +1 -0
index c98c31e..ef2055e 100644
@@ -432,6 +432,7 @@ aux_driver:
 	}
 	return ret;
 }
+EXPORT_SYMBOL_GPL(sysdev_suspend);
 
 
 /**

drivers/base/sys.c int sysdev_resume(void) · line 470

Quoted 7 lines
drivers/base/sys.c +1 -0
@@ -463,6 +464,7 @@ int sysdev_resume(void)
 	}
 	return 0;
 }
+EXPORT_SYMBOL_GPL(sysdev_resume);
 
 
 int __init system_bus_init(void)

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#8 lore raw

Re: [RFC][PATCH 1/2] PM: Split up sysdev_[suspend|resume] from device_power_[down|up]

In reply to Rafael J. Wysocki
Quoted 37 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Move the sysdev_suspend/resume from the callee to the callers, with no real change in semantics, so that we can rework the disabling of interrupts during suspend/hibernation.

This is based on an earlier patch from Linus.

...

arch/x86/kernel/apm_32.c +3 -0
@@ -1192,6 +1192,7 @@ static int suspend(int vetoable)
 	device_suspend(PMSG_SUSPEND);
 	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+	sysdev_suspend(PMSG_SUSPEND);
 
 	local_irq_enable();
 
@@ -1208,6 +1209,7 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 	local_irq_enable();
 	device_resume(PMSG_RESUME);
@@ -1228,6 +1230,7 @@ static void standby(void)
 
 	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+	sysdev_suspend(PMSG_SUSPEND);
 	local_irq_enable();
 
 	err = set_system_power_state(APM_STATE_STANDBY);

arch/x86/kernel/apm_32.c static void standby(void) · line 1244

Quoted 8 lines
arch/x86/kernel/apm_32.c +1 -0
@@ -1235,6 +1238,7 @@ static void standby(void)
 		apm_error("standby", err);
 
 	local_irq_disable();
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 	local_irq_enable();
 }
...

This causes the following build error with CONFIG_APM=m:

<-- snip -->

...
MODPOST 2586 modules
ERROR: "sysdev_resume" [arch/x86/kernel/apm.ko] undefined!
ERROR: "sysdev_suspend" [arch/x86/kernel/apm.ko] undefined! make[2]: *** [__modpost] Error 1

<-- snip -->

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#9 lore raw

Re: [RFC][PATCH 1/2] PM: Split up sysdev_[suspend|resume] from device_power_[down|up]

In reply to Rafael J. Wysocki

Hi!

Quoted 11 lines

Move the sysdev_suspend/resume from the callee to the callers, with no real change in semantics, so that we can rework the disabling of interrupts during suspend/hibernation.

This is based on an earlier patch from Linus.

Index: linux-2.6/drivers/base/power/main.c
===================================================================

drivers/base/power/main.c static void dpm_power_up(pm_message_t st · line 337

Quoted 6 lines
drivers/base/power/main.c +0 -1
@@ -333,7 +333,6 @@ static void dpm_power_up(pm_message_t st
  */
 void device_power_up(pm_message_t state)
 {
-	sysdev_resume();
 	dpm_power_up(state);
 }

And at this point we can rename dpm_power_up -> device_power_up. Good.

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#10 lore raw

Re: [RFC][PATCH 1/2] PM: Split up sysdev_[suspend|resume] from device_power_[down|up]

In reply to Rafael J. Wysocki

Hi!

Quoted 11 lines

Move the sysdev_suspend/resume from the callee to the callers, with no real change in semantics, so that we can rework the disabling of interrupts during suspend/hibernation.

This is based on an earlier patch from Linus.

Index: linux-2.6/drivers/base/power/main.c
===================================================================

drivers/base/power/main.c static void dpm_power_up(pm_message_t st · line 337

Quoted 6 lines
drivers/base/power/main.c +0 -1
@@ -333,7 +333,6 @@ static void dpm_power_up(pm_message_t st
  */
 void device_power_up(pm_message_t state)
 {
-	sysdev_resume();
 	dpm_power_up(state);
 }

And at this point we can rename dpm_power_up -> device_power_up. Good.

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#11 lore raw

RFC PATCH 2/2 PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

---
 arch/x86/kernel/apm_32.c  |   20 ++++++++--
 drivers/xen/manage.c      |   37 ++++++++++++--------
 include/linux/interrupt.h |    3 +
 kernel/irq/manage.c       |   85 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec.c            |   11 ++++-
 kernel/power/disk.c       |   56 +++++++++++++++++++++++++++---
 kernel/power/main.c       |   27 +++++++++++---
 7 files changed, 208 insertions(+), 31 deletions(-)
kernel/irq/manage.c +85 -0
Index: linux-2.6/kernel/irq/manage.c
===================================================================
@@ -746,3 +746,88 @@ int request_irq(unsigned int irq, irq_ha
 	return retval;
 }
 EXPORT_SYMBOL(request_irq);
+
+#ifdef CONFIG_PM_SLEEP
+struct disabled_irq {
+	struct list_head list;
+	int irq;
+};
+
+static LIST_HEAD(resume_irqs_list);
+
+/**
+ *	enable_device_irqs - enable interrupts disabled by disable_device_irqs()
+ *
+ *	Enable all interrupt lines previously disabled by disable_device_irqs()
+ *	that are on resume_irqs_list.
+ */
+void enable_device_irqs(void)
+{
+	struct disabled_irq *resume_irq, *tmp;
+
+	list_for_each_entry_safe(resume_irq, tmp, &resume_irqs_list, list) {
+		enable_irq(resume_irq->irq);
+		list_del(&resume_irq->list);
+		kfree(resume_irq);
+	}
+}
+
+/**
+ *	disable_device_irqs - disable all enabled interrupt lines
+ *
+ *	During system-wide suspend or hibernation device interrupts need to be
+ *	disabled at the chip level and this function is provided for this
+ *	purpose.  It disables all interrupt lines that are enabled at the
+ *	moment and saves their numbers for enable_device_irqs().
+ */
+int disable_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		unsigned long flags;
+		struct disabled_irq *resume_irq;
+		struct irqaction *action;
+		bool is_timer_irq;
+
+		resume_irq = kzalloc(sizeof(*resume_irq), GFP_NOIO);
+		if (!resume_irq) {
+			enable_device_irqs();
+			return -ENOMEM;
+		}
+
+		spin_lock_irqsave(&desc->lock, flags);
+
+		is_timer_irq = false;
+		action = desc->action;
+		while (action) {
+			if (action->flags | IRQF_TIMER) {
+				is_timer_irq = true;
+				break;
+			}
+			action = action->next;
+		}
+
+		if (!is_timer_irq && !desc->depth) {
+			desc->depth++;
+			desc->status |= IRQ_DISABLED;
+			desc->chip->disable(irq);
+		} else {
+			spin_unlock_irqrestore(&desc->lock, flags);
+			kfree(resume_irq);
+			continue;
+		}
+
+		spin_unlock_irqrestore(&desc->lock, flags);
+
+		if (desc->action)
+			synchronize_irq(irq);
+
+		resume_irq->irq = irq;
+		list_add(&resume_irq->list, &resume_irqs_list);
+	}
+
+	return 0;
+}
+#endif /* CONFIG_PM_SLEEP */
include/linux/interrupt.h +3 -0
Index: linux-2.6/include/linux/interrupt.h
===================================================================
@@ -470,4 +470,7 @@ extern int early_irq_init(void);
 extern int arch_early_irq_init(void);
 extern int arch_init_chip_data(struct irq_desc *desc, int cpu);
 
+extern int disable_device_irqs(void);
+extern void enable_device_irqs(void);
+
 #endif
kernel/power/main.c +21 -6
Index: linux-2.6/kernel/power/main.c
===================================================================
@@ -22,6 +22,7 @@
 #include <linux/freezer.h>
 #include <linux/vmstat.h>
 #include <linux/syscalls.h>
+#include <linux/interrupt.h>
 
 #include "power.h"
 
@@ -287,17 +288,25 @@ void __attribute__ ((weak)) arch_suspend
  */
 static int suspend_enter(suspend_state_t state)
 {
-	int error = 0;
+	int error;
 
 	device_pm_lock();
-	arch_suspend_disable_irqs();
-	BUG_ON(!irqs_disabled());
 
-	if ((error = device_power_down(PMSG_SUSPEND))) {
+	error = disable_device_irqs();
+	if (error) {
+		printk(KERN_ERR "PM: Failed to disable device interrupts\n");
+		goto Unlock;
+	}
+
+	error = device_power_down(PMSG_SUSPEND);
+	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down\n");
 		goto Done;
 	}
 
+	arch_suspend_disable_irqs();
+	BUG_ON(!irqs_disabled());
+
 	error = sysdev_suspend(PMSG_SUSPEND);
 	if (!error) {
 		if (!suspend_test(TEST_CORE))
@@ -305,11 +314,17 @@ static int suspend_enter(suspend_state_t
 		sysdev_resume();
 	}
 
-	device_power_up(PMSG_RESUME);
- Done:
 	arch_suspend_enable_irqs();
 	BUG_ON(irqs_disabled());
+
+	device_power_up(PMSG_RESUME);
+
+ Done:
+	enable_device_irqs();
+
+ Unlock:
 	device_pm_unlock();
+
 	return error;
 }
 
kernel/power/disk.c +50 -6

131 lines not on this page · raw message

arch/x86/kernel/apm_32.c +16 -4
Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================
@@ -228,6 +228,7 @@
 #include <linux/suspend.h>
 #include <linux/kthread.h>
 #include <linux/jiffies.h>
+#include <linux/interrupt.h>
 
 #include <asm/system.h>
 #include <asm/uaccess.h>
@@ -1190,8 +1191,11 @@ static int suspend(int vetoable)
 	struct apm_user	*as;
 
 	device_suspend(PMSG_SUSPEND);
-	local_irq_disable();
+
+	disable_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 
 	local_irq_enable();
@@ -1209,9 +1213,13 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
+	enable_device_irqs();
+
 	device_resume(PMSG_RESUME);
 	queue_event(APM_NORMAL_RESUME, NULL);
 	spin_lock(&user_list_lock);
@@ -1228,8 +1236,10 @@ static void standby(void)
 {
 	int err;
 
-	local_irq_disable();
+	disable_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 	local_irq_enable();
 
@@ -1239,8 +1249,10 @@ static void standby(void)
 
 	local_irq_disable();
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
+	enable_device_irqs();
 }
 
 static apm_event_t get_event(void)
drivers/xen/manage.c +24 -13
Index: linux-2.6/drivers/xen/manage.c
===================================================================
@@ -39,12 +39,6 @@ static int xen_suspend(void *data)
 
 	BUG_ON(!irqs_disabled());
 
-	err = device_power_down(PMSG_SUSPEND);
-	if (err) {
-		printk(KERN_ERR "xen_suspend: device_power_down failed: %d\n",
-		       err);
-		return err;
-	}
 	err = sysdev_suspend(PMSG_SUSPEND);
 	if (err) {
 		printk(KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n",
@@ -69,13 +63,6 @@ static int xen_suspend(void *data)
 	xen_mm_unpin_all();
 
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
-
-	if (!*cancelled) {
-		xen_irq_resume();
-		xen_console_resume();
-		xen_timer_resume();
-	}
 
 	return 0;
 }
@@ -108,6 +95,18 @@ static void do_suspend(void)
 	/* XXX use normal device tree? */
 	xenbus_suspend();
 
+	err = disable_device_irqs();
+	if (err) {
+		printk(KERN_ERR "disable_device_irqs failed: %d\n", err);
+		goto resume_devices;
+	}
+
+	err = device_power_down(PMSG_SUSPEND);
+	if (err) {
+		printk(KERN_ERR "device_power_down failed: %d\n", err);
+		goto enable_irqs;
+	}
+
 	err = stop_machine(xen_suspend, &cancelled, &cpumask_of_cpu(0));
 	if (err) {
 		printk(KERN_ERR "failed to start xen_suspend: %d\n", err);
@@ -120,6 +119,18 @@ static void do_suspend(void)
 	} else
 		xenbus_suspend_cancel();
 
+	device_power_up(PMSG_RESUME);
+
+	if (!cancelled) {
+		xen_irq_resume();
+		xen_console_resume();
+		xen_timer_resume();
+	}
+
+enable_irqs:
+	enable_device_irqs();
+
+resume_devices:
 	device_resume(PMSG_RESUME);
 
 	/* Make sure timer events get retriggered on all CPUs */
kernel/kexec.c +9 -2
Index: linux-2.6/kernel/kexec.c
===================================================================
@@ -1454,7 +1454,11 @@ int kernel_kexec(void)
 		if (error)
 			goto Resume_devices;
 		device_pm_lock();
-		local_irq_disable();
+
+		error = disable_device_irqs();
+		if (error)
+			goto Unlock_pm;
+
 		/* At this point, device_suspend() has been called,
 		 * but *not* device_power_down(). We *must*
 		 * device_power_down() now.  Otherwise, drivers for
@@ -1466,6 +1470,7 @@ int kernel_kexec(void)
 		if (error)
 			goto Enable_irqs;
 
+		local_irq_disable();
 		/* Suspend system devices */
 		error = sysdev_suspend(PMSG_FREEZE);
 		if (error)
@@ -1484,9 +1489,11 @@ int kernel_kexec(void)
 	if (kexec_image->preserve_context) {
 		sysdev_resume();
  Power_up_devices:
+		local_irq_enable();
 		device_power_up(PMSG_RESTORE);
  Enable_irqs:
-		local_irq_enable();
+		enable_device_irqs();
+ Unlock_pm:
 		device_pm_unlock();
 		enable_nonboot_cpus();
  Resume_devices:
#12 lore raw

RFC PATCH 2/2 PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

---
 arch/x86/kernel/apm_32.c  |   20 ++++++++--
 drivers/xen/manage.c      |   37 ++++++++++++--------
 include/linux/interrupt.h |    3 +
 kernel/irq/manage.c       |   85 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec.c            |   11 ++++-
 kernel/power/disk.c       |   56 +++++++++++++++++++++++++++---
 kernel/power/main.c       |   27 +++++++++++---
 7 files changed, 208 insertions(+), 31 deletions(-)
kernel/irq/manage.c +85 -0
Index: linux-2.6/kernel/irq/manage.c
===================================================================
@@ -746,3 +746,88 @@ int request_irq(unsigned int irq, irq_ha
 	return retval;
 }
 EXPORT_SYMBOL(request_irq);
+
+#ifdef CONFIG_PM_SLEEP
+struct disabled_irq {
+	struct list_head list;
+	int irq;
+};
+
+static LIST_HEAD(resume_irqs_list);
+
+/**
+ *	enable_device_irqs - enable interrupts disabled by disable_device_irqs()
+ *
+ *	Enable all interrupt lines previously disabled by disable_device_irqs()
+ *	that are on resume_irqs_list.
+ */
+void enable_device_irqs(void)
+{
+	struct disabled_irq *resume_irq, *tmp;
+
+	list_for_each_entry_safe(resume_irq, tmp, &resume_irqs_list, list) {
+		enable_irq(resume_irq->irq);
+		list_del(&resume_irq->list);
+		kfree(resume_irq);
+	}
+}
+
+/**
+ *	disable_device_irqs - disable all enabled interrupt lines
+ *
+ *	During system-wide suspend or hibernation device interrupts need to be
+ *	disabled at the chip level and this function is provided for this
+ *	purpose.  It disables all interrupt lines that are enabled at the
+ *	moment and saves their numbers for enable_device_irqs().
+ */
+int disable_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		unsigned long flags;
+		struct disabled_irq *resume_irq;
+		struct irqaction *action;
+		bool is_timer_irq;
+
+		resume_irq = kzalloc(sizeof(*resume_irq), GFP_NOIO);
+		if (!resume_irq) {
+			enable_device_irqs();
+			return -ENOMEM;
+		}
+
+		spin_lock_irqsave(&desc->lock, flags);
+
+		is_timer_irq = false;
+		action = desc->action;
+		while (action) {
+			if (action->flags | IRQF_TIMER) {
+				is_timer_irq = true;
+				break;
+			}
+			action = action->next;
+		}
+
+		if (!is_timer_irq && !desc->depth) {
+			desc->depth++;
+			desc->status |= IRQ_DISABLED;
+			desc->chip->disable(irq);
+		} else {
+			spin_unlock_irqrestore(&desc->lock, flags);
+			kfree(resume_irq);
+			continue;
+		}
+
+		spin_unlock_irqrestore(&desc->lock, flags);
+
+		if (desc->action)
+			synchronize_irq(irq);
+
+		resume_irq->irq = irq;
+		list_add(&resume_irq->list, &resume_irqs_list);
+	}
+
+	return 0;
+}
+#endif /* CONFIG_PM_SLEEP */
include/linux/interrupt.h +3 -0
Index: linux-2.6/include/linux/interrupt.h
===================================================================
@@ -470,4 +470,7 @@ extern int early_irq_init(void);
 extern int arch_early_irq_init(void);
 extern int arch_init_chip_data(struct irq_desc *desc, int cpu);
 
+extern int disable_device_irqs(void);
+extern void enable_device_irqs(void);
+
 #endif
kernel/power/main.c +21 -6
Index: linux-2.6/kernel/power/main.c
===================================================================
@@ -22,6 +22,7 @@
 #include <linux/freezer.h>
 #include <linux/vmstat.h>
 #include <linux/syscalls.h>
+#include <linux/interrupt.h>
 
 #include "power.h"
 
@@ -287,17 +288,25 @@ void __attribute__ ((weak)) arch_suspend
  */
 static int suspend_enter(suspend_state_t state)
 {
-	int error = 0;
+	int error;
 
 	device_pm_lock();
-	arch_suspend_disable_irqs();
-	BUG_ON(!irqs_disabled());
 
-	if ((error = device_power_down(PMSG_SUSPEND))) {
+	error = disable_device_irqs();
+	if (error) {
+		printk(KERN_ERR "PM: Failed to disable device interrupts\n");
+		goto Unlock;
+	}
+
+	error = device_power_down(PMSG_SUSPEND);
+	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down\n");
 		goto Done;
 	}
 
+	arch_suspend_disable_irqs();
+	BUG_ON(!irqs_disabled());
+
 	error = sysdev_suspend(PMSG_SUSPEND);
 	if (!error) {
 		if (!suspend_test(TEST_CORE))
@@ -305,11 +314,17 @@ static int suspend_enter(suspend_state_t
 		sysdev_resume();
 	}
 
-	device_power_up(PMSG_RESUME);
- Done:
 	arch_suspend_enable_irqs();
 	BUG_ON(irqs_disabled());
+
+	device_power_up(PMSG_RESUME);
+
+ Done:
+	enable_device_irqs();
+
+ Unlock:
 	device_pm_unlock();
+
 	return error;
 }
 
kernel/power/disk.c +50 -6

131 lines not on this page · raw message

arch/x86/kernel/apm_32.c +16 -4
Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================
@@ -228,6 +228,7 @@
 #include <linux/suspend.h>
 #include <linux/kthread.h>
 #include <linux/jiffies.h>
+#include <linux/interrupt.h>
 
 #include <asm/system.h>
 #include <asm/uaccess.h>
@@ -1190,8 +1191,11 @@ static int suspend(int vetoable)
 	struct apm_user	*as;
 
 	device_suspend(PMSG_SUSPEND);
-	local_irq_disable();
+
+	disable_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 
 	local_irq_enable();
@@ -1209,9 +1213,13 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
+	enable_device_irqs();
+
 	device_resume(PMSG_RESUME);
 	queue_event(APM_NORMAL_RESUME, NULL);
 	spin_lock(&user_list_lock);
@@ -1228,8 +1236,10 @@ static void standby(void)
 {
 	int err;
 
-	local_irq_disable();
+	disable_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 	local_irq_enable();
 
@@ -1239,8 +1249,10 @@ static void standby(void)
 
 	local_irq_disable();
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
+	enable_device_irqs();
 }
 
 static apm_event_t get_event(void)
drivers/xen/manage.c +24 -13
Index: linux-2.6/drivers/xen/manage.c
===================================================================
@@ -39,12 +39,6 @@ static int xen_suspend(void *data)
 
 	BUG_ON(!irqs_disabled());
 
-	err = device_power_down(PMSG_SUSPEND);
-	if (err) {
-		printk(KERN_ERR "xen_suspend: device_power_down failed: %d\n",
-		       err);
-		return err;
-	}
 	err = sysdev_suspend(PMSG_SUSPEND);
 	if (err) {
 		printk(KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n",
@@ -69,13 +63,6 @@ static int xen_suspend(void *data)
 	xen_mm_unpin_all();
 
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
-
-	if (!*cancelled) {
-		xen_irq_resume();
-		xen_console_resume();
-		xen_timer_resume();
-	}
 
 	return 0;
 }
@@ -108,6 +95,18 @@ static void do_suspend(void)
 	/* XXX use normal device tree? */
 	xenbus_suspend();
 
+	err = disable_device_irqs();
+	if (err) {
+		printk(KERN_ERR "disable_device_irqs failed: %d\n", err);
+		goto resume_devices;
+	}
+
+	err = device_power_down(PMSG_SUSPEND);
+	if (err) {
+		printk(KERN_ERR "device_power_down failed: %d\n", err);
+		goto enable_irqs;
+	}
+
 	err = stop_machine(xen_suspend, &cancelled, &cpumask_of_cpu(0));
 	if (err) {
 		printk(KERN_ERR "failed to start xen_suspend: %d\n", err);
@@ -120,6 +119,18 @@ static void do_suspend(void)
 	} else
 		xenbus_suspend_cancel();
 
+	device_power_up(PMSG_RESUME);
+
+	if (!cancelled) {
+		xen_irq_resume();
+		xen_console_resume();
+		xen_timer_resume();
+	}
+
+enable_irqs:
+	enable_device_irqs();
+
+resume_devices:
 	device_resume(PMSG_RESUME);
 
 	/* Make sure timer events get retriggered on all CPUs */
kernel/kexec.c +9 -2
Index: linux-2.6/kernel/kexec.c
===================================================================
@@ -1454,7 +1454,11 @@ int kernel_kexec(void)
 		if (error)
 			goto Resume_devices;
 		device_pm_lock();
-		local_irq_disable();
+
+		error = disable_device_irqs();
+		if (error)
+			goto Unlock_pm;
+
 		/* At this point, device_suspend() has been called,
 		 * but *not* device_power_down(). We *must*
 		 * device_power_down() now.  Otherwise, drivers for
@@ -1466,6 +1470,7 @@ int kernel_kexec(void)
 		if (error)
 			goto Enable_irqs;
 
+		local_irq_disable();
 		/* Suspend system devices */
 		error = sysdev_suspend(PMSG_FREEZE);
 		if (error)
@@ -1484,9 +1489,11 @@ int kernel_kexec(void)
 	if (kexec_image->preserve_context) {
 		sysdev_resume();
  Power_up_devices:
+		local_irq_enable();
 		device_power_up(PMSG_RESTORE);
  Enable_irqs:
-		local_irq_enable();
+		enable_device_irqs();
+ Unlock_pm:
 		device_pm_unlock();
 		enable_nonboot_cpus();
  Resume_devices:
#13 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 7 lines · Rafael J. Wysocki

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

I think this patch is actually a bit too complicated.

Quoted 13 lines
kernel/irq/manage.c +13 -0
+struct disabled_irq {
+	struct list_head list;
+	int irq;
+};
+
+static LIST_HEAD(resume_irqs_list);
+
+/**
+ *	enable_device_irqs - enable interrupts disabled by disable_device_irqs()
+ *
+ *	Enable all interrupt lines previously disabled by disable_device_irqs()
+ *	that are on resume_irqs_list.
+ */

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 773

Quoted 10 lines
kernel/irq/manage.c +10 -0
+void enable_device_irqs(void)
+{
+	struct disabled_irq *resume_irq, *tmp;
+
+	list_for_each_entry_safe(resume_irq, tmp, &resume_irqs_list, list) {
+		enable_irq(resume_irq->irq);
+		list_del(&resume_irq->list);
+		kfree(resume_irq);
+	}
+}

Don't do this whole separate list. Instead, just add a per-irq-descriptor flag to the desc->status field that says "suspended". IOW, just do something like

	diff --git a/include/linux/irq.h b/include/linux/irq.h
	index f899b50..7bc2a31 100644
	--- a/include/linux/irq.h
	+++ b/include/linux/irq.h
	@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsigned int irq,
	 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
	 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
	 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
	+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
	 
	 #ifdef CONFIG_IRQ_PER_CPU
	 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)

and then just make the suspend sequence do

	for_each_irq_desc(irq, desc) {
		.. check desc if we should disable it ..
		disable_irq(irq);
		desc->status |= IRQ_SUSPENDED;
	}

and the resume sequence do

	for_each_irq_desc(irq, desc) {
		if (!(desc->status & IRQ_SUSPENDED))
			continue;
		desc->status &= ~IRQ_SUSPENDED;
		enabled_irq(irq);
	}

And that simplifcation then gets rid of

Quoted 14 lines
kernel/irq/manage.c +14 -0
+/**
+ *	disable_device_irqs - disable all enabled interrupt lines
+ *
+ *	During system-wide suspend or hibernation device interrupts need to be
+ *	disabled at the chip level and this function is provided for this
+ *	purpose.  It disables all interrupt lines that are enabled at the
+ *	moment and saves their numbers for enable_device_irqs().
+ */
+int disable_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 798

Quoted 10 lines
kernel/irq/manage.c +10 -0
+		unsigned long flags;
+		struct disabled_irq *resume_irq;
+		struct irqaction *action;
+		bool is_timer_irq;
+
+		resume_irq = kzalloc(sizeof(*resume_irq), GFP_NOIO);
+		if (!resume_irq) {
+			enable_device_irqs();
+			return -ENOMEM;
+		}

this just goes away.

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 810

Quoted 9 lines
kernel/irq/manage.c +9 -0
+		is_timer_irq = false;
+		action = desc->action;
+		while (action) {
+			if (action->flags | IRQF_TIMER) {
+				is_timer_irq = true;
+				break;
+			}
+			action = action->next;
+		}

This is also pointless and wrong (and buggy). You should use '&' to test that flag, not '|', but more importantly, if you share interrupts with a timer irq, there's nothing sane the irq layer can do ANYWAY, so just ignore the whole problem. Just look at the first one, don't try to be clever, because your clever code doesn't buy anything at all.

So get rid of the loop, and just do

	if (desc->action && !(desc->action->flags & IRQF_TIMER)) {
		desc->depth++;
		desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
		desc->chip->disable(irq);
	}
	spin_unlock_irqrestore(&desc->lock, flags);

and you're done.

Also, I'd actually suggest that the whole "synchronize_irq()" be handled in a separate loop after the main one, so make that one just be

	for_each_irq_desc(irq, desc) {
		if (desc->status & IRQ_SUSPENDED)
			serialize_irq(irq);
	}

at the end. No need for desc->lock, since the IRQ_SUSPENDED bit is stable.

Finally:

include/linux/interrupt.h extern int early_irq_init(void); · line 474

Quoted 2 lines
include/linux/interrupt.h +2 -0
+extern int disable_device_irqs(void);
+extern void enable_device_irqs(void);

I think the naming is not great. It's not about disable/enable, it's very much about suspend/resume. In your version, it had that global "disabled_irq" list, and in mine it has that IRQ_SUSPENDED bit - and in both cases you can't nest things, and you can't consider them in any way "generic" enable/disable things, they are very specialized "shut up everything but the timer irq".

I also don't think there is any reasonable error case, so just make the "suspend" thing return 'void', and don't complicate the caller. We don't error out on the simple "disable_irq()" either. It's a imperative statement, not a "please can you try to do that" thing.

Linus

#14 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 7 lines · Rafael J. Wysocki

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

I think this patch is actually a bit too complicated.

Quoted 13 lines
kernel/irq/manage.c +13 -0
+struct disabled_irq {
+	struct list_head list;
+	int irq;
+};
+
+static LIST_HEAD(resume_irqs_list);
+
+/**
+ *	enable_device_irqs - enable interrupts disabled by disable_device_irqs()
+ *
+ *	Enable all interrupt lines previously disabled by disable_device_irqs()
+ *	that are on resume_irqs_list.
+ */

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 773

Quoted 10 lines
kernel/irq/manage.c +10 -0
+void enable_device_irqs(void)
+{
+	struct disabled_irq *resume_irq, *tmp;
+
+	list_for_each_entry_safe(resume_irq, tmp, &resume_irqs_list, list) {
+		enable_irq(resume_irq->irq);
+		list_del(&resume_irq->list);
+		kfree(resume_irq);
+	}
+}

Don't do this whole separate list. Instead, just add a per-irq-descriptor flag to the desc->status field that says "suspended". IOW, just do something like

	diff --git a/include/linux/irq.h b/include/linux/irq.h
	index f899b50..7bc2a31 100644
	--- a/include/linux/irq.h
	+++ b/include/linux/irq.h
	@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsigned int irq,
	 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
	 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
	 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
	+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
	 
	 #ifdef CONFIG_IRQ_PER_CPU
	 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)

and then just make the suspend sequence do

	for_each_irq_desc(irq, desc) {
		.. check desc if we should disable it ..
		disable_irq(irq);
		desc->status |= IRQ_SUSPENDED;
	}

and the resume sequence do

	for_each_irq_desc(irq, desc) {
		if (!(desc->status & IRQ_SUSPENDED))
			continue;
		desc->status &= ~IRQ_SUSPENDED;
		enabled_irq(irq);
	}

And that simplifcation then gets rid of

Quoted 14 lines
kernel/irq/manage.c +14 -0
+/**
+ *	disable_device_irqs - disable all enabled interrupt lines
+ *
+ *	During system-wide suspend or hibernation device interrupts need to be
+ *	disabled at the chip level and this function is provided for this
+ *	purpose.  It disables all interrupt lines that are enabled at the
+ *	moment and saves their numbers for enable_device_irqs().
+ */
+int disable_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 798

Quoted 10 lines
kernel/irq/manage.c +10 -0
+		unsigned long flags;
+		struct disabled_irq *resume_irq;
+		struct irqaction *action;
+		bool is_timer_irq;
+
+		resume_irq = kzalloc(sizeof(*resume_irq), GFP_NOIO);
+		if (!resume_irq) {
+			enable_device_irqs();
+			return -ENOMEM;
+		}

this just goes away.

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 810

Quoted 9 lines
kernel/irq/manage.c +9 -0
+		is_timer_irq = false;
+		action = desc->action;
+		while (action) {
+			if (action->flags | IRQF_TIMER) {
+				is_timer_irq = true;
+				break;
+			}
+			action = action->next;
+		}

This is also pointless and wrong (and buggy). You should use '&' to test that flag, not '|', but more importantly, if you share interrupts with a timer irq, there's nothing sane the irq layer can do ANYWAY, so just ignore the whole problem. Just look at the first one, don't try to be clever, because your clever code doesn't buy anything at all.

So get rid of the loop, and just do

	if (desc->action && !(desc->action->flags & IRQF_TIMER)) {
		desc->depth++;
		desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
		desc->chip->disable(irq);
	}
	spin_unlock_irqrestore(&desc->lock, flags);

and you're done.

Also, I'd actually suggest that the whole "synchronize_irq()" be handled in a separate loop after the main one, so make that one just be

	for_each_irq_desc(irq, desc) {
		if (desc->status & IRQ_SUSPENDED)
			serialize_irq(irq);
	}

at the end. No need for desc->lock, since the IRQ_SUSPENDED bit is stable.

Finally:

include/linux/interrupt.h extern int early_irq_init(void); · line 474

Quoted 2 lines
include/linux/interrupt.h +2 -0
+extern int disable_device_irqs(void);
+extern void enable_device_irqs(void);

I think the naming is not great. It's not about disable/enable, it's very much about suspend/resume. In your version, it had that global "disabled_irq" list, and in mine it has that IRQ_SUSPENDED bit - and in both cases you can't nest things, and you can't consider them in any way "generic" enable/disable things, they are very specialized "shut up everything but the timer irq".

I also don't think there is any reasonable error case, so just make the "suspend" thing return 'void', and don't complicate the caller. We don't error out on the simple "disable_irq()" either. It's a imperative statement, not a "please can you try to do that" thing.

Linus

#15 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 31 lines · Linus Torvalds
Quoted 7 lines · Rafael J. Wysocki

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

I think this patch is actually a bit too complicated.

Quoted 19 lines
kernel/irq/manage.c +19 -0
+struct disabled_irq {
+	struct list_head list;
+	int irq;
+};
+
+static LIST_HEAD(resume_irqs_list);
+
+/**
+ *	enable_device_irqs - enable interrupts disabled by disable_device_irqs()
+ *
+ *	Enable all interrupt lines previously disabled by disable_device_irqs()
+ *	that are on resume_irqs_list.
+ */
+void enable_device_irqs(void)
+{
+	struct disabled_irq *resume_irq, *tmp;
+
+	list_for_each_entry_safe(resume_irq, tmp, &resume_irqs_list, list) {
+		enable_irq(resume_irq->irq);
Quoted 8 lines

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 773

Quoted 4 lines
kernel/irq/manage.c +4 -0
+		list_del(&resume_irq->list);
+		kfree(resume_irq);
+	}
+}

Don't do this whole separate list. Instead, just add a per-irq-descriptor flag to the desc->status field that says "suspended". IOW, just do something like

OK

Quoted 59 lines
	diff --git a/include/linux/irq.h b/include/linux/irq.h
	index f899b50..7bc2a31 100644
	--- a/include/linux/irq.h
	+++ b/include/linux/irq.h
	@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsigned int irq,
	 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
	 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
	 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
	+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
	 
	 #ifdef CONFIG_IRQ_PER_CPU
	 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)

and then just make the suspend sequence do

	for_each_irq_desc(irq, desc) {
		.. check desc if we should disable it ..
		disable_irq(irq);
		desc->status |= IRQ_SUSPENDED;
	}

and the resume sequence do

	for_each_irq_desc(irq, desc) {
		if (!(desc->status & IRQ_SUSPENDED))
			continue;
		desc->status &= ~IRQ_SUSPENDED;
		enabled_irq(irq);
	}

And that simplifcation then gets rid of

Quoted 14 lines
kernel/irq/manage.c +14 -0
+/**
+ *	disable_device_irqs - disable all enabled interrupt lines
+ *
+ *	During system-wide suspend or hibernation device interrupts need to be
+ *	disabled at the chip level and this function is provided for this
+ *	purpose.  It disables all interrupt lines that are enabled at the
+ *	moment and saves their numbers for enable_device_irqs().
+ */
+int disable_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 798

Quoted 10 lines
kernel/irq/manage.c +10 -0
+		unsigned long flags;
+		struct disabled_irq *resume_irq;
+		struct irqaction *action;
+		bool is_timer_irq;
+
+		resume_irq = kzalloc(sizeof(*resume_irq), GFP_NOIO);
+		if (!resume_irq) {
+			enable_device_irqs();
+			return -ENOMEM;
+		}

this just goes away.

Quoted 12 lines

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 810

Quoted 9 lines
kernel/irq/manage.c +9 -0
+		is_timer_irq = false;
+		action = desc->action;
+		while (action) {
+			if (action->flags | IRQF_TIMER) {
+				is_timer_irq = true;
+				break;
+			}
+			action = action->next;
+		}

This is also pointless and wrong (and buggy). You should use '&' to test that flag, not '|',

Ouch, sorry.

Quoted 7 lines

but more importantly, if you share interrupts with a timer irq, there's nothing sane the irq layer can do ANYWAY, so just ignore the whole problem. Just look at the first one, don't try to be clever, because your clever code doesn't buy anything at all.

So get rid of the loop, and just do

Quoted 8 lines
	if (desc->action && !(desc->action->flags & IRQF_TIMER)) {
		desc->depth++;
		desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
		desc->chip->disable(irq);
	}
	spin_unlock_irqrestore(&desc->lock, flags);

and you're done.

OK

Quoted 9 lines

Also, I'd actually suggest that the whole "synchronize_irq()" be handled in a separate loop after the main one, so make that one just be

	for_each_irq_desc(irq, desc) {
		if (desc->status & IRQ_SUSPENDED)
			serialize_irq(irq);
	}

at the end. No need for desc->lock, since the IRQ_SUSPENDED bit is stable.

OK

Quoted 11 lines

Finally:

include/linux/interrupt.h extern int early_irq_init(void); · line 474

Quoted 2 lines
include/linux/interrupt.h +2 -0
+extern int disable_device_irqs(void);
+extern void enable_device_irqs(void);

I think the naming is not great. It's not about disable/enable, it's very much about suspend/resume. In your version, it had that global "disabled_irq" list, and in mine it has that IRQ_SUSPENDED bit - and in both cases you can't nest things, and you can't consider them in any way "generic" enable/disable things, they are very specialized "shut up everything but the timer irq".

OK, would

extern void suspend_device_irqs(void);
extern void resume_device_irqs(void);

be better?

Quoted 4 lines

I also don't think there is any reasonable error case, so just make the "suspend" thing return 'void', and don't complicate the caller. We don't error out on the simple "disable_irq()" either. It's a imperative statement, not a "please can you try to do that" thing.

The error is there just because the memory allocation can fail. With the IRQ_SUSPENDED flag as per your suggestion it won't be necessary any more.

Thanks a lot for your comments, I'll send an updated patch shortly.

Rafael

#16 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 31 lines · Linus Torvalds
Quoted 7 lines · Rafael J. Wysocki

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

I think this patch is actually a bit too complicated.

Quoted 19 lines
kernel/irq/manage.c +19 -0
+struct disabled_irq {
+	struct list_head list;
+	int irq;
+};
+
+static LIST_HEAD(resume_irqs_list);
+
+/**
+ *	enable_device_irqs - enable interrupts disabled by disable_device_irqs()
+ *
+ *	Enable all interrupt lines previously disabled by disable_device_irqs()
+ *	that are on resume_irqs_list.
+ */
+void enable_device_irqs(void)
+{
+	struct disabled_irq *resume_irq, *tmp;
+
+	list_for_each_entry_safe(resume_irq, tmp, &resume_irqs_list, list) {
+		enable_irq(resume_irq->irq);
Quoted 8 lines

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 773

Quoted 4 lines
kernel/irq/manage.c +4 -0
+		list_del(&resume_irq->list);
+		kfree(resume_irq);
+	}
+}

Don't do this whole separate list. Instead, just add a per-irq-descriptor flag to the desc->status field that says "suspended". IOW, just do something like

OK

Quoted 59 lines
	diff --git a/include/linux/irq.h b/include/linux/irq.h
	index f899b50..7bc2a31 100644
	--- a/include/linux/irq.h
	+++ b/include/linux/irq.h
	@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsigned int irq,
	 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
	 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
	 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
	+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
	 
	 #ifdef CONFIG_IRQ_PER_CPU
	 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)

and then just make the suspend sequence do

	for_each_irq_desc(irq, desc) {
		.. check desc if we should disable it ..
		disable_irq(irq);
		desc->status |= IRQ_SUSPENDED;
	}

and the resume sequence do

	for_each_irq_desc(irq, desc) {
		if (!(desc->status & IRQ_SUSPENDED))
			continue;
		desc->status &= ~IRQ_SUSPENDED;
		enabled_irq(irq);
	}

And that simplifcation then gets rid of

Quoted 14 lines
kernel/irq/manage.c +14 -0
+/**
+ *	disable_device_irqs - disable all enabled interrupt lines
+ *
+ *	During system-wide suspend or hibernation device interrupts need to be
+ *	disabled at the chip level and this function is provided for this
+ *	purpose.  It disables all interrupt lines that are enabled at the
+ *	moment and saves their numbers for enable_device_irqs().
+ */
+int disable_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 798

Quoted 10 lines
kernel/irq/manage.c +10 -0
+		unsigned long flags;
+		struct disabled_irq *resume_irq;
+		struct irqaction *action;
+		bool is_timer_irq;
+
+		resume_irq = kzalloc(sizeof(*resume_irq), GFP_NOIO);
+		if (!resume_irq) {
+			enable_device_irqs();
+			return -ENOMEM;
+		}

this just goes away.

Quoted 12 lines

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 810

Quoted 9 lines
kernel/irq/manage.c +9 -0
+		is_timer_irq = false;
+		action = desc->action;
+		while (action) {
+			if (action->flags | IRQF_TIMER) {
+				is_timer_irq = true;
+				break;
+			}
+			action = action->next;
+		}

This is also pointless and wrong (and buggy). You should use '&' to test that flag, not '|',

Ouch, sorry.

Quoted 7 lines

but more importantly, if you share interrupts with a timer irq, there's nothing sane the irq layer can do ANYWAY, so just ignore the whole problem. Just look at the first one, don't try to be clever, because your clever code doesn't buy anything at all.

So get rid of the loop, and just do

Quoted 8 lines
	if (desc->action && !(desc->action->flags & IRQF_TIMER)) {
		desc->depth++;
		desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
		desc->chip->disable(irq);
	}
	spin_unlock_irqrestore(&desc->lock, flags);

and you're done.

OK

Quoted 9 lines

Also, I'd actually suggest that the whole "synchronize_irq()" be handled in a separate loop after the main one, so make that one just be

	for_each_irq_desc(irq, desc) {
		if (desc->status & IRQ_SUSPENDED)
			serialize_irq(irq);
	}

at the end. No need for desc->lock, since the IRQ_SUSPENDED bit is stable.

OK

Quoted 11 lines

Finally:

include/linux/interrupt.h extern int early_irq_init(void); · line 474

Quoted 2 lines
include/linux/interrupt.h +2 -0
+extern int disable_device_irqs(void);
+extern void enable_device_irqs(void);

I think the naming is not great. It's not about disable/enable, it's very much about suspend/resume. In your version, it had that global "disabled_irq" list, and in mine it has that IRQ_SUSPENDED bit - and in both cases you can't nest things, and you can't consider them in any way "generic" enable/disable things, they are very specialized "shut up everything but the timer irq".

OK, would

extern void suspend_device_irqs(void);
extern void resume_device_irqs(void);

be better?

Quoted 4 lines

I also don't think there is any reasonable error case, so just make the "suspend" thing return 'void', and don't complicate the caller. We don't error out on the simple "disable_irq()" either. It's a imperative statement, not a "please can you try to do that" thing.

The error is there just because the memory allocation can fail. With the IRQ_SUSPENDED flag as per your suggestion it won't be necessary any more.

Thanks a lot for your comments, I'll send an updated patch shortly.

Rafael

#17 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Linus Torvalds

On Sun, 22 Feb 2009, Rafael J. Wysocki wrote:

[--snip--]

Quoted 2 lines

Thanks a lot for your comments, I'll send an updated patch shortly.

The updated patch is appended.

It has been initially tested, but requires more testing, especially with APM, XEN, kexec jump etc.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM: Rework handling of interrupts during suspend-resume (rev. 2)

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

---
 arch/x86/kernel/apm_32.c  |   20 ++++++++++++----
 drivers/xen/manage.c      |   32 +++++++++++++++----------
 include/linux/interrupt.h |    3 ++
 include/linux/irq.h       |    1 
 kernel/irq/manage.c       |   57 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec.c            |   10 ++++----
 kernel/power/disk.c       |   46 +++++++++++++++++++++++++++++--------
 kernel/power/main.c       |   20 +++++++++++-----
 8 files changed, 152 insertions(+), 37 deletions(-)
kernel/irq/manage.c +57 -0
Index: linux-2.6/kernel/irq/manage.c
===================================================================
@@ -746,3 +746,60 @@ int request_irq(unsigned int irq, irq_ha
 	return retval;
 }
 EXPORT_SYMBOL(request_irq);
+
+#ifdef CONFIG_PM_SLEEP
+/**
+ *	suspend_device_irqs - disable all currently enabled interrupt lines
+ *
+ *	During system-wide suspend or hibernation device interrupts need to be
+ *	disabled at the chip level and this function is provided for this
+ *	purpose.  It disables all interrupt lines that are enabled at the
+ *	moment and sets the IRQ_SUSPENDED flag for them.
+ */
+void suspend_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		unsigned long flags;
+
+		spin_lock_irqsave(&desc->lock, flags);
+
+		if (!desc->depth && desc->action
+		    && !(desc->action->flags & IRQF_TIMER)) {
+			desc->depth++;
+			desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
+			desc->chip->disable(irq);
+		}
+
+		spin_unlock_irqrestore(&desc->lock, flags);
+	}
+
+	for_each_irq_desc(irq, desc) {
+		if (desc->status & IRQ_SUSPENDED)
+			synchronize_irq(irq);
+	}
+}
+EXPORT_SYMBOL_GPL(suspend_device_irqs);
+
+/**
+ *	resume_device_irqs - enable interrupts disabled by suspend_device_irqs()
+ *
+ *	Enable all interrupt lines previously disabled by suspend_device_irqs()
+ *	that have the IRQ_SUSPENDED flag set.
+ */
+void resume_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		if (!(desc->status & IRQ_SUSPENDED))
+			continue;
+		desc->status &= ~IRQ_SUSPENDED;
+		enable_irq(irq);
+	}
+}
+EXPORT_SYMBOL_GPL(resume_device_irqs);
+#endif /* CONFIG_PM_SLEEP */
include/linux/interrupt.h +3 -0
Index: linux-2.6/include/linux/interrupt.h
===================================================================
@@ -470,4 +470,7 @@ extern int early_irq_init(void);
 extern int arch_early_irq_init(void);
 extern int arch_init_chip_data(struct irq_desc *desc, int cpu);
 
+extern void suspend_device_irqs(void);
+extern void resume_device_irqs(void);
+
 #endif
kernel/power/main.c +14 -6
Index: linux-2.6/kernel/power/main.c
===================================================================
@@ -22,6 +22,7 @@
 #include <linux/freezer.h>
 #include <linux/vmstat.h>
 #include <linux/syscalls.h>
+#include <linux/interrupt.h>
 
 #include "power.h"
 
@@ -287,17 +288,20 @@ void __attribute__ ((weak)) arch_suspend
  */
 static int suspend_enter(suspend_state_t state)
 {
-	int error = 0;
+	int error;
 
 	device_pm_lock();
-	arch_suspend_disable_irqs();
-	BUG_ON(!irqs_disabled());
+	suspend_device_irqs();
 
-	if ((error = device_power_down(PMSG_SUSPEND))) {
+	error = device_power_down(PMSG_SUSPEND);
+	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down\n");
 		goto Done;
 	}
 
+	arch_suspend_disable_irqs();
+	BUG_ON(!irqs_disabled());
+
 	error = sysdev_suspend(PMSG_SUSPEND);
 	if (!error) {
 		if (!suspend_test(TEST_CORE))
@@ -305,11 +309,15 @@ static int suspend_enter(suspend_state_t
 		sysdev_resume();
 	}
 
-	device_power_up(PMSG_RESUME);
- Done:
 	arch_suspend_enable_irqs();
 	BUG_ON(irqs_disabled());
+
+	device_power_up(PMSG_RESUME);
+
+ Done:
+	resume_device_irqs();
 	device_pm_unlock();
+
 	return error;
 }
 
kernel/power/disk.c +36 -10

120 lines not on this page · raw message

arch/x86/kernel/apm_32.c +16 -4
Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================
@@ -228,6 +228,7 @@
 #include <linux/suspend.h>
 #include <linux/kthread.h>
 #include <linux/jiffies.h>
+#include <linux/interrupt.h>
 
 #include <asm/system.h>
 #include <asm/uaccess.h>
@@ -1190,8 +1191,11 @@ static int suspend(int vetoable)
 	struct apm_user	*as;
 
 	device_suspend(PMSG_SUSPEND);
-	local_irq_disable();
+
+	suspend_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 
 	local_irq_enable();
@@ -1209,9 +1213,13 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
+	resume_device_irqs();
+
 	device_resume(PMSG_RESUME);
 	queue_event(APM_NORMAL_RESUME, NULL);
 	spin_lock(&user_list_lock);
@@ -1228,8 +1236,10 @@ static void standby(void)
 {
 	int err;
 
-	local_irq_disable();
+	suspend_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 	local_irq_enable();
 
@@ -1239,8 +1249,10 @@ static void standby(void)
 
 	local_irq_disable();
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
+	resume_device_irqs();
 }
 
 static apm_event_t get_event(void)
drivers/xen/manage.c +19 -13
Index: linux-2.6/drivers/xen/manage.c
===================================================================
@@ -39,12 +39,6 @@ static int xen_suspend(void *data)
 
 	BUG_ON(!irqs_disabled());
 
-	err = device_power_down(PMSG_SUSPEND);
-	if (err) {
-		printk(KERN_ERR "xen_suspend: device_power_down failed: %d\n",
-		       err);
-		return err;
-	}
 	err = sysdev_suspend(PMSG_SUSPEND);
 	if (err) {
 		printk(KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n",
@@ -69,13 +63,6 @@ static int xen_suspend(void *data)
 	xen_mm_unpin_all();
 
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
-
-	if (!*cancelled) {
-		xen_irq_resume();
-		xen_console_resume();
-		xen_timer_resume();
-	}
 
 	return 0;
 }
@@ -108,6 +95,14 @@ static void do_suspend(void)
 	/* XXX use normal device tree? */
 	xenbus_suspend();
 
+	suspend_device_irqs();
+
+	err = device_power_down(PMSG_SUSPEND);
+	if (err) {
+		printk(KERN_ERR "device_power_down failed: %d\n", err);
+		goto resume_devices;
+	}
+
 	err = stop_machine(xen_suspend, &cancelled, &cpumask_of_cpu(0));
 	if (err) {
 		printk(KERN_ERR "failed to start xen_suspend: %d\n", err);
@@ -120,6 +115,17 @@ static void do_suspend(void)
 	} else
 		xenbus_suspend_cancel();
 
+	device_power_up(PMSG_RESUME);
+
+	if (!cancelled) {
+		xen_irq_resume();
+		xen_console_resume();
+		xen_timer_resume();
+	}
+
+resume_devices:
+	resume_device_irqs();
+
 	device_resume(PMSG_RESUME);
 
 	/* Make sure timer events get retriggered on all CPUs */
kernel/kexec.c +6 -4
Index: linux-2.6/kernel/kexec.c
===================================================================
@@ -1454,7 +1454,7 @@ int kernel_kexec(void)
 		if (error)
 			goto Resume_devices;
 		device_pm_lock();
-		local_irq_disable();
+		suspend_device_irqs();
 		/* At this point, device_suspend() has been called,
 		 * but *not* device_power_down(). We *must*
 		 * device_power_down() now.  Otherwise, drivers for
@@ -1464,8 +1464,9 @@ int kernel_kexec(void)
 		 */
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
-			goto Enable_irqs;
+			goto Resume_irqs;
 
+		local_irq_disable();
 		/* Suspend system devices */
 		error = sysdev_suspend(PMSG_FREEZE);
 		if (error)
@@ -1484,9 +1485,10 @@ int kernel_kexec(void)
 	if (kexec_image->preserve_context) {
 		sysdev_resume();
  Power_up_devices:
-		device_power_up(PMSG_RESTORE);
- Enable_irqs:
 		local_irq_enable();
+		device_power_up(PMSG_RESTORE);
+ Resume_irqs:
+		resume_device_irqs();
 		device_pm_unlock();
 		enable_nonboot_cpus();
  Resume_devices:
include/linux/irq.h +1 -0
Index: linux-2.6/include/linux/irq.h
===================================================================
@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsig
 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
 
 #ifdef CONFIG_IRQ_PER_CPU
 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)
#18 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Linus Torvalds

On Sun, 22 Feb 2009, Rafael J. Wysocki wrote:

[--snip--]

Quoted 2 lines

Thanks a lot for your comments, I'll send an updated patch shortly.

The updated patch is appended.

It has been initially tested, but requires more testing, especially with APM, XEN, kexec jump etc.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM: Rework handling of interrupts during suspend-resume (rev. 2)

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

---
 arch/x86/kernel/apm_32.c  |   20 ++++++++++++----
 drivers/xen/manage.c      |   32 +++++++++++++++----------
 include/linux/interrupt.h |    3 ++
 include/linux/irq.h       |    1 
 kernel/irq/manage.c       |   57 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec.c            |   10 ++++----
 kernel/power/disk.c       |   46 +++++++++++++++++++++++++++++--------
 kernel/power/main.c       |   20 +++++++++++-----
 8 files changed, 152 insertions(+), 37 deletions(-)
kernel/irq/manage.c +57 -0
Index: linux-2.6/kernel/irq/manage.c
===================================================================
@@ -746,3 +746,60 @@ int request_irq(unsigned int irq, irq_ha
 	return retval;
 }
 EXPORT_SYMBOL(request_irq);
+
+#ifdef CONFIG_PM_SLEEP
+/**
+ *	suspend_device_irqs - disable all currently enabled interrupt lines
+ *
+ *	During system-wide suspend or hibernation device interrupts need to be
+ *	disabled at the chip level and this function is provided for this
+ *	purpose.  It disables all interrupt lines that are enabled at the
+ *	moment and sets the IRQ_SUSPENDED flag for them.
+ */
+void suspend_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		unsigned long flags;
+
+		spin_lock_irqsave(&desc->lock, flags);
+
+		if (!desc->depth && desc->action
+		    && !(desc->action->flags & IRQF_TIMER)) {
+			desc->depth++;
+			desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
+			desc->chip->disable(irq);
+		}
+
+		spin_unlock_irqrestore(&desc->lock, flags);
+	}
+
+	for_each_irq_desc(irq, desc) {
+		if (desc->status & IRQ_SUSPENDED)
+			synchronize_irq(irq);
+	}
+}
+EXPORT_SYMBOL_GPL(suspend_device_irqs);
+
+/**
+ *	resume_device_irqs - enable interrupts disabled by suspend_device_irqs()
+ *
+ *	Enable all interrupt lines previously disabled by suspend_device_irqs()
+ *	that have the IRQ_SUSPENDED flag set.
+ */
+void resume_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		if (!(desc->status & IRQ_SUSPENDED))
+			continue;
+		desc->status &= ~IRQ_SUSPENDED;
+		enable_irq(irq);
+	}
+}
+EXPORT_SYMBOL_GPL(resume_device_irqs);
+#endif /* CONFIG_PM_SLEEP */
include/linux/interrupt.h +3 -0
Index: linux-2.6/include/linux/interrupt.h
===================================================================
@@ -470,4 +470,7 @@ extern int early_irq_init(void);
 extern int arch_early_irq_init(void);
 extern int arch_init_chip_data(struct irq_desc *desc, int cpu);
 
+extern void suspend_device_irqs(void);
+extern void resume_device_irqs(void);
+
 #endif
kernel/power/main.c +14 -6
Index: linux-2.6/kernel/power/main.c
===================================================================
@@ -22,6 +22,7 @@
 #include <linux/freezer.h>
 #include <linux/vmstat.h>
 #include <linux/syscalls.h>
+#include <linux/interrupt.h>
 
 #include "power.h"
 
@@ -287,17 +288,20 @@ void __attribute__ ((weak)) arch_suspend
  */
 static int suspend_enter(suspend_state_t state)
 {
-	int error = 0;
+	int error;
 
 	device_pm_lock();
-	arch_suspend_disable_irqs();
-	BUG_ON(!irqs_disabled());
+	suspend_device_irqs();
 
-	if ((error = device_power_down(PMSG_SUSPEND))) {
+	error = device_power_down(PMSG_SUSPEND);
+	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down\n");
 		goto Done;
 	}
 
+	arch_suspend_disable_irqs();
+	BUG_ON(!irqs_disabled());
+
 	error = sysdev_suspend(PMSG_SUSPEND);
 	if (!error) {
 		if (!suspend_test(TEST_CORE))
@@ -305,11 +309,15 @@ static int suspend_enter(suspend_state_t
 		sysdev_resume();
 	}
 
-	device_power_up(PMSG_RESUME);
- Done:
 	arch_suspend_enable_irqs();
 	BUG_ON(irqs_disabled());
+
+	device_power_up(PMSG_RESUME);
+
+ Done:
+	resume_device_irqs();
 	device_pm_unlock();
+
 	return error;
 }
 
kernel/power/disk.c +36 -10

120 lines not on this page · raw message

arch/x86/kernel/apm_32.c +16 -4
Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================
@@ -228,6 +228,7 @@
 #include <linux/suspend.h>
 #include <linux/kthread.h>
 #include <linux/jiffies.h>
+#include <linux/interrupt.h>
 
 #include <asm/system.h>
 #include <asm/uaccess.h>
@@ -1190,8 +1191,11 @@ static int suspend(int vetoable)
 	struct apm_user	*as;
 
 	device_suspend(PMSG_SUSPEND);
-	local_irq_disable();
+
+	suspend_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 
 	local_irq_enable();
@@ -1209,9 +1213,13 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
+	resume_device_irqs();
+
 	device_resume(PMSG_RESUME);
 	queue_event(APM_NORMAL_RESUME, NULL);
 	spin_lock(&user_list_lock);
@@ -1228,8 +1236,10 @@ static void standby(void)
 {
 	int err;
 
-	local_irq_disable();
+	suspend_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 	local_irq_enable();
 
@@ -1239,8 +1249,10 @@ static void standby(void)
 
 	local_irq_disable();
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
+	resume_device_irqs();
 }
 
 static apm_event_t get_event(void)
drivers/xen/manage.c +19 -13
Index: linux-2.6/drivers/xen/manage.c
===================================================================
@@ -39,12 +39,6 @@ static int xen_suspend(void *data)
 
 	BUG_ON(!irqs_disabled());
 
-	err = device_power_down(PMSG_SUSPEND);
-	if (err) {
-		printk(KERN_ERR "xen_suspend: device_power_down failed: %d\n",
-		       err);
-		return err;
-	}
 	err = sysdev_suspend(PMSG_SUSPEND);
 	if (err) {
 		printk(KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n",
@@ -69,13 +63,6 @@ static int xen_suspend(void *data)
 	xen_mm_unpin_all();
 
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
-
-	if (!*cancelled) {
-		xen_irq_resume();
-		xen_console_resume();
-		xen_timer_resume();
-	}
 
 	return 0;
 }
@@ -108,6 +95,14 @@ static void do_suspend(void)
 	/* XXX use normal device tree? */
 	xenbus_suspend();
 
+	suspend_device_irqs();
+
+	err = device_power_down(PMSG_SUSPEND);
+	if (err) {
+		printk(KERN_ERR "device_power_down failed: %d\n", err);
+		goto resume_devices;
+	}
+
 	err = stop_machine(xen_suspend, &cancelled, &cpumask_of_cpu(0));
 	if (err) {
 		printk(KERN_ERR "failed to start xen_suspend: %d\n", err);
@@ -120,6 +115,17 @@ static void do_suspend(void)
 	} else
 		xenbus_suspend_cancel();
 
+	device_power_up(PMSG_RESUME);
+
+	if (!cancelled) {
+		xen_irq_resume();
+		xen_console_resume();
+		xen_timer_resume();
+	}
+
+resume_devices:
+	resume_device_irqs();
+
 	device_resume(PMSG_RESUME);
 
 	/* Make sure timer events get retriggered on all CPUs */
kernel/kexec.c +6 -4
Index: linux-2.6/kernel/kexec.c
===================================================================
@@ -1454,7 +1454,7 @@ int kernel_kexec(void)
 		if (error)
 			goto Resume_devices;
 		device_pm_lock();
-		local_irq_disable();
+		suspend_device_irqs();
 		/* At this point, device_suspend() has been called,
 		 * but *not* device_power_down(). We *must*
 		 * device_power_down() now.  Otherwise, drivers for
@@ -1464,8 +1464,9 @@ int kernel_kexec(void)
 		 */
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
-			goto Enable_irqs;
+			goto Resume_irqs;
 
+		local_irq_disable();
 		/* Suspend system devices */
 		error = sysdev_suspend(PMSG_FREEZE);
 		if (error)
@@ -1484,9 +1485,10 @@ int kernel_kexec(void)
 	if (kexec_image->preserve_context) {
 		sysdev_resume();
  Power_up_devices:
-		device_power_up(PMSG_RESTORE);
- Enable_irqs:
 		local_irq_enable();
+		device_power_up(PMSG_RESTORE);
+ Resume_irqs:
+		resume_device_irqs();
 		device_pm_unlock();
 		enable_nonboot_cpus();
  Resume_devices:
include/linux/irq.h +1 -0
Index: linux-2.6/include/linux/irq.h
===================================================================
@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsig
 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
 
 #ifdef CONFIG_IRQ_PER_CPU
 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)
#19 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 2 lines · Rafael J. Wysocki

The updated patch is appended.

Ok, looks sane to me. I'll try it on my poor eeepc, although right now Fedora-11 rawhide (that poor laptop gets _all_ the crazy stuff thrown at it, and runs btrfs to boot) has broken something in X by enabling DRI2, so I may not get around to it today.

Linus

#20 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 2 lines · Rafael J. Wysocki

The updated patch is appended.

Ok, looks sane to me. I'll try it on my poor eeepc, although right now Fedora-11 rawhide (that poor laptop gets _all_ the crazy stuff thrown at it, and runs btrfs to boot) has broken something in X by enabling DRI2, so I may not get around to it today.

Linus

#21 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 5 lines · Linus Torvalds

Ok, looks sane to me. I'll try it on my poor eeepc, although right now Fedora-11 rawhide (that poor laptop gets _all_ the crazy stuff thrown at it, and runs btrfs to boot) has broken something in X by enabling DRI2, so I may not get around to it today.

Well, the suspend/resume part seems to work for me. My X issues keep my from testing it with compiz, but here's an ack for v2 of the 2/2 patch at least on my EeePC.

Linus

#22 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Linus Torvalds

Ok, looks sane to me. I'll try it on my poor eeepc, although right now Fedora-11 rawhide (that poor laptop gets _all_ the crazy stuff thrown at it, and runs btrfs to boot) has broken something in X by enabling DRI2, so I may not get around to it today.

Well, the suspend/resume part seems to work for me.

Great, thanks for testing.

Rafael

#23 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 27 lines · Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Linus Torvalds

On Sun, 22 Feb 2009, Rafael J. Wysocki wrote:

[--snip--]

Quoted 2 lines

Thanks a lot for your comments, I'll send an updated patch shortly.

The updated patch is appended.

It has been initially tested, but requires more testing, especially with APM, XEN, kexec jump etc.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM: Rework handling of interrupts during suspend-resume (rev. 2)

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Quoted 6 lines

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

I don't have an issue with the code, but I do have an issue with this description of it.

Calling disable especially for ioapics does nothing directly. It simply arranges for the irq to be marked pending and for the irq to be masked if the irq happens.

So what you are doing is arranging so that no interrupts will be delivered to drivers. Not really disabling interrupts at the IO-APIC level.

In addition not all interrupts (even on x86) go through an IO-APIC anymore so describing the patch in terms of an IO-APIC makes it a bit hard to understand what your intent actually is.

Eric

#24 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 44 lines · Eric W. Biederman
Quoted 27 lines · Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Linus Torvalds

On Sun, 22 Feb 2009, Rafael J. Wysocki wrote:

[--snip--]

Quoted 2 lines

Thanks a lot for your comments, I'll send an updated patch shortly.

The updated patch is appended.

It has been initially tested, but requires more testing, especially with APM, XEN, kexec jump etc.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM: Rework handling of interrupts during suspend-resume (rev. 2)

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Quoted 7 lines

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

I don't have an issue with the code, but I do have an issue with this description of it.

Calling disable especially for ioapics does nothing directly. It simply arranges for the irq to be marked pending and for the irq to be masked if the irq happens.

Quoted 8 lines

So what you are doing is arranging so that no interrupts will be delivered to drivers. Not really disabling interrupts at the IO-APIC level.

In addition not all interrupts (even on x86) go through an IO-APIC anymore so describing the patch in terms of an IO-APIC makes it a bit hard to understand what your intent actually is.

I think this aspect has been well-understood during the discussion of this topic and it's just a slightly misleading changelog.

The new suspend code does not rely on truly disabling IRQs on the low level. The purpose is to not get IRQs to drivers - which might crash/hang/race/misbehave.

Still, it might make sense to not just use the ->disable sequence but primarily the ->shutdown irqchip method (when it's available in the irqchip).

While we obviously cannot turn off the PIC that delivers timer IRQs at this stage - there's no theoretical reason why the suspend sequence couldnt power down some secondary PICs as well
- in some arch code, or maybe even in the generic driver suspend sequence if the device tree is structured carefully enough so that the PIC gets turned off last.

So turning off all device IRQs in the most lowlevel way possible would be prudent. I.e. the suspend stage should do:

                if (desc->chip->shutdown)
                        desc->chip->shutdown(irq);
                else
                        desc->chip->disable(irq);

(there's no change needed for the resume stage)

Ingo

#25 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Eric W. Biederman
Quoted 44 lines · Eric W. Biederman
Quoted 27 lines · Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Linus Torvalds

On Sun, 22 Feb 2009, Rafael J. Wysocki wrote:

[--snip--]

Quoted 2 lines

Thanks a lot for your comments, I'll send an updated patch shortly.

The updated patch is appended.

It has been initially tested, but requires more testing, especially with APM, XEN, kexec jump etc.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM: Rework handling of interrupts during suspend-resume (rev. 2)

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Quoted 7 lines

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

I don't have an issue with the code, but I do have an issue with this description of it.

Calling disable especially for ioapics does nothing directly. It simply arranges for the irq to be marked pending and for the irq to be masked if the irq happens.

Quoted 8 lines

So what you are doing is arranging so that no interrupts will be delivered to drivers. Not really disabling interrupts at the IO-APIC level.

In addition not all interrupts (even on x86) go through an IO-APIC anymore so describing the patch in terms of an IO-APIC makes it a bit hard to understand what your intent actually is.

I think this aspect has been well-understood during the discussion of this topic and it's just a slightly misleading changelog.

The new suspend code does not rely on truly disabling IRQs on the low level. The purpose is to not get IRQs to drivers - which might crash/hang/race/misbehave.

Still, it might make sense to not just use the ->disable sequence but primarily the ->shutdown irqchip method (when it's available in the irqchip).

While we obviously cannot turn off the PIC that delivers timer IRQs at this stage - there's no theoretical reason why the suspend sequence couldnt power down some secondary PICs as well
- in some arch code, or maybe even in the generic driver suspend sequence if the device tree is structured carefully enough so that the PIC gets turned off last.

So turning off all device IRQs in the most lowlevel way possible would be prudent. I.e. the suspend stage should do:

                if (desc->chip->shutdown)
                        desc->chip->shutdown(irq);
                else
                        desc->chip->disable(irq);

(there's no change needed for the resume stage)

Ingo

#26 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 3 lines · Ingo Molnar

I think this aspect has been well-understood during the discussion of this topic and it's just a slightly misleading changelog.

As I was a member of that discussion I did not see that.

It took me several passes through the patches to realize the goal is to allow drivers to be able to sleep while they are in their late pm shutdown routines.

Why we want this I don't know. But it seems simple enough to implement, and it makes it harder to get the late pm suspend routines wrong, which is always good.

Quoted 3 lines

The new suspend code does not rely on truly disabling IRQs on the low level. The purpose is to not get IRQs to drivers - which might crash/hang/race/misbehave.

Reasonable. I expect one of the problems with drivers getting it wrong is that the interface is too complex for mortal humans to understand.

Quoted 3 lines

Still, it might make sense to not just use the ->disable sequence but primarily the ->shutdown irqchip method (when it's available in the irqchip).

Disable seems fine to me. This is interesting in the context of all of the irqs that will when masked show up somewhere else (think boot interrupts).

Quoted 6 lines

While we obviously cannot turn off the PIC that delivers timer IRQs at this stage - there's no theoretical reason why the suspend sequence couldnt power down some secondary PICs as well
- in some arch code, or maybe even in the generic driver suspend sequence if the device tree is structured carefully enough so that the PIC gets turned off last.

If the point is simply to prevent deliver of irqs to the drivers I don't see the point of anything more than what the patch does now.

Eric

#27 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 8 lines · Eric W. Biederman
Quoted 3 lines · Ingo Molnar

I think this aspect has been well-understood during the discussion of this topic and it's just a slightly misleading changelog.

As I was a member of that discussion I did not see that.

Quoted 7 lines

It took me several passes through the patches to realize the goal is to allow drivers to be able to sleep while they are in their late pm shutdown routines.

Why we want this I don't know. But it seems simple enough to implement, and it makes it harder to get the late pm suspend routines wrong, which is always good.

That's not the only goal. The other goal is to further shrink a particular window of suspend fragility: the irqs-disabled stage of the suspend/resume sequence.

Since suspend/resume is a mini-reboot sequence, there's a large amount of code executed - and the variety of code is large as well. We had repeat cases of random drivers re-enabling interrupts and thus breaking other drivers - and these are nasty to debug.

So this patchset disables device IRQs centrally and serializes with pending work - so there's no races with pending IRQs anymore.

The fact that we keep the timer irq running is two-fold: firstly the timer code is special and not really part of the regular suspend/resume sequence.

Drivers want to take timestamps, sometimes they even want to do a small usleep(), etc. Ideally the suspend/resume code is pretty much _the same_ as a regular bootup (and shutdown) code - so we want to provide a similar environment to how drivers initialize and deinitialize, and we want to enable them to share code between bootup/shutdown and suspend/resume agressively.

So the more generic kernel environment we give these fragile handlers, the better we are off in the end. Since we already had IRQS_TIMER, that was just the natural thing to do.

Quoted 7 lines
Quoted 3 lines

The new suspend code does not rely on truly disabling IRQs on the low level. The purpose is to not get IRQs to drivers
- which might crash/hang/race/misbehave.

Reasonable. I expect one of the problems with drivers getting it wrong is that the interface is too complex for mortal humans to understand.

The suspend/resume state machine certainly used to be a piece of code that makes a seasoned kernel developer weep in fear.

That has changed drastically in the past few months. The suspend+hibernation logic got unified (at least as far as driver methods go), and all the flow and ordering has been cleaned up and has been made more robust.

What makes s2ram fragile is not human failure but the combination of a handful of physical property:

1) Psychology: shutting the lid or pushing the suspend button is a deceivingly 'simple' action to the user. But under the hood, a ton of stuff happens: we deinitialize a lot of things, we go through _all hardware state_, and we do so in a serial fashion. If just one piece fails to do the right thing, the box might not resume. Still, the user expects this 'simple' thing to just work, all the time. No excuses accepted.

2) Length of code: To get a successful s2ram sequence the kernel runs through tens of thousands of lines of code. Code which never gets executed on a normal box - only if we s2ram. If just one step fails, we get a hung box.

3) Debuggability: a lot of s2ram code runs with the console off, making any bugs hard to debug. Furthermore we have no meaningful persistent storage either for kernel bug messages. The RTC trick of PM_DEBUG works but is a very narrow channel of information and it takes a lot of time to debug a bug via that method.

The combination of these factors really makes up for a perfect storm in terms of kernel technology: we have this very-deceivingly-simple-looking but complex-and-rarely-executed piece of code, which is very hard to debug.

Even just one of these factors would be enough to make an otherwise healthy subsystem fragile - no wonder s2ram has been a problem ever since it existed in the upstream kernel.

So now we need just one thing: patience and more of the same good stuff that happened lately.

Quoted 15 lines
Quoted 3 lines

Still, it might make sense to not just use the ->disable sequence but primarily the ->shutdown irqchip method (when it's available in the irqchip).

Disable seems fine to me. This is interesting in the context of all of the irqs that will when masked show up somewhere else (think boot interrupts).

Quoted 6 lines

While we obviously cannot turn off the PIC that delivers timer IRQs at this stage - there's no theoretical reason why the suspend sequence couldnt power down some secondary PICs as well - in some arch code, or maybe even in the generic driver suspend sequence if the device tree is structured carefully enough so that the PIC gets turned off last.

Quoted 3 lines

If the point is simply to prevent deliver of irqs to the drivers I don't see the point of anything more than what the patch does now.

... except for the usecase i described above. Say some PIC sits on a piece of silicon which gets turned off. I'm not talking about x86 but some custom device. We really dont want that IRQ line to send half of an IRQ message (un-ACK-ed) when it gets turned off. So physically 'suspending' all IRQ lines does make a certain level of long-term sense.

Especially if it's just 3 extra lines of code to the existing patch.

There _might_ be one downside: overhead of ->shutdown() methods. With a typical IRQ count on the typical netbook i doubt it's more than ~50 usecs combined.

Ingo

#28 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Eric W. Biederman
Quoted 8 lines · Eric W. Biederman
Quoted 3 lines · Ingo Molnar

I think this aspect has been well-understood during the discussion of this topic and it's just a slightly misleading changelog.

As I was a member of that discussion I did not see that.

Quoted 7 lines

It took me several passes through the patches to realize the goal is to allow drivers to be able to sleep while they are in their late pm shutdown routines.

Why we want this I don't know. But it seems simple enough to implement, and it makes it harder to get the late pm suspend routines wrong, which is always good.

That's not the only goal. The other goal is to further shrink a particular window of suspend fragility: the irqs-disabled stage of the suspend/resume sequence.

Since suspend/resume is a mini-reboot sequence, there's a large amount of code executed - and the variety of code is large as well. We had repeat cases of random drivers re-enabling interrupts and thus breaking other drivers - and these are nasty to debug.

So this patchset disables device IRQs centrally and serializes with pending work - so there's no races with pending IRQs anymore.

The fact that we keep the timer irq running is two-fold: firstly the timer code is special and not really part of the regular suspend/resume sequence.

Drivers want to take timestamps, sometimes they even want to do a small usleep(), etc. Ideally the suspend/resume code is pretty much _the same_ as a regular bootup (and shutdown) code - so we want to provide a similar environment to how drivers initialize and deinitialize, and we want to enable them to share code between bootup/shutdown and suspend/resume agressively.

So the more generic kernel environment we give these fragile handlers, the better we are off in the end. Since we already had IRQS_TIMER, that was just the natural thing to do.

Quoted 7 lines
Quoted 3 lines

The new suspend code does not rely on truly disabling IRQs on the low level. The purpose is to not get IRQs to drivers
- which might crash/hang/race/misbehave.

Reasonable. I expect one of the problems with drivers getting it wrong is that the interface is too complex for mortal humans to understand.

The suspend/resume state machine certainly used to be a piece of code that makes a seasoned kernel developer weep in fear.

That has changed drastically in the past few months. The suspend+hibernation logic got unified (at least as far as driver methods go), and all the flow and ordering has been cleaned up and has been made more robust.

What makes s2ram fragile is not human failure but the combination of a handful of physical property:

1) Psychology: shutting the lid or pushing the suspend button is a deceivingly 'simple' action to the user. But under the hood, a ton of stuff happens: we deinitialize a lot of things, we go through _all hardware state_, and we do so in a serial fashion. If just one piece fails to do the right thing, the box might not resume. Still, the user expects this 'simple' thing to just work, all the time. No excuses accepted.

2) Length of code: To get a successful s2ram sequence the kernel runs through tens of thousands of lines of code. Code which never gets executed on a normal box - only if we s2ram. If just one step fails, we get a hung box.

3) Debuggability: a lot of s2ram code runs with the console off, making any bugs hard to debug. Furthermore we have no meaningful persistent storage either for kernel bug messages. The RTC trick of PM_DEBUG works but is a very narrow channel of information and it takes a lot of time to debug a bug via that method.

The combination of these factors really makes up for a perfect storm in terms of kernel technology: we have this very-deceivingly-simple-looking but complex-and-rarely-executed piece of code, which is very hard to debug.

Even just one of these factors would be enough to make an otherwise healthy subsystem fragile - no wonder s2ram has been a problem ever since it existed in the upstream kernel.

So now we need just one thing: patience and more of the same good stuff that happened lately.

Quoted 15 lines
Quoted 3 lines

Still, it might make sense to not just use the ->disable sequence but primarily the ->shutdown irqchip method (when it's available in the irqchip).

Disable seems fine to me. This is interesting in the context of all of the irqs that will when masked show up somewhere else (think boot interrupts).

Quoted 6 lines

While we obviously cannot turn off the PIC that delivers timer IRQs at this stage - there's no theoretical reason why the suspend sequence couldnt power down some secondary PICs as well - in some arch code, or maybe even in the generic driver suspend sequence if the device tree is structured carefully enough so that the PIC gets turned off last.

Quoted 3 lines

If the point is simply to prevent deliver of irqs to the drivers I don't see the point of anything more than what the patch does now.

... except for the usecase i described above. Say some PIC sits on a piece of silicon which gets turned off. I'm not talking about x86 but some custom device. We really dont want that IRQ line to send half of an IRQ message (un-ACK-ed) when it gets turned off. So physically 'suspending' all IRQ lines does make a certain level of long-term sense.

Especially if it's just 3 extra lines of code to the existing patch.

There _might_ be one downside: overhead of ->shutdown() methods. With a typical IRQ count on the typical netbook i doubt it's more than ~50 usecs combined.

Ingo

#29 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 43 lines · Ingo Molnar
Quoted 8 lines · Eric W. Biederman
Quoted 3 lines · Ingo Molnar

I think this aspect has been well-understood during the discussion of this topic and it's just a slightly misleading changelog.

As I was a member of that discussion I did not see that.

Quoted 7 lines

It took me several passes through the patches to realize the goal is to allow drivers to be able to sleep while they are in their late pm shutdown routines.

Why we want this I don't know. But it seems simple enough to implement, and it makes it harder to get the late pm suspend routines wrong, which is always good.

That's not the only goal. The other goal is to further shrink a particular window of suspend fragility: the irqs-disabled stage of the suspend/resume sequence.

Since suspend/resume is a mini-reboot sequence, there's a large amount of code executed - and the variety of code is large as well. We had repeat cases of random drivers re-enabling interrupts and thus breaking other drivers - and these are nasty to debug.

So this patchset disables device IRQs centrally and serializes with pending work - so there's no races with pending IRQs anymore.

The fact that we keep the timer irq running is two-fold: firstly the timer code is special and not really part of the regular suspend/resume sequence.

Drivers want to take timestamps, sometimes they even want to do a small usleep(), etc. Ideally the suspend/resume code is pretty much _the same_ as a regular bootup (and shutdown) code - so we want to provide a similar environment to how drivers initialize and deinitialize, and we want to enable them to share code between bootup/shutdown and suspend/resume agressively.

Quoted 3 lines

So the more generic kernel environment we give these fragile handlers, the better we are off in the end. Since we already had IRQS_TIMER, that was just the natural thing to do.

I am all for sharing code, especially if we can factor if we can find common factors that do the same thing.

I don't know how many times I have found drivers doing something weird in their shutdown routines that they don't know how to get the device out of. The e1000 driver has shown up several times because it likes to suspend the device on shutdown.

The fact that the methods exposed to drivers were only defined to be usable on the s2ram/hibernate path is something I have brought up on more than one occasion as a bad choice.

I'm really not convinced that the rational for separating out the shutdown methods from the remove methods has been very good. That of we don't need to clean up the in-kernel data structures on reboot so why do something extra that can introduce instability.

So having been watching a smaller form of this drama on the reboot path for several years. Having had a device method with fixed semantics, and not the dwm sematics of the historical suspend routing. I expect there is still a ways to go before it is simple and easy for drivers to figure out what they need to implement out of the confusing variety of possible device methods.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

The new suspend code does not rely on truly disabling IRQs on the low level. The purpose is to not get IRQs to drivers
- which might crash/hang/race/misbehave.

Reasonable. I expect one of the problems with drivers getting it wrong is that the interface is too complex for mortal humans to understand.

Quoted 7 lines

The suspend/resume state machine certainly used to be a piece of code that makes a seasoned kernel developer weep in fear.

That has changed drastically in the past few months. The suspend+hibernation logic got unified (at least as far as driver methods go), and all the flow and ordering has been cleaned up and has been made more robust.

I will have to look again. My impression is that overloading a single method is part of what got us into this mess in the first place.

No that I don't see things getting better.

Quoted 17 lines

What makes s2ram fragile is not human failure but the combination of a handful of physical property:

1) Psychology: shutting the lid or pushing the suspend button is a deceivingly 'simple' action to the user. But under the hood, a ton of stuff happens: we deinitialize a lot of things, we go through _all hardware state_, and we do so in a serial fashion. If just one piece fails to do the right thing, the box might not resume. Still, the user expects this 'simple' thing to just work, all the time. No excuses accepted.

2) Length of code: To get a successful s2ram sequence the kernel runs through tens of thousands of lines of code. Code which never gets executed on a normal box - only if we s2ram. If just one step fails, we get a hung box.

Quoted 6 lines

3) Debuggability: a lot of s2ram code runs with the console off, making any bugs hard to debug. Furthermore we have no meaningful persistent storage either for kernel bug messages. The RTC trick of PM_DEBUG works but is a very narrow channel of information and it takes a lot of time to debug a bug via that method.

Yep that is an issue.

Quoted 4 lines

The combination of these factors really makes up for a perfect storm in terms of kernel technology: we have this very-deceivingly-simple-looking but complex-and-rarely-executed piece of code, which is very hard to debug.

And much of this as you are finding with this piece of code is how the software was designed rather then how the software needed to be.

Quoted 6 lines

Even just one of these factors would be enough to make an otherwise healthy subsystem fragile - no wonder s2ram has been a problem ever since it existed in the upstream kernel.

So now we need just one thing: patience and more of the same good stuff that happened lately.

I think there has been some good progress, and so I am happy to be patient. I will still mention on occasion what it seems we are doing wrong. Unfortunately I don't have time to do a lot more than that.

Quoted 19 lines
Quoted 15 lines
Quoted 3 lines

Still, it might make sense to not just use the ->disable sequence but primarily the ->shutdown irqchip method (when it's available in the irqchip).

Disable seems fine to me. This is interesting in the context of all of the irqs that will when masked show up somewhere else (think boot interrupts).

Quoted 6 lines

While we obviously cannot turn off the PIC that delivers timer IRQs at this stage - there's no theoretical reason why the suspend sequence couldnt power down some secondary PICs as well - in some arch code, or maybe even in the generic driver suspend sequence if the device tree is structured carefully enough so that the PIC gets turned off last.

Quoted 3 lines

If the point is simply to prevent deliver of irqs to the drivers I don't see the point of anything more than what the patch does now.

Quoted 6 lines

... except for the usecase i described above. Say some PIC sits on a piece of silicon which gets turned off. I'm not talking about x86 but some custom device. We really dont want that IRQ line to send half of an IRQ message (un-ACK-ed) when it gets turned off. So physically 'suspending' all IRQ lines does make a certain level of long-term sense.

Good point. We will loose both level and edge triggered events that occur between suspending the irqs and restoring them but that is inevitable. So we might as well call shutdown and totally turn off the irqs if we can.

I don't know where in the state machine this is getting called but I would suggest doing this before we shutdown cpus. We are quickly reaching the point where laptops will exceed the 8 core limit, of lowest priority delivery mode. And only in lowest priority delivery mode is it possible to migrate irqs outside of the interrupt handlers.

That plus if we suspend the irqs before shutting down the cpus it means we can safely support more vectors than a single cpu can catch.

I was a little a worried about the shutdown code path because it requires in the worst case acking a level triggered irq when we have it disabled, but looking at ack_apic_level that appears to be a well tested code path. We just can't reprogram the vector.

Quoted 3 lines

There _might_ be one downside: overhead of ->shutdown() methods. With a typical IRQ count on the typical netbook i doubt it's more than ~50 usecs combined.

Eric

#30 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 170 lines · Eric W. Biederman
Quoted 43 lines · Ingo Molnar
Quoted 8 lines · Eric W. Biederman
Quoted 3 lines · Ingo Molnar

I think this aspect has been well-understood during the discussion of this topic and it's just a slightly misleading changelog.

As I was a member of that discussion I did not see that.

Quoted 7 lines

It took me several passes through the patches to realize the goal is to allow drivers to be able to sleep while they are in their late pm shutdown routines.

Why we want this I don't know. But it seems simple enough to implement, and it makes it harder to get the late pm suspend routines wrong, which is always good.

That's not the only goal. The other goal is to further shrink a particular window of suspend fragility: the irqs-disabled stage of the suspend/resume sequence.

Since suspend/resume is a mini-reboot sequence, there's a large amount of code executed - and the variety of code is large as well. We had repeat cases of random drivers re-enabling interrupts and thus breaking other drivers - and these are nasty to debug.

So this patchset disables device IRQs centrally and serializes with pending work - so there's no races with pending IRQs anymore.

The fact that we keep the timer irq running is two-fold: firstly the timer code is special and not really part of the regular suspend/resume sequence.

Drivers want to take timestamps, sometimes they even want to do a small usleep(), etc. Ideally the suspend/resume code is pretty much _the same_ as a regular bootup (and shutdown) code - so we want to provide a similar environment to how drivers initialize and deinitialize, and we want to enable them to share code between bootup/shutdown and suspend/resume agressively.

Quoted 3 lines

So the more generic kernel environment we give these fragile handlers, the better we are off in the end. Since we already had IRQS_TIMER, that was just the natural thing to do.

I am all for sharing code, especially if we can factor if we can find common factors that do the same thing.

I don't know how many times I have found drivers doing something weird in their shutdown routines that they don't know how to get the device out of. The e1000 driver has shown up several times because it likes to suspend the device on shutdown.

The fact that the methods exposed to drivers were only defined to be usable on the s2ram/hibernate path is something I have brought up on more than one occasion as a bad choice.

I'm really not convinced that the rational for separating out the shutdown methods from the remove methods has been very good. That of we don't need to clean up the in-kernel data structures on reboot so why do something extra that can introduce instability.

So having been watching a smaller form of this drama on the reboot path for several years. Having had a device method with fixed semantics, and not the dwm sematics of the historical suspend routing. I expect there is still a ways to go before it is simple and easy for drivers to figure out what they need to implement out of the confusing variety of possible device methods.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

The new suspend code does not rely on truly disabling IRQs on the low level. The purpose is to not get IRQs to drivers
- which might crash/hang/race/misbehave.

Reasonable. I expect one of the problems with drivers getting it wrong is that the interface is too complex for mortal humans to understand.

Quoted 7 lines

The suspend/resume state machine certainly used to be a piece of code that makes a seasoned kernel developer weep in fear.

That has changed drastically in the past few months. The suspend+hibernation logic got unified (at least as far as driver methods go), and all the flow and ordering has been cleaned up and has been made more robust.

I will have to look again. My impression is that overloading a single method is part of what got us into this mess in the first place.

No that I don't see things getting better.

Quoted 17 lines

What makes s2ram fragile is not human failure but the combination of a handful of physical property:

1) Psychology: shutting the lid or pushing the suspend button is a deceivingly 'simple' action to the user. But under the hood, a ton of stuff happens: we deinitialize a lot of things, we go through _all hardware state_, and we do so in a serial fashion. If just one piece fails to do the right thing, the box might not resume. Still, the user expects this 'simple' thing to just work, all the time. No excuses accepted.

2) Length of code: To get a successful s2ram sequence the kernel runs through tens of thousands of lines of code. Code which never gets executed on a normal box - only if we s2ram. If just one step fails, we get a hung box.

Quoted 6 lines

3) Debuggability: a lot of s2ram code runs with the console off, making any bugs hard to debug. Furthermore we have no meaningful persistent storage either for kernel bug messages. The RTC trick of PM_DEBUG works but is a very narrow channel of information and it takes a lot of time to debug a bug via that method.

Yep that is an issue.

Quoted 4 lines

The combination of these factors really makes up for a perfect storm in terms of kernel technology: we have this very-deceivingly-simple-looking but complex-and-rarely-executed piece of code, which is very hard to debug.

And much of this as you are finding with this piece of code is how the software was designed rather then how the software needed to be.

Quoted 6 lines

Even just one of these factors would be enough to make an otherwise healthy subsystem fragile - no wonder s2ram has been a problem ever since it existed in the upstream kernel.

So now we need just one thing: patience and more of the same good stuff that happened lately.

I think there has been some good progress, and so I am happy to be patient. I will still mention on occasion what it seems we are doing wrong. Unfortunately I don't have time to do a lot more than that.

Quoted 19 lines
Quoted 15 lines
Quoted 3 lines

Still, it might make sense to not just use the ->disable sequence but primarily the ->shutdown irqchip method (when it's available in the irqchip).

Disable seems fine to me. This is interesting in the context of all of the irqs that will when masked show up somewhere else (think boot interrupts).

Quoted 6 lines

While we obviously cannot turn off the PIC that delivers timer IRQs at this stage - there's no theoretical reason why the suspend sequence couldnt power down some secondary PICs as well - in some arch code, or maybe even in the generic driver suspend sequence if the device tree is structured carefully enough so that the PIC gets turned off last.

Quoted 3 lines

If the point is simply to prevent deliver of irqs to the drivers I don't see the point of anything more than what the patch does now.

Quoted 6 lines

... except for the usecase i described above. Say some PIC sits on a piece of silicon which gets turned off. I'm not talking about x86 but some custom device. We really dont want that IRQ line to send half of an IRQ message (un-ACK-ed) when it gets turned off. So physically 'suspending' all IRQ lines does make a certain level of long-term sense.

Quoted 7 lines

Good point. We will loose both level and edge triggered events that occur between suspending the irqs and restoring them but that is inevitable. So we might as well call shutdown and totally turn off the irqs if we can.

I don't know where in the state machine this is getting called but I would suggest doing this before we shutdown cpus.

This is the plan. In fact, I'm going to do this in the next patch after the $subject one has been tested and found acceptable.

Thanks,
Rafael

#31 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Eric W. Biederman
Quoted 170 lines · Eric W. Biederman
Quoted 43 lines · Ingo Molnar
Quoted 8 lines · Eric W. Biederman
Quoted 3 lines · Ingo Molnar

I think this aspect has been well-understood during the discussion of this topic and it's just a slightly misleading changelog.

As I was a member of that discussion I did not see that.

Quoted 7 lines

It took me several passes through the patches to realize the goal is to allow drivers to be able to sleep while they are in their late pm shutdown routines.

Why we want this I don't know. But it seems simple enough to implement, and it makes it harder to get the late pm suspend routines wrong, which is always good.

That's not the only goal. The other goal is to further shrink a particular window of suspend fragility: the irqs-disabled stage of the suspend/resume sequence.

Since suspend/resume is a mini-reboot sequence, there's a large amount of code executed - and the variety of code is large as well. We had repeat cases of random drivers re-enabling interrupts and thus breaking other drivers - and these are nasty to debug.

So this patchset disables device IRQs centrally and serializes with pending work - so there's no races with pending IRQs anymore.

The fact that we keep the timer irq running is two-fold: firstly the timer code is special and not really part of the regular suspend/resume sequence.

Drivers want to take timestamps, sometimes they even want to do a small usleep(), etc. Ideally the suspend/resume code is pretty much _the same_ as a regular bootup (and shutdown) code - so we want to provide a similar environment to how drivers initialize and deinitialize, and we want to enable them to share code between bootup/shutdown and suspend/resume agressively.

Quoted 3 lines

So the more generic kernel environment we give these fragile handlers, the better we are off in the end. Since we already had IRQS_TIMER, that was just the natural thing to do.

I am all for sharing code, especially if we can factor if we can find common factors that do the same thing.

I don't know how many times I have found drivers doing something weird in their shutdown routines that they don't know how to get the device out of. The e1000 driver has shown up several times because it likes to suspend the device on shutdown.

The fact that the methods exposed to drivers were only defined to be usable on the s2ram/hibernate path is something I have brought up on more than one occasion as a bad choice.

I'm really not convinced that the rational for separating out the shutdown methods from the remove methods has been very good. That of we don't need to clean up the in-kernel data structures on reboot so why do something extra that can introduce instability.

So having been watching a smaller form of this drama on the reboot path for several years. Having had a device method with fixed semantics, and not the dwm sematics of the historical suspend routing. I expect there is still a ways to go before it is simple and easy for drivers to figure out what they need to implement out of the confusing variety of possible device methods.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

The new suspend code does not rely on truly disabling IRQs on the low level. The purpose is to not get IRQs to drivers
- which might crash/hang/race/misbehave.

Reasonable. I expect one of the problems with drivers getting it wrong is that the interface is too complex for mortal humans to understand.

Quoted 7 lines

The suspend/resume state machine certainly used to be a piece of code that makes a seasoned kernel developer weep in fear.

That has changed drastically in the past few months. The suspend+hibernation logic got unified (at least as far as driver methods go), and all the flow and ordering has been cleaned up and has been made more robust.

I will have to look again. My impression is that overloading a single method is part of what got us into this mess in the first place.

No that I don't see things getting better.

Quoted 17 lines

What makes s2ram fragile is not human failure but the combination of a handful of physical property:

1) Psychology: shutting the lid or pushing the suspend button is a deceivingly 'simple' action to the user. But under the hood, a ton of stuff happens: we deinitialize a lot of things, we go through _all hardware state_, and we do so in a serial fashion. If just one piece fails to do the right thing, the box might not resume. Still, the user expects this 'simple' thing to just work, all the time. No excuses accepted.

2) Length of code: To get a successful s2ram sequence the kernel runs through tens of thousands of lines of code. Code which never gets executed on a normal box - only if we s2ram. If just one step fails, we get a hung box.

Quoted 6 lines

3) Debuggability: a lot of s2ram code runs with the console off, making any bugs hard to debug. Furthermore we have no meaningful persistent storage either for kernel bug messages. The RTC trick of PM_DEBUG works but is a very narrow channel of information and it takes a lot of time to debug a bug via that method.

Yep that is an issue.

Quoted 4 lines

The combination of these factors really makes up for a perfect storm in terms of kernel technology: we have this very-deceivingly-simple-looking but complex-and-rarely-executed piece of code, which is very hard to debug.

And much of this as you are finding with this piece of code is how the software was designed rather then how the software needed to be.

Quoted 6 lines

Even just one of these factors would be enough to make an otherwise healthy subsystem fragile - no wonder s2ram has been a problem ever since it existed in the upstream kernel.

So now we need just one thing: patience and more of the same good stuff that happened lately.

I think there has been some good progress, and so I am happy to be patient. I will still mention on occasion what it seems we are doing wrong. Unfortunately I don't have time to do a lot more than that.

Quoted 19 lines
Quoted 15 lines
Quoted 3 lines

Still, it might make sense to not just use the ->disable sequence but primarily the ->shutdown irqchip method (when it's available in the irqchip).

Disable seems fine to me. This is interesting in the context of all of the irqs that will when masked show up somewhere else (think boot interrupts).

Quoted 6 lines

While we obviously cannot turn off the PIC that delivers timer IRQs at this stage - there's no theoretical reason why the suspend sequence couldnt power down some secondary PICs as well - in some arch code, or maybe even in the generic driver suspend sequence if the device tree is structured carefully enough so that the PIC gets turned off last.

Quoted 3 lines

If the point is simply to prevent deliver of irqs to the drivers I don't see the point of anything more than what the patch does now.

Quoted 6 lines

... except for the usecase i described above. Say some PIC sits on a piece of silicon which gets turned off. I'm not talking about x86 but some custom device. We really dont want that IRQ line to send half of an IRQ message (un-ACK-ed) when it gets turned off. So physically 'suspending' all IRQ lines does make a certain level of long-term sense.

Quoted 7 lines

Good point. We will loose both level and edge triggered events that occur between suspending the irqs and restoring them but that is inevitable. So we might as well call shutdown and totally turn off the irqs if we can.

I don't know where in the state machine this is getting called but I would suggest doing this before we shutdown cpus.

This is the plan. In fact, I'm going to do this in the next patch after the $subject one has been tested and found acceptable.

Thanks,
Rafael

#32 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines

I don't know where in the state machine this is getting called but I would suggest doing this before we shutdown cpus.

This is the plan. In fact, I'm going to do this in the next patch after the $subject one has been tested and found acceptable.

Good to hear. Then let's please get a version of the irq disable that calls shutdown, so we can be certain we don't have hardware irqs in flight.

For the drivers it should not matter for clean cpu shutdown it will.

Eric

#33 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 5 lines · Eric W. Biederman
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines

I don't know where in the state machine this is getting called but I would suggest doing this before we shutdown cpus.

Quoted 7 lines
Quoted 2 lines

This is the plan. In fact, I'm going to do this in the next patch after the $subject one has been tested and found acceptable.

Good to hear. Then let's please get a version of the irq disable that calls shutdown, so we can be certain we don't have hardware irqs in flight.

For the drivers it should not matter for clean cpu shutdown it will.

OK, I will.

Thanks,
Rafael

#34 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Eric W. Biederman
Quoted 5 lines · Eric W. Biederman
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines

I don't know where in the state machine this is getting called but I would suggest doing this before we shutdown cpus.

Quoted 7 lines
Quoted 2 lines

This is the plan. In fact, I'm going to do this in the next patch after the $subject one has been tested and found acceptable.

Good to hear. Then let's please get a version of the irq disable that calls shutdown, so we can be certain we don't have hardware irqs in flight.

For the drivers it should not matter for clean cpu shutdown it will.

OK, I will.

Thanks,
Rafael

#35 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 9 lines · Rafael J. Wysocki
Quoted 8 lines · Eric W. Biederman
Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines

I don't know where in the state machine this is getting called but I would suggest doing this before we shutdown cpus.

This is the plan. In fact, I'm going to do this in the next patch after the $subject one has been tested and found acceptable.

Quoted 6 lines
Quoted 4 lines

Good to hear. Then let's please get a version of the irq disable that calls shutdown, so we can be certain we don't have hardware irqs in flight.

For the drivers it should not matter for clean cpu shutdown it will.

OK, I will.

My apologies I was wrong. Calling shutdown is not safe.

I just remembered that masking an ioapic from anywhere besides the irq handler can lock the ioapic state machine, and lead to non-recoverable interrupts. It is rare but I have seen it happen. I wanted to figure out how to migrate interrupts outside of interrupt context and this was what prevented me. A suspend/resume cycle might be enough of a reset to get the ioapic out of that state but I don't know.

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Therefore doing anything with the irqs at the point where we are suspending them is a formality, and perhaps simply code that ensures in-flight irqs don't make it past a certain point.

I believe we just need to call disable() and print a big nasty warning if any irq comes in after the suspend stage.

Eric

#36 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 27 lines · Eric W. Biederman
Quoted 9 lines · Rafael J. Wysocki
Quoted 8 lines · Eric W. Biederman
Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines

I don't know where in the state machine this is getting called but I would suggest doing this before we shutdown cpus.

This is the plan. In fact, I'm going to do this in the next patch after the $subject one has been tested and found acceptable.

Quoted 6 lines
Quoted 4 lines

Good to hear. Then let's please get a version of the irq disable that calls shutdown, so we can be certain we don't have hardware irqs in flight.

For the drivers it should not matter for clean cpu shutdown it will.

OK, I will.

My apologies I was wrong. Calling shutdown is not safe.

I just remembered that masking an ioapic from anywhere besides the irq handler can lock the ioapic state machine, and lead to non-recoverable interrupts. It is rare but I have seen it happen. I wanted to figure out how to migrate interrupts outside of interrupt context and this was what prevented me. A suspend/resume cycle might be enough of a reset to get the ioapic out of that state but I don't know.

Quoted 3 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

I think the most important source of level triggered interrupts are PCI devices, so perhaps we can make the PCI PM core use bit 10 of the PCI Device Control register to prevent devices from generating INTx after the drivers' suspend routines have been executed?

Quoted 6 lines

Therefore doing anything with the irqs at the point where we are suspending them is a formality, and perhaps simply code that ensures in-flight irqs don't make it past a certain point.

I believe we just need to call disable() and print a big nasty warning if any irq comes in after the suspend stage.

At the moment we're safe, since PCI devices are put into low power states in the suspend stage. However, we'd like to make that happen in the "late suspend" stage to avoid a problem with a shared interrupt occuring after one of the devices using it has been suspended and its driver's irq handler can't cope with that.

Thanks,
Rafael

#37 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Eric W. Biederman
Quoted 27 lines · Eric W. Biederman
Quoted 9 lines · Rafael J. Wysocki
Quoted 8 lines · Eric W. Biederman
Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines

I don't know where in the state machine this is getting called but I would suggest doing this before we shutdown cpus.

This is the plan. In fact, I'm going to do this in the next patch after the $subject one has been tested and found acceptable.

Quoted 6 lines
Quoted 4 lines

Good to hear. Then let's please get a version of the irq disable that calls shutdown, so we can be certain we don't have hardware irqs in flight.

For the drivers it should not matter for clean cpu shutdown it will.

OK, I will.

My apologies I was wrong. Calling shutdown is not safe.

I just remembered that masking an ioapic from anywhere besides the irq handler can lock the ioapic state machine, and lead to non-recoverable interrupts. It is rare but I have seen it happen. I wanted to figure out how to migrate interrupts outside of interrupt context and this was what prevented me. A suspend/resume cycle might be enough of a reset to get the ioapic out of that state but I don't know.

Quoted 3 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

I think the most important source of level triggered interrupts are PCI devices, so perhaps we can make the PCI PM core use bit 10 of the PCI Device Control register to prevent devices from generating INTx after the drivers' suspend routines have been executed?

Quoted 6 lines

Therefore doing anything with the irqs at the point where we are suspending them is a formality, and perhaps simply code that ensures in-flight irqs don't make it past a certain point.

I believe we just need to call disable() and print a big nasty warning if any irq comes in after the suspend stage.

At the moment we're safe, since PCI devices are put into low power states in the suspend stage. However, we'd like to make that happen in the "late suspend" stage to avoid a problem with a shared interrupt occuring after one of the devices using it has been suspended and its driver's irq handler can't cope with that.

Thanks,
Rafael

#38 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 9 lines · Rafael J. Wysocki
Quoted 4 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

Calling "disable_irq()" is perfectly fine.

What is not possible on that broken IO-APIC (among other things) is to actually turn the interrupts off at the apic (ie the whole ->shutdown() thing). But that's not what we even want to do. What we care about is just disabling the interrupt from a drievr perspective.

IOW, the patches I have seen are fine, and all the comments from Eric are just confusion about what we want done.

WE DO NOT WANT TO TURN OFF THE IO-APIC. That may or may happen later, but that's totally unrelated to this whole "suspend_device_irq()" thing.

Linus

#39 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki
Quoted 4 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

Calling "disable_irq()" is perfectly fine.

What is not possible on that broken IO-APIC (among other things) is to actually turn the interrupts off at the apic (ie the whole ->shutdown() thing). But that's not what we even want to do. What we care about is just disabling the interrupt from a drievr perspective.

IOW, the patches I have seen are fine, and all the comments from Eric are just confusion about what we want done.

WE DO NOT WANT TO TURN OFF THE IO-APIC. That may or may happen later, but that's totally unrelated to this whole "suspend_device_irq()" thing.

Linus

#40 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 14 lines · Linus Torvalds
Quoted 9 lines · Rafael J. Wysocki
Quoted 4 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

Calling "disable_irq()" is perfectly fine.

Quoted 7 lines

What is not possible on that broken IO-APIC (among other things) is to actually turn the interrupts off at the apic (ie the whole ->shutdown() thing). But that's not what we even want to do. What we care about is just disabling the interrupt from a drievr perspective.

IOW, the patches I have seen are fine, and all the comments from Eric are just confusion about what we want done.

Ah, OK. Thanks for the explanation, I got confused too.

Quoted 2 lines

WE DO NOT WANT TO TURN OFF THE IO-APIC. That may or may happen later, but that's totally unrelated to this whole "suspend_device_irq()" thing.

Yeah.

Rafael

#41 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 14 lines · Linus Torvalds
Quoted 9 lines · Rafael J. Wysocki
Quoted 4 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

Calling "disable_irq()" is perfectly fine.

Quoted 7 lines

What is not possible on that broken IO-APIC (among other things) is to actually turn the interrupts off at the apic (ie the whole ->shutdown() thing). But that's not what we even want to do. What we care about is just disabling the interrupt from a drievr perspective.

IOW, the patches I have seen are fine, and all the comments from Eric are just confusion about what we want done.

Ah, OK. Thanks for the explanation, I got confused too.

Quoted 2 lines

WE DO NOT WANT TO TURN OFF THE IO-APIC. That may or may happen later, but that's totally unrelated to this whole "suspend_device_irq()" thing.

Yeah.

Rafael

#42 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 24 lines · Rafael J. Wysocki
Quoted 14 lines · Linus Torvalds
Quoted 9 lines · Rafael J. Wysocki
Quoted 4 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

Calling "disable_irq()" is perfectly fine.

Quoted 8 lines

What is not possible on that broken IO-APIC (among other things) is to actually turn the interrupts off at the apic (ie the whole ->shutdown() thing). But that's not what we even want to do. What we care about is just disabling the interrupt from a drievr perspective.

IOW, the patches I have seen are fine, and all the comments from Eric are just confusion about what we want done.

Quoted 7 lines

Ah, OK. Thanks for the explanation, I got confused too.

Quoted 3 lines

WE DO NOT WANT TO TURN OFF THE IO-APIC. That may or may happen later, but that's totally unrelated to this whole "suspend_device_irq()" thing.

Yeah.

We definitely dont want to turn off x86 IO-APICs - the timer IRQ goes via one of them.

Ingo

#43 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 24 lines · Rafael J. Wysocki
Quoted 14 lines · Linus Torvalds
Quoted 9 lines · Rafael J. Wysocki
Quoted 4 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

Calling "disable_irq()" is perfectly fine.

Quoted 8 lines

What is not possible on that broken IO-APIC (among other things) is to actually turn the interrupts off at the apic (ie the whole ->shutdown() thing). But that's not what we even want to do. What we care about is just disabling the interrupt from a drievr perspective.

IOW, the patches I have seen are fine, and all the comments from Eric are just confusion about what we want done.

Quoted 7 lines

Ah, OK. Thanks for the explanation, I got confused too.

Quoted 3 lines

WE DO NOT WANT TO TURN OFF THE IO-APIC. That may or may happen later, but that's totally unrelated to this whole "suspend_device_irq()" thing.

Yeah.

We definitely dont want to turn off x86 IO-APICs - the timer IRQ goes via one of them.

Ingo

#44 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 29 lines · Ingo Molnar
Quoted 26 lines · Rafael J. Wysocki
Quoted 14 lines · Linus Torvalds
Quoted 9 lines · Rafael J. Wysocki
Quoted 4 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

Calling "disable_irq()" is perfectly fine.

Quoted 8 lines

What is not possible on that broken IO-APIC (among other things) is to actually turn the interrupts off at the apic (ie the whole ->shutdown() thing). But that's not what we even want to do. What we care about is just disabling the interrupt from a drievr perspective.

IOW, the patches I have seen are fine, and all the comments from Eric are just confusion about what we want done.

Ah, OK. Thanks for the explanation, I got confused too.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

WE DO NOT WANT TO TURN OFF THE IO-APIC. That may or may happen later, but that's totally unrelated to this whole "suspend_device_irq()" thing.

Yeah.

We definitely dont want to turn off x86 IO-APICs - the timer IRQ goes via one of them.

No, we don't. At least not at this point.

BTW, appended is the current (3rd) version of the $subject patch with some of your comments taken into account. In particular, I did the following:
- moved [suspend|resume]_device_irqs() to a separate file (pm.c)
- fixed interrupt.h so that their headers are at a better place
- made enable_irq() clear IRQ_SUSPENDED
- made device_power_down() and device_power_up() call suspend_device_irqs() and resume_device_irqs(), respectively, which simplified the callers quite a bit (it changed the Xen code ordering, though, but I _think_ it still should work).

Please have a look.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM: Rework handling of interrupts during suspend-resume (rev. 3)

Introduce two helper functions allowing us to prevent device drivers from getting any interrupts (without disabling interrupts on the CPU) during suspend (or hibernation) and to make them start to receive interrupts again during the subsequent resume, respectively. These functions make it possible to keep timer interrupts enabled while the "late" suspend and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device drivers will be prevented from receiving interrupts, with the help of the new helper function, before their "late" suspend callbacks run (and analogously during resume).

---
 arch/x86/kernel/apm_32.c  |   15 ++++++++--
 drivers/base/power/main.c |   20 ++++++++------
 drivers/xen/manage.c      |   16 ++++++-----
 include/linux/interrupt.h |    4 ++
 include/linux/irq.h       |    1 
 kernel/irq/Makefile       |    1 
 kernel/irq/manage.c       |    3 +-
 kernel/irq/pm.c           |   63 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec.c            |    8 ++---
 kernel/power/disk.c       |   39 +++++++++++++++++++++-------
 kernel/power/main.c       |   17 ++++++++----
 11 files changed, 146 insertions(+), 41 deletions(-)
include/linux/interrupt.h +4 -0
Index: linux-2.6/include/linux/interrupt.h
===================================================================
@@ -106,6 +106,10 @@ extern void disable_irq_nosync(unsigned 
 extern void disable_irq(unsigned int irq);
 extern void enable_irq(unsigned int irq);
 
+/* The following two functions are for the core kernel use only. */
+extern void suspend_device_irqs(void);
+extern void resume_device_irqs(void);
+
 #if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_HARDIRQS)
 
 extern cpumask_var_t irq_default_affinity;
kernel/power/main.c +11 -6
Index: linux-2.6/kernel/power/main.c
===================================================================
@@ -287,17 +287,19 @@ void __attribute__ ((weak)) arch_suspend
  */
 static int suspend_enter(suspend_state_t state)
 {
-	int error = 0;
+	int error;
 
 	device_pm_lock();
-	arch_suspend_disable_irqs();
-	BUG_ON(!irqs_disabled());
 
-	if ((error = device_power_down(PMSG_SUSPEND))) {
+	error = device_power_down(PMSG_SUSPEND);
+	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down\n");
 		goto Done;
 	}
 
+	arch_suspend_disable_irqs();
+	BUG_ON(!irqs_disabled());
+
 	error = sysdev_suspend(PMSG_SUSPEND);
 	if (!error) {
 		if (!suspend_test(TEST_CORE))
@@ -305,11 +307,14 @@ static int suspend_enter(suspend_state_t
 		sysdev_resume();
 	}
 
-	device_power_up(PMSG_RESUME);
- Done:
 	arch_suspend_enable_irqs();
 	BUG_ON(irqs_disabled());
+
+	device_power_up(PMSG_RESUME);
+
+ Done:
 	device_pm_unlock();
+
 	return error;
 }
 
kernel/power/disk.c +29 -10
Index: linux-2.6/kernel/power/disk.c
===================================================================
@@ -214,7 +214,7 @@ static int create_image(int platform_mod
 		return error;
 
 	device_pm_lock();
-	local_irq_disable();
+
 	/* At this point, device_suspend() has been called, but *not*
 	 * device_power_down(). We *must* call device_power_down() now.
 	 * Otherwise, drivers for some devices (e.g. interrupt controllers)
@@ -225,8 +225,11 @@ static int create_image(int platform_mod
 	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down, "
 			"aborting hibernation\n");
-		goto Enable_irqs;
+		goto Unlock;
 	}
+
+	local_irq_disable();
+
 	sysdev_suspend(PMSG_FREEZE);
 	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down, "
@@ -252,12 +255,16 @@ static int create_image(int platform_mod
 	/* NOTE:  device_power_up() is just a resume() for devices
 	 * that suspended with irqs off ... no overall powerup.
 	 */
+
  Power_up_devices:
+	local_irq_enable();
+
 	device_power_up(in_suspend ?
 		(error ? PMSG_RECOVER : PMSG_THAW) : PMSG_RESTORE);
- Enable_irqs:
-	local_irq_enable();
+
+ Unlock:
 	device_pm_unlock();
+
 	return error;
 }
 
@@ -336,13 +343,16 @@ static int resume_target_kernel(void)
 	int error;
 
 	device_pm_lock();
-	local_irq_disable();
+
 	error = device_power_down(PMSG_QUIESCE);
 	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down, "
 			"aborting resume\n");
-		goto Enable_irqs;
+		goto Unlock;
 	}
+
+	local_irq_disable();
+
 	sysdev_suspend(PMSG_QUIESCE);
 	/* We'll ignore saved state, but this gets preempt count (etc) right */
 	save_processor_state();
@@ -366,11 +376,16 @@ static int resume_target_kernel(void)
 	swsusp_free();
 	restore_processor_state();
 	touch_softlockup_watchdog();
+
 	sysdev_resume();
-	device_power_up(PMSG_RECOVER);
- Enable_irqs:
+
 	local_irq_enable();
+
+	device_power_up(PMSG_RECOVER);
+
+ Unlock:
 	device_pm_unlock();
+
 	return error;
 }
 
@@ -447,15 +462,16 @@ int hibernation_platform_enter(void)
 		goto Finish;
 
 	device_pm_lock();
-	local_irq_disable();
+
 	error = device_power_down(PMSG_HIBERNATE);
 	if (!error) {
+		local_irq_disable();
 		sysdev_suspend(PMSG_HIBERNATE);
 		hibernation_ops->enter();
 		/* We should never get here */
 		while (1);
 	}
-	local_irq_enable();
+
 	device_pm_unlock();
 
 	/*
@@ -464,12 +480,15 @@ int hibernation_platform_enter(void)
 	 */
  Finish:
 	hibernation_ops->finish();
+
  Resume_devices:
 	entering_platform_hibernation = false;
 	device_resume(PMSG_RESTORE);
 	resume_console();
+
  Close:
 	hibernation_ops->end();
+
 	return error;
 }
 
arch/x86/kernel/apm_32.c +11 -4
Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================
@@ -1190,8 +1190,10 @@ static int suspend(int vetoable)
 	struct apm_user	*as;
 
 	device_suspend(PMSG_SUSPEND);
-	local_irq_disable();
+
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 
 	local_irq_enable();
@@ -1209,9 +1211,12 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
+
 	device_resume(PMSG_RESUME);
 	queue_event(APM_NORMAL_RESUME, NULL);
 	spin_lock(&user_list_lock);
@@ -1228,8 +1233,9 @@ static void standby(void)
 {
 	int err;
 
-	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 	local_irq_enable();
 
@@ -1239,8 +1245,9 @@ static void standby(void)
 
 	local_irq_disable();
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
 }
 
 static apm_event_t get_event(void)
drivers/xen/manage.c +9 -7
Index: linux-2.6/drivers/xen/manage.c
===================================================================
@@ -39,12 +39,6 @@ static int xen_suspend(void *data)
 
 	BUG_ON(!irqs_disabled());
 
-	err = device_power_down(PMSG_SUSPEND);
-	if (err) {
-		printk(KERN_ERR "xen_suspend: device_power_down failed: %d\n",
-		       err);
-		return err;
-	}
 	err = sysdev_suspend(PMSG_SUSPEND);
 	if (err) {
 		printk(KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n",
@@ -69,7 +63,6 @@ static int xen_suspend(void *data)
 	xen_mm_unpin_all();
 
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 
 	if (!*cancelled) {
 		xen_irq_resume();
@@ -108,6 +101,12 @@ static void do_suspend(void)
 	/* XXX use normal device tree? */
 	xenbus_suspend();
 
+	err = device_power_down(PMSG_SUSPEND);
+	if (err) {
+		printk(KERN_ERR "device_power_down failed: %d\n", err);
+		goto resume_devices;
+	}
+
 	err = stop_machine(xen_suspend, &cancelled, &cpumask_of_cpu(0));
 	if (err) {
 		printk(KERN_ERR "failed to start xen_suspend: %d\n", err);
@@ -120,6 +119,9 @@ static void do_suspend(void)
 	} else
 		xenbus_suspend_cancel();
 
+	device_power_up(PMSG_RESUME);
+
+resume_devices:
 	device_resume(PMSG_RESUME);
 
 	/* Make sure timer events get retriggered on all CPUs */
kernel/kexec.c +4 -4
Index: linux-2.6/kernel/kexec.c
===================================================================
@@ -1454,7 +1454,6 @@ int kernel_kexec(void)
 		if (error)
 			goto Resume_devices;
 		device_pm_lock();
-		local_irq_disable();
 		/* At this point, device_suspend() has been called,
 		 * but *not* device_power_down(). We *must*
 		 * device_power_down() now.  Otherwise, drivers for
@@ -1464,8 +1463,9 @@ int kernel_kexec(void)
 		 */
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
-			goto Enable_irqs;
+			goto Unlock_pm;
 
+		local_irq_disable();
 		/* Suspend system devices */
 		error = sysdev_suspend(PMSG_FREEZE);
 		if (error)
@@ -1484,9 +1484,9 @@ int kernel_kexec(void)
 	if (kexec_image->preserve_context) {
 		sysdev_resume();
  Power_up_devices:
-		device_power_up(PMSG_RESTORE);
- Enable_irqs:
 		local_irq_enable();
+		device_power_up(PMSG_RESTORE);
+ Unlock_pm:
 		device_pm_unlock();
 		enable_nonboot_cpus();
  Resume_devices:
include/linux/irq.h +1 -0
Index: linux-2.6/include/linux/irq.h
===================================================================
@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsig
 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
 
 #ifdef CONFIG_IRQ_PER_CPU
 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)
kernel/irq/pm.c new file +63 -0

63 lines not on this page · raw message

kernel/irq/Makefile +1 -0
Index: linux-2.6/kernel/irq/Makefile
===================================================================
@@ -4,3 +4,4 @@ obj-$(CONFIG_GENERIC_IRQ_PROBE) += autop
 obj-$(CONFIG_PROC_FS) += proc.o
 obj-$(CONFIG_GENERIC_PENDING_IRQ) += migration.o
 obj-$(CONFIG_NUMA_MIGRATE_IRQ_DESC) += numa_migrate.o
+obj-$(CONFIG_PM_SLEEP) += pm.o
kernel/irq/manage.c +2 -1
Index: linux-2.6/kernel/irq/manage.c
===================================================================
@@ -222,8 +222,9 @@ static void __enable_irq(struct irq_desc
 		WARN(1, KERN_WARNING "Unbalanced enable for IRQ %d\n", irq);
 		break;
 	case 1: {
-		unsigned int status = desc->status & ~IRQ_DISABLED;
+		unsigned int status;
 
+		status = desc->status & ~(IRQ_DISABLED | IRQ_SUSPENDED);
 		/* Prevent probing on this irq: */
 		desc->status = status | IRQ_NOPROBE;
 		check_irq_resend(desc, irq);
drivers/base/power/main.c +11 -9
Index: linux-2.6/drivers/base/power/main.c
===================================================================
@@ -23,6 +23,7 @@
 #include <linux/pm.h>
 #include <linux/resume-trace.h>
 #include <linux/rwsem.h>
+#include <linux/interrupt.h>
 
 #include "../base.h"
 #include "power.h"
@@ -305,7 +306,8 @@ static int resume_device_noirq(struct de
  *	Execute the appropriate "noirq resume" callback for all devices marked
  *	as DPM_OFF_IRQ.
  *
- *	Must be called with interrupts disabled and only one CPU running.
+ *	Must be called under dpm_list_mtx.  Device drivers should not receive
+ *	interrupts while it's being executed.
  */
 static void dpm_power_up(pm_message_t state)
 {
@@ -326,14 +328,13 @@ static void dpm_power_up(pm_message_t st
  *	device_power_up - Turn on all devices that need special attention.
  *	@state: PM transition of the system being carried out.
  *
- *	Power on system devices, then devices that required we shut them down
- *	with interrupts disabled.
- *
- *	Must be called with interrupts disabled.
+ *	Call the "early" resume handlers and enable device drivers to receive
+ *	interrupts.
  */
 void device_power_up(pm_message_t state)
 {
 	dpm_power_up(state);
+	resume_device_irqs();
 }
 EXPORT_SYMBOL_GPL(device_power_up);
 
@@ -558,16 +559,17 @@ static int suspend_device_noirq(struct d
  *	device_power_down - Shut down special devices.
  *	@state: PM transition of the system being carried out.
  *
- *	Power down devices that require interrupts to be disabled.
- *	Then power down system devices.
+ *	Prevent device drivers from receiving interrupts and call the "late"
+ *	suspend handlers.
  *
- *	Must be called with interrupts disabled and only one CPU running.
+ *	Must be called under dpm_list_mtx.
  */
 int device_power_down(pm_message_t state)
 {
 	struct device *dev;
 	int error = 0;
 
+	suspend_device_irqs();
 	list_for_each_entry_reverse(dev, &dpm_list, power.entry) {
 		error = suspend_device_noirq(dev, state);
 		if (error) {
@@ -577,7 +579,7 @@ int device_power_down(pm_message_t state
 		dev->power.status = DPM_OFF_IRQ;
 	}
 	if (error)
-		dpm_power_up(resume_event(state));
+		device_power_up(resume_event(state));
 	return error;
 }
 EXPORT_SYMBOL_GPL(device_power_down);
#45 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Ingo Molnar
Quoted 29 lines · Ingo Molnar
Quoted 26 lines · Rafael J. Wysocki
Quoted 14 lines · Linus Torvalds
Quoted 9 lines · Rafael J. Wysocki
Quoted 4 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

Calling "disable_irq()" is perfectly fine.

Quoted 8 lines

What is not possible on that broken IO-APIC (among other things) is to actually turn the interrupts off at the apic (ie the whole ->shutdown() thing). But that's not what we even want to do. What we care about is just disabling the interrupt from a drievr perspective.

IOW, the patches I have seen are fine, and all the comments from Eric are just confusion about what we want done.

Ah, OK. Thanks for the explanation, I got confused too.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

WE DO NOT WANT TO TURN OFF THE IO-APIC. That may or may happen later, but that's totally unrelated to this whole "suspend_device_irq()" thing.

Yeah.

We definitely dont want to turn off x86 IO-APICs - the timer IRQ goes via one of them.

No, we don't. At least not at this point.

BTW, appended is the current (3rd) version of the $subject patch with some of your comments taken into account. In particular, I did the following:
- moved [suspend|resume]_device_irqs() to a separate file (pm.c)
- fixed interrupt.h so that their headers are at a better place
- made enable_irq() clear IRQ_SUSPENDED
- made device_power_down() and device_power_up() call suspend_device_irqs() and resume_device_irqs(), respectively, which simplified the callers quite a bit (it changed the Xen code ordering, though, but I _think_ it still should work).

Please have a look.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM: Rework handling of interrupts during suspend-resume (rev. 3)

Introduce two helper functions allowing us to prevent device drivers from getting any interrupts (without disabling interrupts on the CPU) during suspend (or hibernation) and to make them start to receive interrupts again during the subsequent resume, respectively. These functions make it possible to keep timer interrupts enabled while the "late" suspend and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device drivers will be prevented from receiving interrupts, with the help of the new helper function, before their "late" suspend callbacks run (and analogously during resume).

---
 arch/x86/kernel/apm_32.c  |   15 ++++++++--
 drivers/base/power/main.c |   20 ++++++++------
 drivers/xen/manage.c      |   16 ++++++-----
 include/linux/interrupt.h |    4 ++
 include/linux/irq.h       |    1 
 kernel/irq/Makefile       |    1 
 kernel/irq/manage.c       |    3 +-
 kernel/irq/pm.c           |   63 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec.c            |    8 ++---
 kernel/power/disk.c       |   39 +++++++++++++++++++++-------
 kernel/power/main.c       |   17 ++++++++----
 11 files changed, 146 insertions(+), 41 deletions(-)
include/linux/interrupt.h +4 -0
Index: linux-2.6/include/linux/interrupt.h
===================================================================
@@ -106,6 +106,10 @@ extern void disable_irq_nosync(unsigned 
 extern void disable_irq(unsigned int irq);
 extern void enable_irq(unsigned int irq);
 
+/* The following two functions are for the core kernel use only. */
+extern void suspend_device_irqs(void);
+extern void resume_device_irqs(void);
+
 #if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_HARDIRQS)
 
 extern cpumask_var_t irq_default_affinity;
kernel/power/main.c +11 -6
Index: linux-2.6/kernel/power/main.c
===================================================================
@@ -287,17 +287,19 @@ void __attribute__ ((weak)) arch_suspend
  */
 static int suspend_enter(suspend_state_t state)
 {
-	int error = 0;
+	int error;
 
 	device_pm_lock();
-	arch_suspend_disable_irqs();
-	BUG_ON(!irqs_disabled());
 
-	if ((error = device_power_down(PMSG_SUSPEND))) {
+	error = device_power_down(PMSG_SUSPEND);
+	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down\n");
 		goto Done;
 	}
 
+	arch_suspend_disable_irqs();
+	BUG_ON(!irqs_disabled());
+
 	error = sysdev_suspend(PMSG_SUSPEND);
 	if (!error) {
 		if (!suspend_test(TEST_CORE))
@@ -305,11 +307,14 @@ static int suspend_enter(suspend_state_t
 		sysdev_resume();
 	}
 
-	device_power_up(PMSG_RESUME);
- Done:
 	arch_suspend_enable_irqs();
 	BUG_ON(irqs_disabled());
+
+	device_power_up(PMSG_RESUME);
+
+ Done:
 	device_pm_unlock();
+
 	return error;
 }
 
kernel/power/disk.c +29 -10
Index: linux-2.6/kernel/power/disk.c
===================================================================
@@ -214,7 +214,7 @@ static int create_image(int platform_mod
 		return error;
 
 	device_pm_lock();
-	local_irq_disable();
+
 	/* At this point, device_suspend() has been called, but *not*
 	 * device_power_down(). We *must* call device_power_down() now.
 	 * Otherwise, drivers for some devices (e.g. interrupt controllers)
@@ -225,8 +225,11 @@ static int create_image(int platform_mod
 	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down, "
 			"aborting hibernation\n");
-		goto Enable_irqs;
+		goto Unlock;
 	}
+
+	local_irq_disable();
+
 	sysdev_suspend(PMSG_FREEZE);
 	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down, "
@@ -252,12 +255,16 @@ static int create_image(int platform_mod
 	/* NOTE:  device_power_up() is just a resume() for devices
 	 * that suspended with irqs off ... no overall powerup.
 	 */
+
  Power_up_devices:
+	local_irq_enable();
+
 	device_power_up(in_suspend ?
 		(error ? PMSG_RECOVER : PMSG_THAW) : PMSG_RESTORE);
- Enable_irqs:
-	local_irq_enable();
+
+ Unlock:
 	device_pm_unlock();
+
 	return error;
 }
 
@@ -336,13 +343,16 @@ static int resume_target_kernel(void)
 	int error;
 
 	device_pm_lock();
-	local_irq_disable();
+
 	error = device_power_down(PMSG_QUIESCE);
 	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down, "
 			"aborting resume\n");
-		goto Enable_irqs;
+		goto Unlock;
 	}
+
+	local_irq_disable();
+
 	sysdev_suspend(PMSG_QUIESCE);
 	/* We'll ignore saved state, but this gets preempt count (etc) right */
 	save_processor_state();
@@ -366,11 +376,16 @@ static int resume_target_kernel(void)
 	swsusp_free();
 	restore_processor_state();
 	touch_softlockup_watchdog();
+
 	sysdev_resume();
-	device_power_up(PMSG_RECOVER);
- Enable_irqs:
+
 	local_irq_enable();
+
+	device_power_up(PMSG_RECOVER);
+
+ Unlock:
 	device_pm_unlock();
+
 	return error;
 }
 
@@ -447,15 +462,16 @@ int hibernation_platform_enter(void)
 		goto Finish;
 
 	device_pm_lock();
-	local_irq_disable();
+
 	error = device_power_down(PMSG_HIBERNATE);
 	if (!error) {
+		local_irq_disable();
 		sysdev_suspend(PMSG_HIBERNATE);
 		hibernation_ops->enter();
 		/* We should never get here */
 		while (1);
 	}
-	local_irq_enable();
+
 	device_pm_unlock();
 
 	/*
@@ -464,12 +480,15 @@ int hibernation_platform_enter(void)
 	 */
  Finish:
 	hibernation_ops->finish();
+
  Resume_devices:
 	entering_platform_hibernation = false;
 	device_resume(PMSG_RESTORE);
 	resume_console();
+
  Close:
 	hibernation_ops->end();
+
 	return error;
 }
 
arch/x86/kernel/apm_32.c +11 -4
Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================
@@ -1190,8 +1190,10 @@ static int suspend(int vetoable)
 	struct apm_user	*as;
 
 	device_suspend(PMSG_SUSPEND);
-	local_irq_disable();
+
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 
 	local_irq_enable();
@@ -1209,9 +1211,12 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
+
 	device_resume(PMSG_RESUME);
 	queue_event(APM_NORMAL_RESUME, NULL);
 	spin_lock(&user_list_lock);
@@ -1228,8 +1233,9 @@ static void standby(void)
 {
 	int err;
 
-	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 	local_irq_enable();
 
@@ -1239,8 +1245,9 @@ static void standby(void)
 
 	local_irq_disable();
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
 }
 
 static apm_event_t get_event(void)
drivers/xen/manage.c +9 -7
Index: linux-2.6/drivers/xen/manage.c
===================================================================
@@ -39,12 +39,6 @@ static int xen_suspend(void *data)
 
 	BUG_ON(!irqs_disabled());
 
-	err = device_power_down(PMSG_SUSPEND);
-	if (err) {
-		printk(KERN_ERR "xen_suspend: device_power_down failed: %d\n",
-		       err);
-		return err;
-	}
 	err = sysdev_suspend(PMSG_SUSPEND);
 	if (err) {
 		printk(KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n",
@@ -69,7 +63,6 @@ static int xen_suspend(void *data)
 	xen_mm_unpin_all();
 
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 
 	if (!*cancelled) {
 		xen_irq_resume();
@@ -108,6 +101,12 @@ static void do_suspend(void)
 	/* XXX use normal device tree? */
 	xenbus_suspend();
 
+	err = device_power_down(PMSG_SUSPEND);
+	if (err) {
+		printk(KERN_ERR "device_power_down failed: %d\n", err);
+		goto resume_devices;
+	}
+
 	err = stop_machine(xen_suspend, &cancelled, &cpumask_of_cpu(0));
 	if (err) {
 		printk(KERN_ERR "failed to start xen_suspend: %d\n", err);
@@ -120,6 +119,9 @@ static void do_suspend(void)
 	} else
 		xenbus_suspend_cancel();
 
+	device_power_up(PMSG_RESUME);
+
+resume_devices:
 	device_resume(PMSG_RESUME);
 
 	/* Make sure timer events get retriggered on all CPUs */
kernel/kexec.c +4 -4
Index: linux-2.6/kernel/kexec.c
===================================================================
@@ -1454,7 +1454,6 @@ int kernel_kexec(void)
 		if (error)
 			goto Resume_devices;
 		device_pm_lock();
-		local_irq_disable();
 		/* At this point, device_suspend() has been called,
 		 * but *not* device_power_down(). We *must*
 		 * device_power_down() now.  Otherwise, drivers for
@@ -1464,8 +1463,9 @@ int kernel_kexec(void)
 		 */
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
-			goto Enable_irqs;
+			goto Unlock_pm;
 
+		local_irq_disable();
 		/* Suspend system devices */
 		error = sysdev_suspend(PMSG_FREEZE);
 		if (error)
@@ -1484,9 +1484,9 @@ int kernel_kexec(void)
 	if (kexec_image->preserve_context) {
 		sysdev_resume();
  Power_up_devices:
-		device_power_up(PMSG_RESTORE);
- Enable_irqs:
 		local_irq_enable();
+		device_power_up(PMSG_RESTORE);
+ Unlock_pm:
 		device_pm_unlock();
 		enable_nonboot_cpus();
  Resume_devices:
include/linux/irq.h +1 -0
Index: linux-2.6/include/linux/irq.h
===================================================================
@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsig
 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
 
 #ifdef CONFIG_IRQ_PER_CPU
 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)
kernel/irq/pm.c new file +63 -0

63 lines not on this page · raw message

kernel/irq/Makefile +1 -0
Index: linux-2.6/kernel/irq/Makefile
===================================================================
@@ -4,3 +4,4 @@ obj-$(CONFIG_GENERIC_IRQ_PROBE) += autop
 obj-$(CONFIG_PROC_FS) += proc.o
 obj-$(CONFIG_GENERIC_PENDING_IRQ) += migration.o
 obj-$(CONFIG_NUMA_MIGRATE_IRQ_DESC) += numa_migrate.o
+obj-$(CONFIG_PM_SLEEP) += pm.o
kernel/irq/manage.c +2 -1
Index: linux-2.6/kernel/irq/manage.c
===================================================================
@@ -222,8 +222,9 @@ static void __enable_irq(struct irq_desc
 		WARN(1, KERN_WARNING "Unbalanced enable for IRQ %d\n", irq);
 		break;
 	case 1: {
-		unsigned int status = desc->status & ~IRQ_DISABLED;
+		unsigned int status;
 
+		status = desc->status & ~(IRQ_DISABLED | IRQ_SUSPENDED);
 		/* Prevent probing on this irq: */
 		desc->status = status | IRQ_NOPROBE;
 		check_irq_resend(desc, irq);
drivers/base/power/main.c +11 -9
Index: linux-2.6/drivers/base/power/main.c
===================================================================
@@ -23,6 +23,7 @@
 #include <linux/pm.h>
 #include <linux/resume-trace.h>
 #include <linux/rwsem.h>
+#include <linux/interrupt.h>
 
 #include "../base.h"
 #include "power.h"
@@ -305,7 +306,8 @@ static int resume_device_noirq(struct de
  *	Execute the appropriate "noirq resume" callback for all devices marked
  *	as DPM_OFF_IRQ.
  *
- *	Must be called with interrupts disabled and only one CPU running.
+ *	Must be called under dpm_list_mtx.  Device drivers should not receive
+ *	interrupts while it's being executed.
  */
 static void dpm_power_up(pm_message_t state)
 {
@@ -326,14 +328,13 @@ static void dpm_power_up(pm_message_t st
  *	device_power_up - Turn on all devices that need special attention.
  *	@state: PM transition of the system being carried out.
  *
- *	Power on system devices, then devices that required we shut them down
- *	with interrupts disabled.
- *
- *	Must be called with interrupts disabled.
+ *	Call the "early" resume handlers and enable device drivers to receive
+ *	interrupts.
  */
 void device_power_up(pm_message_t state)
 {
 	dpm_power_up(state);
+	resume_device_irqs();
 }
 EXPORT_SYMBOL_GPL(device_power_up);
 
@@ -558,16 +559,17 @@ static int suspend_device_noirq(struct d
  *	device_power_down - Shut down special devices.
  *	@state: PM transition of the system being carried out.
  *
- *	Power down devices that require interrupts to be disabled.
- *	Then power down system devices.
+ *	Prevent device drivers from receiving interrupts and call the "late"
+ *	suspend handlers.
  *
- *	Must be called with interrupts disabled and only one CPU running.
+ *	Must be called under dpm_list_mtx.
  */
 int device_power_down(pm_message_t state)
 {
 	struct device *dev;
 	int error = 0;
 
+	suspend_device_irqs();
 	list_for_each_entry_reverse(dev, &dpm_list, power.entry) {
 		error = suspend_device_noirq(dev, state);
 		if (error) {
@@ -577,7 +579,7 @@ int device_power_down(pm_message_t state
 		dev->power.status = DPM_OFF_IRQ;
 	}
 	if (error)
-		dpm_power_up(resume_event(state));
+		device_power_up(resume_event(state));
 	return error;
 }
 EXPORT_SYMBOL_GPL(device_power_down);
Acked-by
#46 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 51 lines · Rafael J. Wysocki
Quoted 29 lines · Ingo Molnar
Quoted 26 lines · Rafael J. Wysocki
Quoted 14 lines · Linus Torvalds
Quoted 9 lines · Rafael J. Wysocki
Quoted 4 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

Calling "disable_irq()" is perfectly fine.

Quoted 8 lines

What is not possible on that broken IO-APIC (among other things) is to actually turn the interrupts off at the apic (ie the whole ->shutdown() thing). But that's not what we even want to do. What we care about is just disabling the interrupt from a drievr perspective.

IOW, the patches I have seen are fine, and all the comments from Eric are just confusion about what we want done.

Ah, OK. Thanks for the explanation, I got confused too.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

WE DO NOT WANT TO TURN OFF THE IO-APIC. That may or may happen later, but that's totally unrelated to this whole "suspend_device_irq()" thing.

Yeah.

We definitely dont want to turn off x86 IO-APICs - the timer IRQ goes via one of them.

No, we don't. At least not at this point.

BTW, appended is the current (3rd) version of the $subject patch with some of your comments taken into account. In particular, I did the following:
- moved [suspend|resume]_device_irqs() to a separate file (pm.c)
- fixed interrupt.h so that their headers are at a better place
- made enable_irq() clear IRQ_SUSPENDED
- made device_power_down() and device_power_up() call suspend_device_irqs() and resume_device_irqs(), respectively, which simplified the callers quite a bit (it changed the Xen code ordering, though, but I _think_ it still should work).

Quoted 1 line

Please have a look.

Looks good, thanks Rafael!

Ingo

Acked-by
#47 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 51 lines · Rafael J. Wysocki
Quoted 29 lines · Ingo Molnar
Quoted 26 lines · Rafael J. Wysocki
Quoted 14 lines · Linus Torvalds
Quoted 9 lines · Rafael J. Wysocki
Quoted 4 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

Calling "disable_irq()" is perfectly fine.

Quoted 8 lines

What is not possible on that broken IO-APIC (among other things) is to actually turn the interrupts off at the apic (ie the whole ->shutdown() thing). But that's not what we even want to do. What we care about is just disabling the interrupt from a drievr perspective.

IOW, the patches I have seen are fine, and all the comments from Eric are just confusion about what we want done.

Ah, OK. Thanks for the explanation, I got confused too.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

WE DO NOT WANT TO TURN OFF THE IO-APIC. That may or may happen later, but that's totally unrelated to this whole "suspend_device_irq()" thing.

Yeah.

We definitely dont want to turn off x86 IO-APICs - the timer IRQ goes via one of them.

No, we don't. At least not at this point.

BTW, appended is the current (3rd) version of the $subject patch with some of your comments taken into account. In particular, I did the following:
- moved [suspend|resume]_device_irqs() to a separate file (pm.c)
- fixed interrupt.h so that their headers are at a better place
- made enable_irq() clear IRQ_SUSPENDED
- made device_power_down() and device_power_up() call suspend_device_irqs() and resume_device_irqs(), respectively, which simplified the callers quite a bit (it changed the Xen code ordering, though, but I _think_ it still should work).

Quoted 1 line

Please have a look.

Looks good, thanks Rafael!

Ingo

#48 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki

BTW, appended is the current (3rd) version of the $subject patch with some of your comments taken into account.  In particular, I did the following:
- moved [suspend|resume]_device_irqs() to a separate file (pm.c)
- fixed interrupt.h so that their headers are at a better place
- made enable_irq() clear IRQ_SUSPENDED
- made device_power_down() and device_power_up() call suspend_device_irqs() and resume_device_irqs(), respectively, which simplified the callers quite a bit (it changed the Xen code ordering, though, but I _think_ it still should work).

Do you plan to fix edge triggered wakeup interrupts? It still looks like edge triggered wakeup interrupts that occur between suspend_device_irqs and local_irq_disable will not cause a wakeup.

Signature · 1 line
Arve Hjønnevåg
#49 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki

BTW, appended is the current (3rd) version of the $subject patch with some of your comments taken into account.  In particular, I did the following:
- moved [suspend|resume]_device_irqs() to a separate file (pm.c)
- fixed interrupt.h so that their headers are at a better place
- made enable_irq() clear IRQ_SUSPENDED
- made device_power_down() and device_power_up() call suspend_device_irqs() and resume_device_irqs(), respectively, which simplified the callers quite a bit (it changed the Xen code ordering, though, but I _think_ it still should work).

Do you plan to fix edge triggered wakeup interrupts? It still looks like edge triggered wakeup interrupts that occur between suspend_device_irqs and local_irq_disable will not cause a wakeup.

Signature · 1 line
Arve Hjønnevåg
#50 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 4 lines · Arve Hjønnevåg

Do you plan to fix edge triggered wakeup interrupts? It still looks like edge triggered wakeup interrupts that occur between suspend_device_irqs and local_irq_disable will not cause a wakeup.

IF we ever see this as a real issue, we can either see it in the IRQ_PENDING flag, or we can mark such interrupts specially. So it would be solvable. That said, I haven't actually heard any real usage cases. Normal wakeup events are _not_ interrupts in the regular "device interrupt controller" sense.

So can you actually point to an explicit example of something where this is a real issue?

Linus

#51 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Arve Hjønnevåg
Quoted 4 lines · Arve Hjønnevåg

Do you plan to fix edge triggered wakeup interrupts? It still looks like edge triggered wakeup interrupts that occur between suspend_device_irqs and local_irq_disable will not cause a wakeup.

IF we ever see this as a real issue, we can either see it in the IRQ_PENDING flag, or we can mark such interrupts specially. So it would be solvable. That said, I haven't actually heard any real usage cases. Normal wakeup events are _not_ interrupts in the regular "device interrupt controller" sense.

So can you actually point to an explicit example of something where this is a real issue?

Linus

#52 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Arve Hjønnevåg

Do you plan to fix edge triggered wakeup interrupts? It still looks like edge triggered wakeup interrupts that occur between suspend_device_irqs and local_irq_disable will not cause a wakeup.

Quoted 8 lines

IF we ever see this as a real issue, we can either see it in the IRQ_PENDING flag, or we can mark such interrupts specially. So it would be solvable. That said, I haven't actually heard any real usage cases. Normal wakeup events are _not_ interrupts in the regular "device interrupt controller" sense.

So can you actually point to an explicit example of something where this is a real issue?

On the msm platform the keyboard driver currently leave the interrupts enabled when suspended. If the interrupt handler is called, we use a wakelock to abort suspend (without wakelocks you would need to set a flag and abort in suspend_late instead). If the interrupt occurs after local_irq_disable, it will still be pending when we get to the suspend enter hook and suspend will be aborted there.

As far as I can tell, this change breaks this. If you press a key at the right time, it will be ignored.

Signature · 1 line
Arve Hjønnevåg
#53 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Arve Hjønnevåg

Do you plan to fix edge triggered wakeup interrupts? It still looks like edge triggered wakeup interrupts that occur between suspend_device_irqs and local_irq_disable will not cause a wakeup.

Quoted 8 lines

IF we ever see this as a real issue, we can either see it in the IRQ_PENDING flag, or we can mark such interrupts specially. So it would be solvable. That said, I haven't actually heard any real usage cases. Normal wakeup events are _not_ interrupts in the regular "device interrupt controller" sense.

So can you actually point to an explicit example of something where this is a real issue?

On the msm platform the keyboard driver currently leave the interrupts enabled when suspended. If the interrupt handler is called, we use a wakelock to abort suspend (without wakelocks you would need to set a flag and abort in suspend_late instead). If the interrupt occurs after local_irq_disable, it will still be pending when we get to the suspend enter hook and suspend will be aborted there.

As far as I can tell, this change breaks this. If you press a key at the right time, it will be ignored.

Signature · 1 line
Arve Hjønnevåg
#54 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 10 lines · Arve Hjønnevåg

On the msm platform the keyboard driver currently leave the interrupts enabled when suspended. If the interrupt handler is called, we use a wakelock to abort suspend (without wakelocks you would need to set a flag and abort in suspend_late instead). If the interrupt occurs after local_irq_disable, it will still be pending when we get to the suspend enter hook and suspend will be aborted there.

As far as I can tell, this change breaks this. If you press a key at the right time, it will be ignored.

Is the irq on a private non-shared interrupt line? If so, you could just mark it as IRQF_TIMER, and the irq disable logic won't touch it.

What keyboard driver does this mfm thing, btw?

Linus

#55 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Arve Hjønnevåg
Quoted 10 lines · Arve Hjønnevåg

On the msm platform the keyboard driver currently leave the interrupts enabled when suspended. If the interrupt handler is called, we use a wakelock to abort suspend (without wakelocks you would need to set a flag and abort in suspend_late instead). If the interrupt occurs after local_irq_disable, it will still be pending when we get to the suspend enter hook and suspend will be aborted there.

As far as I can tell, this change breaks this. If you press a key at the right time, it will be ignored.

Is the irq on a private non-shared interrupt line? If so, you could just mark it as IRQF_TIMER, and the irq disable logic won't touch it.

What keyboard driver does this mfm thing, btw?

Linus

#56 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 11 lines · Linus Torvalds
Quoted 8 lines · Arve Hjønnevåg

On the msm platform the keyboard driver currently leave the interrupts enabled when suspended. If the interrupt handler is called, we use a wakelock to abort suspend (without wakelocks you would need to set a flag and abort in suspend_late instead). If the interrupt occurs after local_irq_disable, it will still be pending when we get to the suspend enter hook and suspend will be aborted there.

Quoted 6 lines
Quoted 2 lines

As far as I can tell, this change breaks this. If you press a key at the right time, it will be ignored.

Is the irq on a private non-shared interrupt line? If so, you could just mark it as IRQF_TIMER, and the irq disable logic won't touch it.

Hm, if that solves the problem then it would be nice to have a new IRQF_NO_SUSPEND flag for it, in addition to IRQF_TIMER:

./interrupt.h: * IRQF_TIMER - Flag to mark this interrupt as timer interrupt
./interrupt.h:#define IRQF_TIMER 0x00000200

to express such quirks cleanly.

and the suspend code can check the (IRQF_TIMER |
IRQF_NO_SUSPEND) mask - so no extra cost.

Right now we have a clean enumeration of timer interrupts, would be nice to keep that.

Ingo

#57 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 8 lines · Arve Hjønnevåg

On the msm platform the keyboard driver currently leave the interrupts enabled when suspended. If the interrupt handler is called, we use a wakelock to abort suspend (without wakelocks you would need to set a flag and abort in suspend_late instead). If the interrupt occurs after local_irq_disable, it will still be pending when we get to the suspend enter hook and suspend will be aborted there.

Quoted 6 lines
Quoted 2 lines

As far as I can tell, this change breaks this. If you press a key at the right time, it will be ignored.

Is the irq on a private non-shared interrupt line? If so, you could just mark it as IRQF_TIMER, and the irq disable logic won't touch it.

Hm, if that solves the problem then it would be nice to have a new IRQF_NO_SUSPEND flag for it, in addition to IRQF_TIMER:

./interrupt.h: * IRQF_TIMER - Flag to mark this interrupt as timer interrupt
./interrupt.h:#define IRQF_TIMER 0x00000200

to express such quirks cleanly.

and the suspend code can check the (IRQF_TIMER |
IRQF_NO_SUSPEND) mask - so no extra cost.

Right now we have a clean enumeration of timer interrupts, would be nice to keep that.

Ingo

#58 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 14 lines · Ingo Molnar
Quoted 11 lines · Linus Torvalds
Quoted 8 lines · Arve Hjønnevåg

On the msm platform the keyboard driver currently leave the interrupts enabled when suspended. If the interrupt handler is called, we use a wakelock to abort suspend (without wakelocks you would need to set a flag and abort in suspend_late instead). If the interrupt occurs after local_irq_disable, it will still be pending when we get to the suspend enter hook and suspend will be aborted there.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

As far as I can tell, this change breaks this. If you press a key at the right time, it will be ignored.

Is the irq on a private non-shared interrupt line? If so, you could just mark it as IRQF_TIMER, and the irq disable logic won't touch it.

That would not work without wakelocks support, since the interrupt could occur after suspend_late which is the last chance for the driver to abort sleep. (The patch also breaks my current wakelock implementation since I use a suspend_late hook to abort sleep, but this should be easy to fix)

Quoted 2 lines

Hm, if that solves the problem then it would be nice to have a new IRQF_NO_SUSPEND flag for it, in addition to IRQF_TIMER:

I think the right fix is for any interrupt that has IRQ_WAKEUP set to abort suspend if it is pending. I don't know if anyone relies on these interrupts being dropped now though.

Signature · 1 line
Arve Hjønnevåg
#59 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Ingo Molnar
Quoted 14 lines · Ingo Molnar
Quoted 11 lines · Linus Torvalds
Quoted 8 lines · Arve Hjønnevåg

On the msm platform the keyboard driver currently leave the interrupts enabled when suspended. If the interrupt handler is called, we use a wakelock to abort suspend (without wakelocks you would need to set a flag and abort in suspend_late instead). If the interrupt occurs after local_irq_disable, it will still be pending when we get to the suspend enter hook and suspend will be aborted there.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

As far as I can tell, this change breaks this. If you press a key at the right time, it will be ignored.

Is the irq on a private non-shared interrupt line? If so, you could just mark it as IRQF_TIMER, and the irq disable logic won't touch it.

That would not work without wakelocks support, since the interrupt could occur after suspend_late which is the last chance for the driver to abort sleep. (The patch also breaks my current wakelock implementation since I use a suspend_late hook to abort sleep, but this should be easy to fix)

Quoted 2 lines

Hm, if that solves the problem then it would be nice to have a new IRQF_NO_SUSPEND flag for it, in addition to IRQF_TIMER:

I think the right fix is for any interrupt that has IRQ_WAKEUP set to abort suspend if it is pending. I don't know if anyone relies on these interrupts being dropped now though.

Signature · 1 line
Arve Hjønnevåg
#60 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 6 lines · Arve Hjønnevåg

That would not work without wakelocks support, since the interrupt could occur after suspend_late which is the last chance for the driver to abort sleep. (The patch also breaks my current wakelock implementation since I use a suspend_late hook to abort sleep, but this should be easy to fix)

Since this must be some very deep arch-specific thing anyway, just make the dang thing be a "sysdev". At that point, its "suspend" function gets called way later (at which point CPU interrupts are off).

Quoted 6 lines
Quoted 2 lines

Hm, if that solves the problem then it would be nice to have a new IRQF_NO_SUSPEND flag for it, in addition to IRQF_TIMER:

I think the right fix is for any interrupt that has IRQ_WAKEUP set to abort suspend if it is pending. I don't know if anyone relies on these interrupts being dropped now though.

We could add something like that, but quite frankly, I'd hate to unless there is some seriously common case. If it's just an oddball hacky special case, it's easier to just say "hey, you have that crazy system device, you handle it yourself".

Linus

#61 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 10 lines · Linus Torvalds
Quoted 6 lines · Arve Hjønnevåg

That would not work without wakelocks support, since the interrupt could occur after suspend_late which is the last chance for the driver to abort sleep. (The patch also breaks my current wakelock implementation since I use a suspend_late hook to abort sleep, but this should be easy to fix)

Quoted 3 lines

Since this must be some very deep arch-specific thing anyway, just make the dang thing be a "sysdev". At that point, its "suspend" function gets called way later (at which point CPU interrupts are off).

Wakelocks can use a sysdev, but I don't think a keyboard driver should be a sysdev.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Hm, if that solves the problem then it would be nice to have a new IRQF_NO_SUSPEND flag for it, in addition to IRQF_TIMER:

Quoted 8 lines
Quoted 3 lines

I think the right fix is for any interrupt that has IRQ_WAKEUP set to abort suspend if it is pending. I don't know if anyone relies on these interrupts being dropped now though.

We could add something like that, but quite frankly, I'd hate to unless there is some seriously common case. If it's just an oddball hacky special case, it's easier to just say "hey, you have that crazy system device, you handle it yourself".

I don't think this is a oddball case. It is very common to connect keys or keypads to gpios. If these keys are wakeup keys, it is not OK to loose interrupts during the suspend phase.

Signature · 1 line
Arve Hjønnevåg
#62 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 6 lines · Arve Hjønnevåg

That would not work without wakelocks support, since the interrupt could occur after suspend_late which is the last chance for the driver to abort sleep. (The patch also breaks my current wakelock implementation since I use a suspend_late hook to abort sleep, but this should be easy to fix)

Quoted 3 lines

Since this must be some very deep arch-specific thing anyway, just make the dang thing be a "sysdev". At that point, its "suspend" function gets called way later (at which point CPU interrupts are off).

Wakelocks can use a sysdev, but I don't think a keyboard driver should be a sysdev.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Hm, if that solves the problem then it would be nice to have a new IRQF_NO_SUSPEND flag for it, in addition to IRQF_TIMER:

Quoted 8 lines
Quoted 3 lines

I think the right fix is for any interrupt that has IRQ_WAKEUP set to abort suspend if it is pending. I don't know if anyone relies on these interrupts being dropped now though.

We could add something like that, but quite frankly, I'd hate to unless there is some seriously common case. If it's just an oddball hacky special case, it's easier to just say "hey, you have that crazy system device, you handle it yourself".

I don't think this is a oddball case. It is very common to connect keys or keypads to gpios. If these keys are wakeup keys, it is not OK to loose interrupts during the suspend phase.

Signature · 1 line
Arve Hjønnevåg
#63 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 4 lines · Arve Hjønnevåg

I don't think this is a oddball case. It is very common to connect keys or keypads to gpios. If these keys are wakeup keys, it is not OK to loose interrupts during the suspend phase.

.. and how many drivers is that? Is it one or two "gpio input drivers" or is it a hundred?

The "common" is not so much about "how many machines", but "in how many drivers would you actually do this".

Linus

#64 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Arve Hjønnevåg
Quoted 4 lines · Arve Hjønnevåg

I don't think this is a oddball case. It is very common to connect keys or keypads to gpios. If these keys are wakeup keys, it is not OK to loose interrupts during the suspend phase.

.. and how many drivers is that? Is it one or two "gpio input drivers" or is it a hundred?

The "common" is not so much about "how many machines", but "in how many drivers would you actually do this".

Linus

#65 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Arve Hjønnevåg

I don't think this is a oddball case. It is very common to connect keys or keypads to gpios. If these keys are wakeup keys, it is not OK to loose interrupts during the suspend phase.

Quoted 5 lines

.. and how many drivers is that? Is it one or two "gpio input drivers" or is it a hundred?

The "common" is not so much about "how many machines", but "in how many drivers would you actually do this".

We only have one gpio input driver, but I don't think is good to loose any wakeup interrupts. Any driver that needs an edge triggered wakeup interrupt will have problems if the hardware does not regenerate the interrupt when the host does not respond.

It is not hard to work around this problem in the platform specific interrupt code, but I think it is a generic problem worth fixing for every platform.

Signature · 1 line
Arve Hjønnevåg
#66 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Arve Hjønnevåg

I don't think this is a oddball case. It is very common to connect keys or keypads to gpios. If these keys are wakeup keys, it is not OK to loose interrupts during the suspend phase.

Quoted 5 lines

.. and how many drivers is that? Is it one or two "gpio input drivers" or is it a hundred?

The "common" is not so much about "how many machines", but "in how many drivers would you actually do this".

We only have one gpio input driver, but I don't think is good to loose any wakeup interrupts. Any driver that needs an edge triggered wakeup interrupt will have problems if the hardware does not regenerate the interrupt when the host does not respond.

It is not hard to work around this problem in the platform specific interrupt code, but I think it is a generic problem worth fixing for every platform.

Signature · 1 line
Arve Hjønnevåg
#67 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 4 lines · Arve Hjønnevåg

We only have one gpio input driver, but I don't think is good to loose any wakeup interrupts. Any driver that needs an edge triggered wakeup interrupt will have problems if the hardware does not regenerate the interrupt when the host does not respond.

We are not loosing interrupts. The normal implementation of disable is a software disable and sets IRQ_PENDING to ensure we don't loose interrupts when the interrupt is disabled.

Eric

#68 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 10 lines · Eric W. Biederman
Quoted 4 lines · Arve Hjønnevåg

We only have one gpio input driver, but I don't think is good to loose any wakeup interrupts. Any driver that needs an edge triggered wakeup interrupt will have problems if the hardware does not regenerate the interrupt when the host does not respond.

We are not loosing interrupts.  The normal implementation of disable is a software disable and sets IRQ_PENDING to ensure we don't loose interrupts when the interrupt is disabled.

We loose the wakeup, but yes, the interrupt will be delivered if the system wakes up for any other reason.

Signature · 1 line
Arve Hjønnevåg
#69 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Eric W. Biederman
Quoted 10 lines · Eric W. Biederman
Quoted 4 lines · Arve Hjønnevåg

We only have one gpio input driver, but I don't think is good to loose any wakeup interrupts. Any driver that needs an edge triggered wakeup interrupt will have problems if the hardware does not regenerate the interrupt when the host does not respond.

We are not loosing interrupts.  The normal implementation of disable is a software disable and sets IRQ_PENDING to ensure we don't loose interrupts when the interrupt is disabled.

We loose the wakeup, but yes, the interrupt will be delivered if the system wakes up for any other reason.

Signature · 1 line
Arve Hjønnevåg
#70 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Arve Hjønnevåg
Quoted 4 lines · Arve Hjønnevåg

We only have one gpio input driver, but I don't think is good to loose any wakeup interrupts. Any driver that needs an edge triggered wakeup interrupt will have problems if the hardware does not regenerate the interrupt when the host does not respond.

We are not loosing interrupts. The normal implementation of disable is a software disable and sets IRQ_PENDING to ensure we don't loose interrupts when the interrupt is disabled.

Eric
_______________________________________________
linux-pm mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/linux-pm

#71 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Arve Hjønnevåg
Quoted 11 lines · Arve Hjønnevåg
Quoted 9 lines · Rafael J. Wysocki

BTW, appended is the current (3rd) version of the $subject patch with some of your comments taken into account. In particular, I did the following:
- moved [suspend|resume]_device_irqs() to a separate file (pm.c)
- fixed interrupt.h so that their headers are at a better place
- made enable_irq() clear IRQ_SUSPENDED
- made device_power_down() and device_power_up() call suspend_device_irqs() and resume_device_irqs(), respectively, which simplified the callers quite a bit (it changed the Xen code ordering, though, but I _think_ it still should work).

Quoted 3 lines

Do you plan to fix edge triggered wakeup interrupts? It still looks like edge triggered wakeup interrupts that occur between suspend_device_irqs and local_irq_disable will not cause a wakeup.

In the current version of the patch the interrupts that have IRQ_WAKEUP set in status are not disabled. Is this not enough?

Rafael

#72 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Arve Hjønnevåg
Quoted 11 lines · Arve Hjønnevåg
Quoted 9 lines · Rafael J. Wysocki

BTW, appended is the current (3rd) version of the $subject patch with some of your comments taken into account. In particular, I did the following:
- moved [suspend|resume]_device_irqs() to a separate file (pm.c)
- fixed interrupt.h so that their headers are at a better place
- made enable_irq() clear IRQ_SUSPENDED
- made device_power_down() and device_power_up() call suspend_device_irqs() and resume_device_irqs(), respectively, which simplified the callers quite a bit (it changed the Xen code ordering, though, but I _think_ it still should work).

Quoted 3 lines

Do you plan to fix edge triggered wakeup interrupts? It still looks like edge triggered wakeup interrupts that occur between suspend_device_irqs and local_irq_disable will not cause a wakeup.

In the current version of the patch the interrupts that have IRQ_WAKEUP set in status are not disabled. Is this not enough?

Rafael

#73 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 12 lines · Rafael J. Wysocki
Quoted 11 lines · Arve Hjønnevåg
Quoted 9 lines · Rafael J. Wysocki

BTW, appended is the current (3rd) version of the $subject patch with some of your comments taken into account.  In particular, I did the following:
- moved [suspend|resume]_device_irqs() to a separate file (pm.c)
- fixed interrupt.h so that their headers are at a better place
- made enable_irq() clear IRQ_SUSPENDED
- made device_power_down() and device_power_up() call suspend_device_irqs() and resume_device_irqs(), respectively, which simplified the callers quite a bit (it changed the Xen code ordering, though, but I _think_ it still should work).

Quoted 6 lines
Quoted 3 lines

Do you plan to fix edge triggered wakeup interrupts? It still looks like edge triggered wakeup interrupts that occur between suspend_device_irqs and local_irq_disable will not cause a wakeup.

In the current version of the patch the interrupts that have IRQ_WAKEUP set in status are not disabled.  Is this not enough?

That is enough for drivers that use wakelocks to abort suspend (if I fix the wakelock code to not use a platform device as its last abort point). It is not enough if you don't have wakelocks, since the interrupt can occur after suspend_late has been called and the driver has no way to abort suspend.

Signature · 1 line
Arve Hjønnevåg
#74 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 12 lines · Rafael J. Wysocki
Quoted 11 lines · Arve Hjønnevåg
Quoted 9 lines · Rafael J. Wysocki

BTW, appended is the current (3rd) version of the $subject patch with some of your comments taken into account.  In particular, I did the following:
- moved [suspend|resume]_device_irqs() to a separate file (pm.c)
- fixed interrupt.h so that their headers are at a better place
- made enable_irq() clear IRQ_SUSPENDED
- made device_power_down() and device_power_up() call suspend_device_irqs() and resume_device_irqs(), respectively, which simplified the callers quite a bit (it changed the Xen code ordering, though, but I _think_ it still should work).

Quoted 6 lines
Quoted 3 lines

Do you plan to fix edge triggered wakeup interrupts? It still looks like edge triggered wakeup interrupts that occur between suspend_device_irqs and local_irq_disable will not cause a wakeup.

In the current version of the patch the interrupts that have IRQ_WAKEUP set in status are not disabled.  Is this not enough?

That is enough for drivers that use wakelocks to abort suspend (if I fix the wakelock code to not use a platform device as its last abort point). It is not enough if you don't have wakelocks, since the interrupt can occur after suspend_late has been called and the driver has no way to abort suspend.

Signature · 1 line
Arve Hjønnevåg
#75 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 6 lines · Arve Hjønnevåg

That is enough for drivers that use wakelocks to abort suspend (if I fix the wakelock code to not use a platform device as its last abort point). It is not enough if you don't have wakelocks, since the interrupt can occur after suspend_late has been called and the driver has no way to abort suspend.

I still don't quite see how you deal with the race anyway. Ie. Even without Rafael patch, what if the interrupt occurs after your sysdev suspend ?

In general, unless they are level sensitive, wakeup interrupts tend to always be somewhat racy.

Cheers,
Ben.

_______________________________________________
linux-pm mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/linux-pm

#76 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 10 lines · Benjamin Herrenschmidt
Quoted 6 lines · Arve Hjønnevåg

That is enough for drivers that use wakelocks to abort suspend (if I fix the wakelock code to not use a platform device as its last abort point). It is not enough if you don't have wakelocks, since the interrupt can occur after suspend_late has been called and the driver has no way to abort suspend.

I still don't quite see how you deal with the race anyway. Ie. Even without Rafael patch, what if the interrupt occurs after your sysdev suspend ?

After local_irq_disable has been called, the interrupt will no longer be cleared by Linux when it occurs. This means that is still pending when you get to the low level suspend code which will prevent suspend.

Quoted 2 lines

In general, unless they are level sensitive, wakeup interrupts tend to always be somewhat racy.

They don't have to be. If you have a separate hardware component that tracks wakeup interrupts, you need to start this before you stop the main interrupt controller. If any interrupts are pending at this time you abort suspend. After a wakeup you do the reverse.

Signature · 1 line
Arve Hjønnevåg
#77 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Benjamin Herrenschmidt
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 6 lines · Arve Hjønnevåg

That is enough for drivers that use wakelocks to abort suspend (if I fix the wakelock code to not use a platform device as its last abort point). It is not enough if you don't have wakelocks, since the interrupt can occur after suspend_late has been called and the driver has no way to abort suspend.

I still don't quite see how you deal with the race anyway. Ie. Even without Rafael patch, what if the interrupt occurs after your sysdev suspend ?

After local_irq_disable has been called, the interrupt will no longer be cleared by Linux when it occurs. This means that is still pending when you get to the low level suspend code which will prevent suspend.

Quoted 2 lines

In general, unless they are level sensitive, wakeup interrupts tend to always be somewhat racy.

They don't have to be. If you have a separate hardware component that tracks wakeup interrupts, you need to start this before you stop the main interrupt controller. If any interrupts are pending at this time you abort suspend. After a wakeup you do the reverse.

Signature · 1 line
Arve Hjønnevåg
#78 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 9 lines · Arve Hjønnevåg
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 6 lines · Arve Hjønnevåg

That is enough for drivers that use wakelocks to abort suspend (if I fix the wakelock code to not use a platform device as its last abort point). It is not enough if you don't have wakelocks, since the interrupt can occur after suspend_late has been called and the driver has no way to abort suspend.

Quoted 7 lines
Quoted 3 lines

I still don't quite see how you deal with the race anyway. Ie. Even without Rafael patch, what if the interrupt occurs after your sysdev suspend ?

After local_irq_disable has been called, the interrupt will no longer be cleared by Linux when it occurs. This means that is still pending when you get to the low level suspend code which will prevent suspend.

Ok so you want this interrupt to stay pending at the PIC level ? So just marking it so the kernel doesn't disable it should do the trick.

Quoted 7 lines
Quoted 2 lines

In general, unless they are level sensitive, wakeup interrupts tend to always be somewhat racy.

They don't have to be. If you have a separate hardware component that tracks wakeup interrupts, you need to start this before you stop the main interrupt controller. If any interrupts are pending at this time you abort suspend. After a wakeup you do the reverse.

Right but then you can start this earlier and there is no problem. But if you do want the interrupt to remaining pending in the PIC, then you probably need to set that magic flag so we don't disable it, that should do the trick just fine no ?

It's hard to tell without more detailed HW specs of course...

Ben.

_______________________________________________
linux-pm mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/linux-pm

#79 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Arve Hjønnevåg
Quoted 9 lines · Arve Hjønnevåg
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 6 lines · Arve Hjønnevåg

That is enough for drivers that use wakelocks to abort suspend (if I fix the wakelock code to not use a platform device as its last abort point). It is not enough if you don't have wakelocks, since the interrupt can occur after suspend_late has been called and the driver has no way to abort suspend.

Quoted 7 lines
Quoted 3 lines

I still don't quite see how you deal with the race anyway. Ie. Even without Rafael patch, what if the interrupt occurs after your sysdev suspend ?

After local_irq_disable has been called, the interrupt will no longer be cleared by Linux when it occurs. This means that is still pending when you get to the low level suspend code which will prevent suspend.

Ok so you want this interrupt to stay pending at the PIC level ? So just marking it so the kernel doesn't disable it should do the trick.

Quoted 7 lines
Quoted 2 lines

In general, unless they are level sensitive, wakeup interrupts tend to always be somewhat racy.

They don't have to be. If you have a separate hardware component that tracks wakeup interrupts, you need to start this before you stop the main interrupt controller. If any interrupts are pending at this time you abort suspend. After a wakeup you do the reverse.

Right but then you can start this earlier and there is no problem. But if you do want the interrupt to remaining pending in the PIC, then you probably need to set that magic flag so we don't disable it, that should do the trick just fine no ?

It's hard to tell without more detailed HW specs of course...

Ben.

#80 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Arve Hjønnevåg
Quoted 17 lines · Arve Hjønnevåg
Quoted 16 lines · Rafael J. Wysocki
Quoted 14 lines · Arve Hjønnevåg
Quoted 9 lines · Rafael J. Wysocki

BTW, appended is the current (3rd) version of the $subject patch with some of your comments taken into account. In particular, I did the following:
- moved [suspend|resume]_device_irqs() to a separate file (pm.c)
- fixed interrupt.h so that their headers are at a better place
- made enable_irq() clear IRQ_SUSPENDED
- made device_power_down() and device_power_up() call suspend_device_irqs() and resume_device_irqs(), respectively, which simplified the callers quite a bit (it changed the Xen code ordering, though, but I _think_ it still should work).

Do you plan to fix edge triggered wakeup interrupts? It still looks like edge triggered wakeup interrupts that occur between suspend_device_irqs and local_irq_disable will not cause a wakeup.

Quoted 8 lines
Quoted 2 lines

In the current version of the patch the interrupts that have IRQ_WAKEUP set in status are not disabled. Is this not enough?

That is enough for drivers that use wakelocks to abort suspend (if I fix the wakelock code to not use a platform device as its last abort point). It is not enough if you don't have wakelocks, since the interrupt can occur after suspend_late has been called and the driver has no way to abort suspend.

Well, how exactly the $subject patch does cause this problem to happen?

Rafael

#81 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Arve Hjønnevåg
Quoted 17 lines · Arve Hjønnevåg
Quoted 16 lines · Rafael J. Wysocki
Quoted 14 lines · Arve Hjønnevåg
Quoted 9 lines · Rafael J. Wysocki

BTW, appended is the current (3rd) version of the $subject patch with some of your comments taken into account. In particular, I did the following:
- moved [suspend|resume]_device_irqs() to a separate file (pm.c)
- fixed interrupt.h so that their headers are at a better place
- made enable_irq() clear IRQ_SUSPENDED
- made device_power_down() and device_power_up() call suspend_device_irqs() and resume_device_irqs(), respectively, which simplified the callers quite a bit (it changed the Xen code ordering, though, but I _think_ it still should work).

Do you plan to fix edge triggered wakeup interrupts? It still looks like edge triggered wakeup interrupts that occur between suspend_device_irqs and local_irq_disable will not cause a wakeup.

Quoted 8 lines
Quoted 2 lines

In the current version of the patch the interrupts that have IRQ_WAKEUP set in status are not disabled. Is this not enough?

That is enough for drivers that use wakelocks to abort suspend (if I fix the wakelock code to not use a platform device as its last abort point). It is not enough if you don't have wakelocks, since the interrupt can occur after suspend_late has been called and the driver has no way to abort suspend.

Well, how exactly the $subject patch does cause this problem to happen?

Rafael

#82 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 2 lines · Rafael J. Wysocki

Well, how exactly the $subject patch does cause this problem to happen?

Rafael, the problem is that if an interrupt happens while it's disabled - but before the CPU has actually turned all interrupts off - the CPU will ACK the interrupt (but just set a flag for it being PENDING), so now the chipset logic around it will not see it as pending any more, so now the chipset won't auto-wake the CPU immediately (or more likely, it won't even suspend it).

It's trivial to fix multiple ways, so I wouldn't worry. The most trivial way is to just have some sysdev drievr code simply do something like

static int sysdev_suspend()

  {
	for_each_irq(irq,desc) {
		if (!(desc->flags & IRQF_WAKE))
			continue;
		if (desc->flags & IRQ_PENDING)
			return -EBUSY;
	}
	return 0;

}

and that should automatically mean that if any irq is pending, the suspend will fail and we'll immediately wake up again.

It looks trivial, and I don't understand why Arve can't just do the sysdev thing.

Linus

#83 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 2 lines · Rafael J. Wysocki

Well, how exactly the $subject patch does cause this problem to happen?

Rafael, the problem is that if an interrupt happens while it's disabled - but before the CPU has actually turned all interrupts off - the CPU will ACK the interrupt (but just set a flag for it being PENDING), so now the chipset logic around it will not see it as pending any more, so now the chipset won't auto-wake the CPU immediately (or more likely, it won't even suspend it).

It's trivial to fix multiple ways, so I wouldn't worry. The most trivial way is to just have some sysdev drievr code simply do something like

static int sysdev_suspend()

  {
	for_each_irq(irq,desc) {
		if (!(desc->flags & IRQF_WAKE))
			continue;
		if (desc->flags & IRQ_PENDING)
			return -EBUSY;
	}
	return 0;

}

and that should automatically mean that if any irq is pending, the suspend will fail and we'll immediately wake up again.

It looks trivial, and I don't understand why Arve can't just do the sysdev thing.

Linus

#84 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 27 lines · Linus Torvalds
Quoted 2 lines · Rafael J. Wysocki

Well, how exactly the $subject patch does cause this problem to happen?

Rafael, the problem is that if an interrupt happens while it's disabled - but before the CPU has actually turned all interrupts off - the CPU will ACK the interrupt (but just set a flag for it being PENDING), so now the chipset logic around it will not see it as pending any more, so now the chipset won't auto-wake the CPU immediately (or more likely, it won't even suspend it).

It's trivial to fix multiple ways, so I wouldn't worry. The most trivial way is to just have some sysdev drievr code simply do something like

 static int sysdev_suspend()
 {
       for_each_irq(irq,desc) {
               if (!(desc->flags & IRQF_WAKE))
                       continue;
               if (desc->flags & IRQ_PENDING)
                       return -EBUSY;
       }
       return 0;
 }

Quoted 5 lines

and that should automatically mean that if any irq is pending, the suspend will fail and we'll immediately wake up again.

It looks trivial, and I don't understand why Arve can't just do the sysdev thing.

I can. My point is that the patch breaks our existing code. If anyone else uses edge triggered wakeup interrupt it may break from them as well. The main question if this should be fixed separately for every platform that needs it, or if pending wakeup interrupts should always abort sleep.

Signature · 1 line
Arve Hjønnevåg
#85 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 27 lines · Linus Torvalds
Quoted 2 lines · Rafael J. Wysocki

Well, how exactly the $subject patch does cause this problem to happen?

Rafael, the problem is that if an interrupt happens while it's disabled - but before the CPU has actually turned all interrupts off - the CPU will ACK the interrupt (but just set a flag for it being PENDING), so now the chipset logic around it will not see it as pending any more, so now the chipset won't auto-wake the CPU immediately (or more likely, it won't even suspend it).

It's trivial to fix multiple ways, so I wouldn't worry. The most trivial way is to just have some sysdev drievr code simply do something like

 static int sysdev_suspend()
 {
       for_each_irq(irq,desc) {
               if (!(desc->flags & IRQF_WAKE))
                       continue;
               if (desc->flags & IRQ_PENDING)
                       return -EBUSY;
       }
       return 0;
 }

Quoted 5 lines

and that should automatically mean that if any irq is pending, the suspend will fail and we'll immediately wake up again.

It looks trivial, and I don't understand why Arve can't just do the sysdev thing.

I can. My point is that the patch breaks our existing code. If anyone else uses edge triggered wakeup interrupt it may break from them as well. The main question if this should be fixed separately for every platform that needs it, or if pending wakeup interrupts should always abort sleep.

Signature · 1 line
Arve Hjønnevåg
#86 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 29 lines · Arve Hjønnevåg
Quoted 27 lines · Linus Torvalds
Quoted 2 lines · Rafael J. Wysocki

Well, how exactly the $subject patch does cause this problem to happen?

Rafael, the problem is that if an interrupt happens while it's disabled - but before the CPU has actually turned all interrupts off - the CPU will ACK the interrupt (but just set a flag for it being PENDING), so now the chipset logic around it will not see it as pending any more, so now the chipset won't auto-wake the CPU immediately (or more likely, it won't even suspend it).

It's trivial to fix multiple ways, so I wouldn't worry. The most trivial way is to just have some sysdev drievr code simply do something like

static int sysdev_suspend()

 {
       for_each_irq(irq,desc) {
               if (!(desc->flags & IRQF_WAKE))
                       continue;
               if (desc->flags & IRQ_PENDING)
                       return -EBUSY;
       }
       return 0;

}

Quoted 7 lines
Quoted 5 lines

and that should automatically mean that if any irq is pending, the suspend will fail and we'll immediately wake up again.

It looks trivial, and I don't understand why Arve can't just do the sysdev thing.

I can. My point is that the patch breaks our existing code.

Is that a mainline kernel code?

Quoted 4 lines

If anyone else uses edge triggered wakeup interrupt it may break from them as well. The main question if this should be fixed separately for every platform that needs it, or if pending wakeup interrupts should always abort sleep.

Well, I'm not really sure if this is the problem. In fact the problem is that you have a regular device the interrupt of which can be a wake-up one. I think the problem wouldn't have existed at all if it had been a sysdev. Is that correct?

Rafael

#87 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Arve Hjønnevåg
Quoted 29 lines · Arve Hjønnevåg
Quoted 27 lines · Linus Torvalds
Quoted 2 lines · Rafael J. Wysocki

Well, how exactly the $subject patch does cause this problem to happen?

Rafael, the problem is that if an interrupt happens while it's disabled - but before the CPU has actually turned all interrupts off - the CPU will ACK the interrupt (but just set a flag for it being PENDING), so now the chipset logic around it will not see it as pending any more, so now the chipset won't auto-wake the CPU immediately (or more likely, it won't even suspend it).

It's trivial to fix multiple ways, so I wouldn't worry. The most trivial way is to just have some sysdev drievr code simply do something like

static int sysdev_suspend()

 {
       for_each_irq(irq,desc) {
               if (!(desc->flags & IRQF_WAKE))
                       continue;
               if (desc->flags & IRQ_PENDING)
                       return -EBUSY;
       }
       return 0;

}

Quoted 7 lines
Quoted 5 lines

and that should automatically mean that if any irq is pending, the suspend will fail and we'll immediately wake up again.

It looks trivial, and I don't understand why Arve can't just do the sysdev thing.

I can. My point is that the patch breaks our existing code.

Is that a mainline kernel code?

Quoted 4 lines

If anyone else uses edge triggered wakeup interrupt it may break from them as well. The main question if this should be fixed separately for every platform that needs it, or if pending wakeup interrupts should always abort sleep.

Well, I'm not really sure if this is the problem. In fact the problem is that you have a regular device the interrupt of which can be a wake-up one. I think the problem wouldn't have existed at all if it had been a sysdev. Is that correct?

Rafael

#88 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 33 lines · Rafael J. Wysocki
Quoted 32 lines · Arve Hjønnevåg
Quoted 30 lines · Linus Torvalds
Quoted 2 lines · Rafael J. Wysocki

Well, how exactly the $subject patch does cause this problem to happen?

Rafael, the problem is that if an interrupt happens while it's disabled - but before the CPU has actually turned all interrupts off - the CPU will ACK the interrupt (but just set a flag for it being PENDING), so now the chipset logic around it will not see it as pending any more, so now the chipset won't auto-wake the CPU immediately (or more likely, it won't even suspend it).

It's trivial to fix multiple ways, so I wouldn't worry. The most trivial way is to just have some sysdev drievr code simply do something like

 static int sysdev_suspend()
 {
       for_each_irq(irq,desc) {
               if (!(desc->flags & IRQF_WAKE))
                       continue;
               if (desc->flags & IRQ_PENDING)
                       return -EBUSY;
       }
       return 0;
 }

and that should automatically mean that if any irq is pending, the suspend will fail and we'll immediately wake up again.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

It looks trivial, and I don't understand why Arve can't just do the sysdev thing.

I can. My point is that the patch breaks our existing code.

Is that a mainline kernel code?

No, the msm suspend support has not been merged.

Quoted 8 lines
Quoted 5 lines

If anyone else uses edge triggered wakeup interrupt it may break from them as well. The main question if this should be fixed separately for every platform that needs it, or if pending wakeup interrupts should always abort sleep.

Well, I'm not really sure if this is the problem.  In fact the problem is that you have a regular device the interrupt of which can be a wake-up one.  I think

Is that not a common case and what enable_irq_wake is for?

Quoted 2 lines

the problem wouldn't have existed at all if it had been a sysdev.  Is that correct?

How many sysdevs use interrupts?

I found may drivers in the mainline kernel that use enable_irq_wake, but I did not see any that handle this race condition.

Signature · 1 line
Arve Hjønnevåg
#89 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 33 lines · Rafael J. Wysocki
Quoted 32 lines · Arve Hjønnevåg
Quoted 30 lines · Linus Torvalds
Quoted 2 lines · Rafael J. Wysocki

Well, how exactly the $subject patch does cause this problem to happen?

Rafael, the problem is that if an interrupt happens while it's disabled - but before the CPU has actually turned all interrupts off - the CPU will ACK the interrupt (but just set a flag for it being PENDING), so now the chipset logic around it will not see it as pending any more, so now the chipset won't auto-wake the CPU immediately (or more likely, it won't even suspend it).

It's trivial to fix multiple ways, so I wouldn't worry. The most trivial way is to just have some sysdev drievr code simply do something like

 static int sysdev_suspend()
 {
       for_each_irq(irq,desc) {
               if (!(desc->flags & IRQF_WAKE))
                       continue;
               if (desc->flags & IRQ_PENDING)
                       return -EBUSY;
       }
       return 0;
 }

and that should automatically mean that if any irq is pending, the suspend will fail and we'll immediately wake up again.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

It looks trivial, and I don't understand why Arve can't just do the sysdev thing.

I can. My point is that the patch breaks our existing code.

Is that a mainline kernel code?

No, the msm suspend support has not been merged.

Quoted 8 lines
Quoted 5 lines

If anyone else uses edge triggered wakeup interrupt it may break from them as well. The main question if this should be fixed separately for every platform that needs it, or if pending wakeup interrupts should always abort sleep.

Well, I'm not really sure if this is the problem.  In fact the problem is that you have a regular device the interrupt of which can be a wake-up one.  I think

Is that not a common case and what enable_irq_wake is for?

Quoted 2 lines

the problem wouldn't have existed at all if it had been a sysdev.  Is that correct?

How many sysdevs use interrupts?

I found may drivers in the mainline kernel that use enable_irq_wake, but I did not see any that handle this race condition.

Signature · 1 line
Arve Hjønnevåg
#90 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 5 lines · Arve Hjønnevåg

How many sysdevs use interrupts?

I found may drivers in the mainline kernel that use enable_irq_wake, but I did not see any that handle this race condition.

The _only_ driver that does enable_irq_wake() on x86 is the cmos timer driver, and even there it actually doesn't use irq_wake, but ACPI. Why? Because I don't think irq wakeup even _works_ on x86.

So the whole enable_irq_wake is largely some embedded ARM platform issue, and a very special case, and doesn't exist anywhere else.

Maybe I'm missing something, but it's definitely not the normal case.

Linus

#91 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Arve Hjønnevåg
Quoted 5 lines · Arve Hjønnevåg

How many sysdevs use interrupts?

I found may drivers in the mainline kernel that use enable_irq_wake, but I did not see any that handle this race condition.

The _only_ driver that does enable_irq_wake() on x86 is the cmos timer driver, and even there it actually doesn't use irq_wake, but ACPI. Why? Because I don't think irq wakeup even _works_ on x86.

So the whole enable_irq_wake is largely some embedded ARM platform issue, and a very special case, and doesn't exist anywhere else.

Maybe I'm missing something, but it's definitely not the normal case.

Linus

#92 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 8 lines · Linus Torvalds

The _only_ driver that does enable_irq_wake() on x86 is the cmos timer driver, and even there it actually doesn't use irq_wake, but ACPI. Why? Because I don't think irq wakeup even _works_ on x86.

So the whole enable_irq_wake is largely some embedded ARM platform issue, and a very special case, and doesn't exist anywhere else.

Maybe I'm missing something, but it's definitely not the normal case.

What you're missing is that the embedded world is quite a large one. As any member of CELF will tell you, there are lots more embedded systems around than there are desktop/laptop computers. (I admit, I don't know what the ratio is if you restrict your attention to systems running Linux.) We can't afford to regard them as second-class citizens.

Plenty of embedded systems use normal interrupts from GPIO lines as wakeup sources. Don't discount the need for this just because desktop systems don't use them that way. It may not be "normal" in the circles you're accustomed to, but it _is_ normal elsewhere.

Alan Stern

#93 lore raw

Re: [linux-pm] [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds

The _only_ driver that does enable_irq_wake() on x86 is the cmos timer driver, and even there it actually doesn't use irq_wake, but ACPI. Why? Because I don't think irq wakeup even _works_ on x86.

So the whole enable_irq_wake is largely some embedded ARM platform issue, and a very special case, and doesn't exist anywhere else.

Maybe I'm missing something, but it's definitely not the normal case.

What you're missing is that the embedded world is quite a large one. As any member of CELF will tell you, there are lots more embedded systems around than there are desktop/laptop computers. (I admit, I don't know what the ratio is if you restrict your attention to systems running Linux.) We can't afford to regard them as second-class citizens.

Plenty of embedded systems use normal interrupts from GPIO lines as wakeup sources. Don't discount the need for this just because desktop systems don't use them that way. It may not be "normal" in the circles you're accustomed to, but it _is_ normal elsewhere.

Alan Stern

#94 lore raw

Re: [linux-pm] [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 2 lines · Alan Stern

What you're missing is that the embedded world is quite a large one.

I'm gpoing to give you one more clue, and if you don't stop sending out these IDIOTIC emails, I'm going to put you into my killfile.

Got it?

So listen up:
- the number of ARM chips sold doesn't matter one F*CKING WHIT.
- You need to add ONE SINGLE "sysdev" entry for ARM to take care of this FOR EVERY DAMN SINGLE ONE.
- Your inane whining about this AFTER I HAVE TOLD YOU MULTIPLE TIMES HOW TO DO IT, AND AFTER I HAVE TOLD YOU THAT IT'S A SPECIAL CASE, IS F*CKING IRRITATING.

Got it?

I _grepped_ for that enable_irq_wake() use. It looks like it's only used on ARM and maybe BF. Add the five lines of code (just cut and paste them from my earlier email) to your architecture already, AND STOP WHINING.

It's not a generic case. It's not a problem. You can damn well fix it in the ONE SINGLE ARCHITECTURE (or maybe two) that cares. I've told you how.

Why is it so damn hard for you to just accept?

Linus

#95 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Alan Stern

What you're missing is that the embedded world is quite a large one.

I'm gpoing to give you one more clue, and if you don't stop sending out these IDIOTIC emails, I'm going to put you into my killfile.

Got it?

Whoa!! Hold on there! You got too angry too quickly. I'm Alan Stern, not Arve Hjønnevåg; that was the first email I've sent on this topic.

And while perhaps it was idiotic, you shouldn't put the blame for it on Arve.

Quoted 8 lines

So listen up:
- the number of ARM chips sold doesn't matter one F*CKING WHIT.
- You need to add ONE SINGLE "sysdev" entry for ARM to take care of this FOR EVERY DAMN SINGLE ONE.
- Your inane whining about this AFTER I HAVE TOLD YOU MULTIPLE TIMES HOW TO DO IT, AND AFTER I HAVE TOLD YOU THAT IT'S A SPECIAL CASE, IS F*CKING IRRITATING.

Quoted 5 lines

Got it?

I _grepped_ for that enable_irq_wake() use. It looks like it's only used on ARM and maybe BF. Add the five lines of code (just cut and paste them from my earlier email) to your architecture already, AND STOP WHINING.

Really? Let's see (this is using Greg KH's development tree):

$ find . -name '*.[ch]' | xargs grep enable_irq_wake
./drivers/serial/serial_core.c:         enable_irq_wake(port->irq);
./drivers/usb/gadget/at91_udc.c:                enable_irq_wake(udc->udp_irq);
./drivers/usb/gadget/at91_udc.c:                enable_irq_wake(udc->board.vbus_pin);
./drivers/usb/musb/musb_core.c: if (enable_irq_wake(nIrq) == 0) {
./drivers/usb/host/ohci-at91.c:         enable_irq_wake(hcd->irq);
./drivers/input/serio/sa1111ps2.c:      enable_irq_wake(ps2if->dev->irq[0]);
./drivers/input/keyboard/gpio_keys.c:                           enable_irq_wake(irq);
./drivers/input/keyboard/pxa27x_keypad.c:               enable_irq_wake(keypad->irq);
./drivers/input/keyboard/bf54x-keys.c:          enable_irq_wake(bf54x_kpad->irq);
./drivers/pcmcia/at91_cf.c:             enable_irq_wake(board->det_pin);
./drivers/pcmcia/at91_cf.c:                     enable_irq_wake(board->irq_pin);
./drivers/mmc/host/at91_mci.c:          enable_irq_wake(host->board->det_pin);
./drivers/mfd/htc-egpio.c:              enable_irq_wake(ei->chained_irq);
./drivers/mfd/pcf50633-core.c:  if (enable_irq_wake(client->irq) < 0)
./drivers/rtc/rtc-sa1100.c:             enable_irq_wake(IRQ_RTCAlrm);
./drivers/rtc/rtc-omap.c:               enable_irq_wake(omap_rtc_alarm);
./drivers/rtc/rtc-s3c.c:                enable_irq_wake(s3c_rtc_alarmno);
./drivers/rtc/rtc-at91rm9200.c:                 enable_irq_wake(AT91_ID_SYS);
./drivers/rtc/rtc-cmos.c:                       enable_irq_wake(cmos->irq);
./drivers/rtc/rtc-bfin.c:               enable_irq_wake(IRQ_RTC);
./drivers/rtc/rtc-ds1374.c:             enable_irq_wake(client->irq);
./drivers/rtc/rtc-at91sam9.c:                   enable_irq_wake(AT91_ID_SYS);
./drivers/rtc/rtc-pxa.c:                enable_irq_wake(pxa_rtc->irq_Alrm);
./drivers/power/pda_power.c:                    ac_wakeup_enabled = !enable_irq_wake(ac_irq->start);
./drivers/power/pda_power.c:                    usb_wakeup_enabled = !enable_irq_wake(usb_irq->start);
./arch/arm/mach-sa1100/neponset.c:      enable_irq_wake(IRQ_GPIO25);
./arch/arm/mach-s3c2410/mach-amlm5900.c:                enable_irq_wake(IRQ_EINT9);
./arch/arm/mach-omap1/board-osk.c:                      enable_irq_wake(irq);
./arch/arm/mach-omap1/serial.c: enable_irq_wake(gpio_to_irq(gpio_nr));
./arch/arm/plat-omap/gpio.c:                    enable_irq_wake(bank->irq);
./arch/arm/plat-omap/gpio.c:                    enable_irq_wake(bank->irq);
./arch/arm/plat-omap/gpio.c:/* Use disable_irq_wake() and enable_irq_wake() functions from drivers */
./include/linux/interrupt.h:static inline int enable_irq_wake(unsigned int irq)
./include/linux/interrupt.h:static inline int enable_irq_wake(unsigned int irq)

Perhaps these aren't all the sort of usage you're talking about, but I bet most of them are. It certainly looks like more than just ARM. Maybe not all that much more, but definitely more. And the number will only grow in the future.

Quoted 2 lines

It's not a generic case. It's not a problem. You can damn well fix it in the ONE SINGLE ARCHITECTURE (or maybe two) that cares. I've told you how.

I'm not arguing with your suggestion; I'm merely disagreeing with your statement that wakeup interrupts are "definitely not the normal case".

Alan Stern

#96 lore raw

Re: [linux-pm] [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Alan Stern

What you're missing is that the embedded world is quite a large one.

I'm gpoing to give you one more clue, and if you don't stop sending out these IDIOTIC emails, I'm going to put you into my killfile.

Got it?

Whoa!! Hold on there! You got too angry too quickly. I'm Alan Stern, not Arve Hjønnevåg; that was the first email I've sent on this topic.

And while perhaps it was idiotic, you shouldn't put the blame for it on Arve.

Quoted 8 lines

So listen up:
- the number of ARM chips sold doesn't matter one F*CKING WHIT.
- You need to add ONE SINGLE "sysdev" entry for ARM to take care of this FOR EVERY DAMN SINGLE ONE.
- Your inane whining about this AFTER I HAVE TOLD YOU MULTIPLE TIMES HOW TO DO IT, AND AFTER I HAVE TOLD YOU THAT IT'S A SPECIAL CASE, IS F*CKING IRRITATING.

Quoted 5 lines

Got it?

I _grepped_ for that enable_irq_wake() use. It looks like it's only used on ARM and maybe BF. Add the five lines of code (just cut and paste them from my earlier email) to your architecture already, AND STOP WHINING.

Really? Let's see (this is using Greg KH's development tree):

$ find . -name '*.[ch]' | xargs grep enable_irq_wake
./drivers/serial/serial_core.c:         enable_irq_wake(port->irq);
./drivers/usb/gadget/at91_udc.c:                enable_irq_wake(udc->udp_irq);
./drivers/usb/gadget/at91_udc.c:                enable_irq_wake(udc->board.vbus_pin);
./drivers/usb/musb/musb_core.c: if (enable_irq_wake(nIrq) == 0) {
./drivers/usb/host/ohci-at91.c:         enable_irq_wake(hcd->irq);
./drivers/input/serio/sa1111ps2.c:      enable_irq_wake(ps2if->dev->irq[0]);
./drivers/input/keyboard/gpio_keys.c:                           enable_irq_wake(irq);
./drivers/input/keyboard/pxa27x_keypad.c:               enable_irq_wake(keypad->irq);
./drivers/input/keyboard/bf54x-keys.c:          enable_irq_wake(bf54x_kpad->irq);
./drivers/pcmcia/at91_cf.c:             enable_irq_wake(board->det_pin);
./drivers/pcmcia/at91_cf.c:                     enable_irq_wake(board->irq_pin);
./drivers/mmc/host/at91_mci.c:          enable_irq_wake(host->board->det_pin);
./drivers/mfd/htc-egpio.c:              enable_irq_wake(ei->chained_irq);
./drivers/mfd/pcf50633-core.c:  if (enable_irq_wake(client->irq) < 0)
./drivers/rtc/rtc-sa1100.c:             enable_irq_wake(IRQ_RTCAlrm);
./drivers/rtc/rtc-omap.c:               enable_irq_wake(omap_rtc_alarm);
./drivers/rtc/rtc-s3c.c:                enable_irq_wake(s3c_rtc_alarmno);
./drivers/rtc/rtc-at91rm9200.c:                 enable_irq_wake(AT91_ID_SYS);
./drivers/rtc/rtc-cmos.c:                       enable_irq_wake(cmos->irq);
./drivers/rtc/rtc-bfin.c:               enable_irq_wake(IRQ_RTC);
./drivers/rtc/rtc-ds1374.c:             enable_irq_wake(client->irq);
./drivers/rtc/rtc-at91sam9.c:                   enable_irq_wake(AT91_ID_SYS);
./drivers/rtc/rtc-pxa.c:                enable_irq_wake(pxa_rtc->irq_Alrm);
./drivers/power/pda_power.c:                    ac_wakeup_enabled = !enable_irq_wake(ac_irq->start);
./drivers/power/pda_power.c:                    usb_wakeup_enabled = !enable_irq_wake(usb_irq->start);
./arch/arm/mach-sa1100/neponset.c:      enable_irq_wake(IRQ_GPIO25);
./arch/arm/mach-s3c2410/mach-amlm5900.c:                enable_irq_wake(IRQ_EINT9);
./arch/arm/mach-omap1/board-osk.c:                      enable_irq_wake(irq);
./arch/arm/mach-omap1/serial.c: enable_irq_wake(gpio_to_irq(gpio_nr));
./arch/arm/plat-omap/gpio.c:                    enable_irq_wake(bank->irq);
./arch/arm/plat-omap/gpio.c:                    enable_irq_wake(bank->irq);
./arch/arm/plat-omap/gpio.c:/* Use disable_irq_wake() and enable_irq_wake() functions from drivers */
./include/linux/interrupt.h:static inline int enable_irq_wake(unsigned int irq)
./include/linux/interrupt.h:static inline int enable_irq_wake(unsigned int irq)

Perhaps these aren't all the sort of usage you're talking about, but I bet most of them are. It certainly looks like more than just ARM. Maybe not all that much more, but definitely more. And the number will only grow in the future.

Quoted 2 lines

It's not a generic case. It's not a problem. You can damn well fix it in the ONE SINGLE ARCHITECTURE (or maybe two) that cares. I've told you how.

I'm not arguing with your suggestion; I'm merely disagreeing with your statement that wakeup interrupts are "definitely not the normal case".

Alan Stern

#97 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 5 lines · Alan Stern

Perhaps these aren't all the sort of usage you're talking about, but I bet most of them are. It certainly looks like more than just ARM. Maybe not all that much more, but definitely more. And the number will only grow in the future.

Are you really sure? Because it can't be x86. I'm pretty sure that that is simply not how x86 wake events _work_ - they're not interrupts.

And that's the big point that people seem to be missing here: the whole "wake up interrupt" thing is not some generic model in the first place. I strongly suspect that it literally only works on certain architectures.

In other words, I'm getting damn tired of people who CLEARLY DON'T EVEN KNOW HOW THE HARDWARE WORKS arguing over this.

Here's another hint: that whole "enable_irq_wake()" - have you possibly spent even five seconds to look at what it actually does? I bet you haven't. Because what it does is to call the irq controller "set_wake" function.

Now, grep for that. Whay do you find? Like maybe ARM and BlackFin? Oh, and I note one MIPS platform.

The point is, that whole "irq wake" really is system dependent. We can do helper functions for it, but anybody who thinks it's anything "generic" is totally mistaken.

In other words, making it a sysdev thing is the CORRECT thing to do. It really is not just "here's how you work around something". It really is "this is how the hardware FUNDAMENTALLY WORKS".

Please, stop arguing. At least argue only after you understand what the physical hardware actyally does.

Linus

#98 lore raw

Re: [linux-pm] [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Alan Stern
Quoted 5 lines · Alan Stern

Perhaps these aren't all the sort of usage you're talking about, but I bet most of them are. It certainly looks like more than just ARM. Maybe not all that much more, but definitely more. And the number will only grow in the future.

Are you really sure? Because it can't be x86. I'm pretty sure that that is simply not how x86 wake events _work_ - they're not interrupts.

And that's the big point that people seem to be missing here: the whole "wake up interrupt" thing is not some generic model in the first place. I strongly suspect that it literally only works on certain architectures.

In other words, I'm getting damn tired of people who CLEARLY DON'T EVEN KNOW HOW THE HARDWARE WORKS arguing over this.

Here's another hint: that whole "enable_irq_wake()" - have you possibly spent even five seconds to look at what it actually does? I bet you haven't. Because what it does is to call the irq controller "set_wake" function.

Now, grep for that. Whay do you find? Like maybe ARM and BlackFin? Oh, and I note one MIPS platform.

The point is, that whole "irq wake" really is system dependent. We can do helper functions for it, but anybody who thinks it's anything "generic" is totally mistaken.

In other words, making it a sysdev thing is the CORRECT thing to do. It really is not just "here's how you work around something". It really is "this is how the hardware FUNDAMENTALLY WORKS".

Please, stop arguing. At least argue only after you understand what the physical hardware actyally does.

Linus

#99 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Alan Stern

Perhaps these aren't all the sort of usage you're talking about, but I bet most of them are.  It certainly looks like more than just ARM. Maybe not all that much more, but definitely more.  And the number will only grow in the future.

Are you really sure? Because it can't be x86. I'm pretty sure that that is simply not how x86 wake events _work_ - they're not interrupts.

They are not interrupts on every arm platform that implements set_wake either, but it is useful to pretend that they are. If the platform code reads the wakeup status and marks the corresponding interrupt pending, the driver does not need to know if the event occurred before or after the system entered the low power state. I don't know if this can be implemented on x86, but it might be worth looking into.

Quoted 4 lines

And that's the big point that people seem to be missing here: the whole "wake up interrupt" thing is not some generic model in the first place. I strongly suspect that it literally only works on certain architectures.

My point was that it was not specific to our platform. I don't have a problem fixing our platform if this patch is merged, but this is case where a change to the generic code breaks some platforms. I don't think there is a good reason to make the fix arm specific, trivial or not, since any platform implementing set_wake may run into the race condition that this patch introduced. If the platform does not implement set_wake, IRQ_WAKEUP never gets set, and the fix should not have any effect.

Signature · 1 line
Arve Hjønnevåg
#100 lore raw

Re: [linux-pm] [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Alan Stern

Perhaps these aren't all the sort of usage you're talking about, but I bet most of them are.  It certainly looks like more than just ARM. Maybe not all that much more, but definitely more.  And the number will only grow in the future.

Are you really sure? Because it can't be x86. I'm pretty sure that that is simply not how x86 wake events _work_ - they're not interrupts.

They are not interrupts on every arm platform that implements set_wake either, but it is useful to pretend that they are. If the platform code reads the wakeup status and marks the corresponding interrupt pending, the driver does not need to know if the event occurred before or after the system entered the low power state. I don't know if this can be implemented on x86, but it might be worth looking into.

Quoted 4 lines

And that's the big point that people seem to be missing here: the whole "wake up interrupt" thing is not some generic model in the first place. I strongly suspect that it literally only works on certain architectures.

My point was that it was not specific to our platform. I don't have a problem fixing our platform if this patch is merged, but this is case where a change to the generic code breaks some platforms. I don't think there is a good reason to make the fix arm specific, trivial or not, since any platform implementing set_wake may run into the race condition that this patch introduced. If the platform does not implement set_wake, IRQ_WAKEUP never gets set, and the fix should not have any effect.

Signature · 1 line
Arve Hjønnevåg
#101 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 14 lines · Arve Hjønnevåg
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Alan Stern

Perhaps these aren't all the sort of usage you're talking about, but I bet most of them are. It certainly looks like more than just ARM. Maybe not all that much more, but definitely more. And the number will only grow in the future.

Are you really sure? Because it can't be x86. I'm pretty sure that that is simply not how x86 wake events _work_ - they're not interrupts.

Quoted 6 lines

They are not interrupts on every arm platform that implements set_wake either, but it is useful to pretend that they are. If the platform code reads the wakeup status and marks the corresponding interrupt pending, the driver does not need to know if the event occurred before or after the system entered the low power state. I don't know if this can be implemented on x86, but it might be worth looking into.

That would have been a new feature, no? And I don't think anyone except for you does it. So, what you're saying boils down to "please don't break my new feature that hasn't been merged yet".

Quoted 7 lines
Quoted 3 lines

And that's the big point that people seem to be missing here: the whole "wake up interrupt" thing is not some generic model in the first place. I strongly suspect that it literally only works on certain architectures.

My point was that it was not specific to our platform. I don't have a problem fixing our platform if this patch is merged, but this is case where a change to the generic code breaks some platforms.

Quite frankly, I don't really think it will break anything else than your platform.

Still, if Linus agrees, I can put the loop suggested by him directly into sysdev_suspend(). Linus?

Quoted 5 lines

I don't think there is a good reason to make the fix arm specific, trivial or not, since any platform implementing set_wake may run into the race condition that this patch introduced. If the platform does not implement set_wake, IRQ_WAKEUP never gets set, and the fix should not have any effect.

The point is, if we put anything like this into the generic code, platforms start to rely on this and it will become more and more difficult to change at the generic level if need be.

The fact that your platform relies on the generic code to disable IRQs on the CPU at a particular point shows the mechanism very well. :-)

Thanks,
Rafael

#102 lore raw

Re: [linux-pm] [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Arve Hjønnevåg
Quoted 14 lines · Arve Hjønnevåg
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Alan Stern

Perhaps these aren't all the sort of usage you're talking about, but I bet most of them are. It certainly looks like more than just ARM. Maybe not all that much more, but definitely more. And the number will only grow in the future.

Are you really sure? Because it can't be x86. I'm pretty sure that that is simply not how x86 wake events _work_ - they're not interrupts.

Quoted 6 lines

They are not interrupts on every arm platform that implements set_wake either, but it is useful to pretend that they are. If the platform code reads the wakeup status and marks the corresponding interrupt pending, the driver does not need to know if the event occurred before or after the system entered the low power state. I don't know if this can be implemented on x86, but it might be worth looking into.

That would have been a new feature, no? And I don't think anyone except for you does it. So, what you're saying boils down to "please don't break my new feature that hasn't been merged yet".

Quoted 7 lines
Quoted 3 lines

And that's the big point that people seem to be missing here: the whole "wake up interrupt" thing is not some generic model in the first place. I strongly suspect that it literally only works on certain architectures.

My point was that it was not specific to our platform. I don't have a problem fixing our platform if this patch is merged, but this is case where a change to the generic code breaks some platforms.

Quite frankly, I don't really think it will break anything else than your platform.

Still, if Linus agrees, I can put the loop suggested by him directly into sysdev_suspend(). Linus?

Quoted 5 lines

I don't think there is a good reason to make the fix arm specific, trivial or not, since any platform implementing set_wake may run into the race condition that this patch introduced. If the platform does not implement set_wake, IRQ_WAKEUP never gets set, and the fix should not have any effect.

The point is, if we put anything like this into the generic code, platforms start to rely on this and it will become more and more difficult to change at the generic level if need be.

The fact that your platform relies on the generic code to disable IRQs on the CPU at a particular point shows the mechanism very well. :-)

Thanks,
Rafael

#103 lore raw

Re: [linux-pm] [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 3 lines · Rafael J. Wysocki

Still, if Linus agrees, I can put the loop suggested by him directly into sysdev_suspend(). Linus?

I don't much care - it's going to be a no-op on architectures that don't have that kind of "turn an interrupt into a wakeup event" capability. So it's not going to break for things like x86, and it's not like going over the irq list one more time is going to be so expensive as to be noticeable, even if that architecture doesn't ever get any advantage of it.

However - my main worry is that we will notice that different architectures (and possibly even different platforms _within_ the same architecture - depending on which kind of interrupt/pm controller they have) will want to do different things, and actually do something to the interrupt controller itself too at that point.

But we can certainly try starting out with just the generic "if a wakeup interrupt is pending, sysdev_suspend() returns an error immediately".

Linus

#104 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 22 lines · Rafael J. Wysocki
Quoted 14 lines · Arve Hjønnevåg
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Alan Stern

Perhaps these aren't all the sort of usage you're talking about, but I bet most of them are.  It certainly looks like more than just ARM. Maybe not all that much more, but definitely more.  And the number will only grow in the future.

Are you really sure? Because it can't be x86. I'm pretty sure that that is simply not how x86 wake events _work_ - they're not interrupts.

Quoted 6 lines

They are not interrupts on every arm platform that implements set_wake either, but it is useful to pretend that they are. If the platform code reads the wakeup status and marks the corresponding interrupt pending, the driver does not need to know if the event occurred before or after the system entered the low power state. I don't know if this can be implemented on x86, but it might be worth looking into.

Quoted 3 lines

That would have been a new feature, no?  And I don't think anyone except for you does it.  So, what you're saying boils down to "please don't break my new feature that hasn't been merged yet".

I was not referring to our platform, so not this is not a new feature.

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

And that's the big point that people seem to be missing here: the whole "wake up interrupt" thing is not some generic model in the first place. I strongly suspect that it literally only works on certain architectures.

My point was that it was not specific to our platform. I don't have a problem fixing our platform if this patch is merged, but this is case where a change to the generic code breaks some platforms.

Quite frankly, I don't really think it will break anything else than your platform.

That is quite possible, but do other platforms not break because they are already broken? I saw no attempt to avoid race conditions on suspend in the drivers I looked at.

Quoted 2 lines

Still, if Linus agrees, I can put the loop suggested by him directly into sysdev_suspend().  Linus?

I vote for this.

Quoted 7 lines
Quoted 6 lines

I don't think there is a good reason to make the fix arm specific, trivial or not, since any platform implementing set_wake may run into the race condition that this patch introduced. If the platform does not implement set_wake, IRQ_WAKEUP never gets set, and the fix should not have any effect.

Quoted 6 lines

The point is, if we put anything like this into the generic code, platforms start to rely on this and it will become more and more difficult to change at the generic level if need be.

The fact that your platform relies on the generic code to disable IRQs on the CPU at a particular point shows the mechanism very well. :-)

If the generic code did not clear the interrupt I think this would be a stronger point. Since our hardware does not have a mask register, only enable, I find this feature (leave the interrupt enabled, and mask it only if it triggers) of the generic interrupt code quite useful. If the generic code did not do this, the platform code would have to.

Signature · 1 line
Arve Hjønnevåg
#105 lore raw

Re: [linux-pm] [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 22 lines · Rafael J. Wysocki
Quoted 14 lines · Arve Hjønnevåg
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Alan Stern

Perhaps these aren't all the sort of usage you're talking about, but I bet most of them are.  It certainly looks like more than just ARM. Maybe not all that much more, but definitely more.  And the number will only grow in the future.

Are you really sure? Because it can't be x86. I'm pretty sure that that is simply not how x86 wake events _work_ - they're not interrupts.

Quoted 6 lines

They are not interrupts on every arm platform that implements set_wake either, but it is useful to pretend that they are. If the platform code reads the wakeup status and marks the corresponding interrupt pending, the driver does not need to know if the event occurred before or after the system entered the low power state. I don't know if this can be implemented on x86, but it might be worth looking into.

Quoted 3 lines

That would have been a new feature, no?  And I don't think anyone except for you does it.  So, what you're saying boils down to "please don't break my new feature that hasn't been merged yet".

I was not referring to our platform, so not this is not a new feature.

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

And that's the big point that people seem to be missing here: the whole "wake up interrupt" thing is not some generic model in the first place. I strongly suspect that it literally only works on certain architectures.

My point was that it was not specific to our platform. I don't have a problem fixing our platform if this patch is merged, but this is case where a change to the generic code breaks some platforms.

Quite frankly, I don't really think it will break anything else than your platform.

That is quite possible, but do other platforms not break because they are already broken? I saw no attempt to avoid race conditions on suspend in the drivers I looked at.

Quoted 2 lines

Still, if Linus agrees, I can put the loop suggested by him directly into sysdev_suspend().  Linus?

I vote for this.

Quoted 7 lines
Quoted 6 lines

I don't think there is a good reason to make the fix arm specific, trivial or not, since any platform implementing set_wake may run into the race condition that this patch introduced. If the platform does not implement set_wake, IRQ_WAKEUP never gets set, and the fix should not have any effect.

Quoted 6 lines

The point is, if we put anything like this into the generic code, platforms start to rely on this and it will become more and more difficult to change at the generic level if need be.

The fact that your platform relies on the generic code to disable IRQs on the CPU at a particular point shows the mechanism very well. :-)

If the generic code did not clear the interrupt I think this would be a stronger point. Since our hardware does not have a mask register, only enable, I find this feature (leave the interrupt enabled, and mask it only if it triggers) of the generic interrupt code quite useful. If the generic code did not do this, the platform code would have to.

Signature · 1 line
Arve Hjønnevåg
#106 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Rafael J. Wysocki

Well, how exactly the $subject patch does cause this problem to happen?

Rafael, the problem is that if an interrupt happens while it's disabled - but before the CPU has actually turned all interrupts off - the CPU will ACK the interrupt (but just set a flag for it being PENDING), so now the chipset logic around it will not see it as pending any more, so now the chipset won't auto-wake the CPU immediately (or more likely, it won't even suspend it).

Ah, I see now, thanks.

Quoted 12 lines

It's trivial to fix multiple ways, so I wouldn't worry. The most trivial way is to just have some sysdev drievr code simply do something like

static int sysdev_suspend()

  {
	for_each_irq(irq,desc) {
		if (!(desc->flags & IRQF_WAKE))
			continue;
		if (desc->flags & IRQ_PENDING)
			return -EBUSY;
	}
	return 0;
Quoted 4 lines

}

and that should automatically mean that if any irq is pending, the suspend will fail and we'll immediately wake up again.

Yeah.

Thanks,
Rafael

#107 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Rafael J. Wysocki

Well, how exactly the $subject patch does cause this problem to happen?

Rafael, the problem is that if an interrupt happens while it's disabled - but before the CPU has actually turned all interrupts off - the CPU will ACK the interrupt (but just set a flag for it being PENDING), so now the chipset logic around it will not see it as pending any more, so now the chipset won't auto-wake the CPU immediately (or more likely, it won't even suspend it).

Ah, I see now, thanks.

Quoted 12 lines

It's trivial to fix multiple ways, so I wouldn't worry. The most trivial way is to just have some sysdev drievr code simply do something like

static int sysdev_suspend()

  {
	for_each_irq(irq,desc) {
		if (!(desc->flags & IRQF_WAKE))
			continue;
		if (desc->flags & IRQ_PENDING)
			return -EBUSY;
	}
	return 0;
Quoted 4 lines

}

and that should automatically mean that if any irq is pending, the suspend will fail and we'll immediately wake up again.

Yeah.

Thanks,
Rafael

#108 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 5 lines · Rafael J. Wysocki
Quoted 4 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Quoted 6 lines
Quoted 4 lines

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

Calling "disable_irq()" is perfectly fine.

Agreed, I did not mean to indicate otherwise.

Quoted 7 lines

What is not possible on that broken IO-APIC (among other things) is to actually turn the interrupts off at the apic (ie the whole ->shutdown() thing). But that's not what we even want to do. What we care about is just disabling the interrupt from a drievr perspective.

IOW, the patches I have seen are fine, and all the comments from Eric are just confusion about what we want done.

Largely yes.

Quoted 2 lines

WE DO NOT WANT TO TURN OFF THE IO-APIC. That may or may happen later, but that's totally unrelated to this whole "suspend_device_irq()" thing.

Right.

The question I was asking is:
Can we get the broken cpu hotunplug code out of the suspend path?

If we can get the devices into a low power state and not generating interrupts by the time we disable cpus then we do not need to migrate irqs from process context and risk hitting the ioapic bugs.

While related safely suspending cpus is a different problem and a different patch.

Eric

#109 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 2 lines · Eric W. Biederman

The question I was asking is:
Can we get the broken cpu hotunplug code out of the suspend path?

I think we can move it around. I don't think we can get rid of it.

Quoted 3 lines

If we can get the devices into a low power state and not generating interrupts by the time we disable cpus then we do not need to migrate irqs from process context and risk hitting the ioapic bugs.

At least one issue is that the actual final "go to sleep" is something that has to happen on just one CPU. And I'm pretty sure the others have to have gone through the shutdown sequence before that.

And knowing ACPI, the ordering requirements will boil down to something insane, like "you have to turn off the other CPU's _before_ you turn off some od the core devices, because turning off the other CPU's may involve them".

So if what you would _want_ to do is to move the "turn off CPU's" into the very innermost layer, so that different architectures can then decide whether they even need to go through that whole thing or not (because turning off one core will automatically turn off all the others, simply because the power was turned off), I suspect the answer is "no".

So you were probably hoping to never have to have that whole horrible issue with moving interrupts around. I'm afraid I'm not seeing it happen. But maybe we can have it happen after we've disabled all the non-system devices, so that in practice there simply won't be any new interrupts coming in any more.

Linus

#110 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 25 lines · Linus Torvalds
Quoted 2 lines · Eric W. Biederman

The question I was asking is:
Can we get the broken cpu hotunplug code out of the suspend path?

I think we can move it around. I don't think we can get rid of it.

Quoted 3 lines

If we can get the devices into a low power state and not generating interrupts by the time we disable cpus then we do not need to migrate irqs from process context and risk hitting the ioapic bugs.

At least one issue is that the actual final "go to sleep" is something that has to happen on just one CPU. And I'm pretty sure the others have to have gone through the shutdown sequence before that.

And knowing ACPI, the ordering requirements will boil down to something insane, like "you have to turn off the other CPU's _before_ you turn off some od the core devices, because turning off the other CPU's may involve them".

So if what you would _want_ to do is to move the "turn off CPU's" into the very innermost layer, so that different architectures can then decide whether they even need to go through that whole thing or not (because turning off one core will automatically turn off all the others, simply because the power was turned off), I suspect the answer is "no".

Quoted 5 lines

So you were probably hoping to never have to have that whole horrible issue with moving interrupts around. I'm afraid I'm not seeing it happen. But maybe we can have it happen after we've disabled all the non-system devices, so that in practice there simply won't be any new interrupts coming in any more.

Right. That is what I am hoping for. No device interrupts coming into the cpus at the time we turn them off.

We can disable the devices and thus disable the interrupts the devices are sending before we disable the cpus. That should make cpu disable on suspend much easier to get solid then general x86 cpu hot-unplug.

Eric

#111 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 25 lines · Linus Torvalds
Quoted 2 lines · Eric W. Biederman

The question I was asking is:
Can we get the broken cpu hotunplug code out of the suspend path?

I think we can move it around. I don't think we can get rid of it.

Quoted 3 lines

If we can get the devices into a low power state and not generating interrupts by the time we disable cpus then we do not need to migrate irqs from process context and risk hitting the ioapic bugs.

At least one issue is that the actual final "go to sleep" is something that has to happen on just one CPU. And I'm pretty sure the others have to have gone through the shutdown sequence before that.

And knowing ACPI, the ordering requirements will boil down to something insane, like "you have to turn off the other CPU's _before_ you turn off some od the core devices, because turning off the other CPU's may involve them".

So if what you would _want_ to do is to move the "turn off CPU's" into the very innermost layer, so that different architectures can then decide whether they even need to go through that whole thing or not (because turning off one core will automatically turn off all the others, simply because the power was turned off), I suspect the answer is "no".

Quoted 5 lines

So you were probably hoping to never have to have that whole horrible issue with moving interrupts around. I'm afraid I'm not seeing it happen. But maybe we can have it happen after we've disabled all the non-system devices, so that in practice there simply won't be any new interrupts coming in any more.

Right. That is what I am hoping for. No device interrupts coming into the cpus at the time we turn them off.

We can disable the devices and thus disable the interrupts the devices are sending before we disable the cpus. That should make cpu disable on suspend much easier to get solid then general x86 cpu hot-unplug.

Eric

#112 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 5 lines · Rafael J. Wysocki
Quoted 4 lines

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Quoted 6 lines
Quoted 4 lines

Well, that changes things quite a bit, because it means we can't change the suspend-resume sequence in a way we thought we could without fixing all drivers first, but this is exactly what we'd like to avoid by changing the core.

Calling "disable_irq()" is perfectly fine.

Agreed, I did not mean to indicate otherwise.

Quoted 7 lines

What is not possible on that broken IO-APIC (among other things) is to actually turn the interrupts off at the apic (ie the whole ->shutdown() thing). But that's not what we even want to do. What we care about is just disabling the interrupt from a drievr perspective.

IOW, the patches I have seen are fine, and all the comments from Eric are just confusion about what we want done.

Largely yes.

Quoted 2 lines

WE DO NOT WANT TO TURN OFF THE IO-APIC. That may or may happen later, but that's totally unrelated to this whole "suspend_device_irq()" thing.

Right.

The question I was asking is:
Can we get the broken cpu hotunplug code out of the suspend path?

If we can get the devices into a low power state and not generating interrupts by the time we disable cpus then we do not need to migrate irqs from process context and risk hitting the ioapic bugs.

While related safely suspending cpus is a different problem and a different patch.

Eric

#113 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 4 lines · Rafael J. Wysocki

I think the most important source of level triggered interrupts are PCI devices, so perhaps we can make the PCI PM core use bit 10 of the PCI Device Control register to prevent devices from generating INTx after the drivers' suspend routines have been executed?

I wish that were true. As I recall, the original PCI specification did not define this bit, and older PCI devices don't support it. So you can't count on being able to supress interrupt generation this way.

Alan Stern

#114 lore raw

Re: [linux-pm] [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 4 lines · Rafael J. Wysocki

I think the most important source of level triggered interrupts are PCI devices, so perhaps we can make the PCI PM core use bit 10 of the PCI Device Control register to prevent devices from generating INTx after the drivers' suspend routines have been executed?

I wish that were true. As I recall, the original PCI specification did not define this bit, and older PCI devices don't support it. So you can't count on being able to supress interrupt generation this way.

Alan Stern

#115 lore raw

Re: [linux-pm] [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 10 lines · Alan Stern
Quoted 4 lines · Rafael J. Wysocki

I think the most important source of level triggered interrupts are PCI devices, so perhaps we can make the PCI PM core use bit 10 of the PCI Device Control register to prevent devices from generating INTx after the drivers' suspend routines have been executed?

I wish that were true. As I recall, the original PCI specification did not define this bit, and older PCI devices don't support it. So you can't count on being able to supress interrupt generation this way.

It's definitely a new feature. In fact, I think even the current one makes it optional, so even for "new" devices it's very unclear how many of them actually support that bit.

Linus

#116 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki
Quoted 8 lines · Eric W. Biederman
Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines

I don't know where in the state machine this is getting called but I would suggest doing this before we shutdown cpus.

This is the plan. In fact, I'm going to do this in the next patch after the $subject one has been tested and found acceptable.

Quoted 6 lines
Quoted 4 lines

Good to hear. Then let's please get a version of the irq disable that calls shutdown, so we can be certain we don't have hardware irqs in flight.

For the drivers it should not matter for clean cpu shutdown it will.

OK, I will.

My apologies I was wrong. Calling shutdown is not safe.

I just remembered that masking an ioapic from anywhere besides the irq handler can lock the ioapic state machine, and lead to non-recoverable interrupts. It is rare but I have seen it happen. I wanted to figure out how to migrate interrupts outside of interrupt context and this was what prevented me. A suspend/resume cycle might be enough of a reset to get the ioapic out of that state but I don't know.

The only safe way on x86 to shutdown a level triggered ioapic irq outside of irq context is for the driver to program the hardware to not generate an irq.

Therefore doing anything with the irqs at the point where we are suspending them is a formality, and perhaps simply code that ensures in-flight irqs don't make it past a certain point.

I believe we just need to call disable() and print a big nasty warning if any irq comes in after the suspend stage.

Eric

#117 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines

I don't know where in the state machine this is getting called but I would suggest doing this before we shutdown cpus.

This is the plan. In fact, I'm going to do this in the next patch after the $subject one has been tested and found acceptable.

Good to hear. Then let's please get a version of the irq disable that calls shutdown, so we can be certain we don't have hardware irqs in flight.

For the drivers it should not matter for clean cpu shutdown it will.

Eric

#118 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Eric W. Biederman
Quoted 17 lines · Eric W. Biederman
Quoted 17 lines

What makes s2ram fragile is not human failure but the combination of a handful of physical property:

1) Psychology: shutting the lid or pushing the suspend button is a deceivingly 'simple' action to the user. But under the hood, a ton of stuff happens: we deinitialize a lot of things, we go through _all hardware state_, and we do so in a serial fashion. If just one piece fails to do the right thing, the box might not resume. Still, the user expects this 'simple' thing to just work, all the time. No excuses accepted.

2) Length of code: To get a successful s2ram sequence the kernel runs through tens of thousands of lines of code. Code which never gets executed on a normal box - only if we s2ram. If just one step fails, we get a hung box.

Quoted 8 lines
Quoted 6 lines

3) Debuggability: a lot of s2ram code runs with the console off, making any bugs hard to debug. Furthermore we have no meaningful persistent storage either for kernel bug messages. The RTC trick of PM_DEBUG works but is a very narrow channel of information and it takes a lot of time to debug a bug via that method.

Yep that is an issue.

I'd also like to add #4:

4) One more thing that makes s2ram special is that when the resume path finds hardware often in an even more deinitialized form than during normal bootup. During normal bootup the BIOS/firmware has at least done some minimal bootstrap (to get the kernel loaded), which makes life easier for the kernel.

At s2ram stage we've got a completely pure hardware init state, with very minimal firmware activation. So many of the init and deinit problems and bugs we only hit in the s2ram path - which dynamics is again not helpful.

Quoted 9 lines
Quoted 5 lines

The combination of these factors really makes up for a perfect storm in terms of kernel technology: we have this very-deceivingly-simple-looking but complex-and-rarely-executed piece of code, which is very hard to debug.

And much of this as you are finding with this piece of code is how the software was designed rather then how the software needed to be.

Well most of the 4 problems above are externalities and cannot go away just by fixing the kernel.

#1 will always be with us.
#3 needs the hardware to change. It's happening, but slowly. #4 will be with us as long as there's non-Linux BIOSes

#2 is the only thing where we can make a realistic difference, but there's just so much we can do there.

And that still leaves the other three items: each of which is powerful enough of a force to give a bad name to any normal subsystem.

Ingo

#119 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Eric W. Biederman
Quoted 17 lines · Eric W. Biederman
Quoted 17 lines

What makes s2ram fragile is not human failure but the combination of a handful of physical property:

1) Psychology: shutting the lid or pushing the suspend button is a deceivingly 'simple' action to the user. But under the hood, a ton of stuff happens: we deinitialize a lot of things, we go through _all hardware state_, and we do so in a serial fashion. If just one piece fails to do the right thing, the box might not resume. Still, the user expects this 'simple' thing to just work, all the time. No excuses accepted.

2) Length of code: To get a successful s2ram sequence the kernel runs through tens of thousands of lines of code. Code which never gets executed on a normal box - only if we s2ram. If just one step fails, we get a hung box.

Quoted 8 lines
Quoted 6 lines

3) Debuggability: a lot of s2ram code runs with the console off, making any bugs hard to debug. Furthermore we have no meaningful persistent storage either for kernel bug messages. The RTC trick of PM_DEBUG works but is a very narrow channel of information and it takes a lot of time to debug a bug via that method.

Yep that is an issue.

I'd also like to add #4:

4) One more thing that makes s2ram special is that when the resume path finds hardware often in an even more deinitialized form than during normal bootup. During normal bootup the BIOS/firmware has at least done some minimal bootstrap (to get the kernel loaded), which makes life easier for the kernel.

At s2ram stage we've got a completely pure hardware init state, with very minimal firmware activation. So many of the init and deinit problems and bugs we only hit in the s2ram path - which dynamics is again not helpful.

Quoted 9 lines
Quoted 5 lines

The combination of these factors really makes up for a perfect storm in terms of kernel technology: we have this very-deceivingly-simple-looking but complex-and-rarely-executed piece of code, which is very hard to debug.

And much of this as you are finding with this piece of code is how the software was designed rather then how the software needed to be.

Well most of the 4 problems above are externalities and cannot go away just by fixing the kernel.

#1 will always be with us.
#3 needs the hardware to change. It's happening, but slowly. #4 will be with us as long as there's non-Linux BIOSes

#2 is the only thing where we can make a realistic difference, but there's just so much we can do there.

And that still leaves the other three items: each of which is powerful enough of a force to give a bad name to any normal subsystem.

Ingo

#120 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 38 lines · Ingo Molnar
Quoted 17 lines · Eric W. Biederman
Quoted 17 lines

What makes s2ram fragile is not human failure but the combination of a handful of physical property:

1) Psychology: shutting the lid or pushing the suspend button is a deceivingly 'simple' action to the user. But under the hood, a ton of stuff happens: we deinitialize a lot of things, we go through _all hardware state_, and we do so in a serial fashion. If just one piece fails to do the right thing, the box might not resume. Still, the user expects this 'simple' thing to just work, all the time. No excuses accepted.

2) Length of code: To get a successful s2ram sequence the kernel runs through tens of thousands of lines of code. Code which never gets executed on a normal box - only if we s2ram. If just one step fails, we get a hung box.

Quoted 8 lines
Quoted 6 lines

3) Debuggability: a lot of s2ram code runs with the console off, making any bugs hard to debug. Furthermore we have no meaningful persistent storage either for kernel bug messages. The RTC trick of PM_DEBUG works but is a very narrow channel of information and it takes a lot of time to debug a bug via that method.

Yep that is an issue.

I'd also like to add #4:

4) One more thing that makes s2ram special is that when the resume path finds hardware often in an even more deinitialized form than during normal bootup. During normal bootup the BIOS/firmware has at least done some minimal bootstrap (to get the kernel loaded), which makes life easier for the kernel.

Quoted 2 lines

At s2ram stage we've got a completely pure hardware init state, with very minimal firmware activation.

This is very true and at least in some cases done on purpose, AFAICS, due to some timing constraints forced on HW vendors by M$, for example.

Quoted 3 lines

So many of the init and deinit problems and bugs we only hit in the s2ram path - which dynamics is again not helpful.

Plus ACPI requires us to do additional things during suspend-resume that are not done on boot-shutdown and which have their own ordering requirements (not necessarily stated directly, but such that we have do discover experimentally). That also change from one BIOS to another.

Quoted 17 lines
Quoted 9 lines
Quoted 5 lines

The combination of these factors really makes up for a perfect storm in terms of kernel technology: we have this very-deceivingly-simple-looking but complex-and-rarely-executed piece of code, which is very hard to debug.

And much of this as you are finding with this piece of code is how the software was designed rather then how the software needed to be.

Well most of the 4 problems above are externalities and cannot go away just by fixing the kernel.

#1 will always be with us.
#3 needs the hardware to change. It's happening, but slowly. #4 will be with us as long as there's non-Linux BIOSes

Quoted 6 lines

#2 is the only thing where we can make a realistic difference, but there's just so much we can do there.

And that still leaves the other three items: each of which is powerful enough of a force to give a bad name to any normal subsystem.

Agreed.

Thanks,
Rafael

#121 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Ingo Molnar
Quoted 38 lines · Ingo Molnar
Quoted 17 lines · Eric W. Biederman
Quoted 17 lines

What makes s2ram fragile is not human failure but the combination of a handful of physical property:

1) Psychology: shutting the lid or pushing the suspend button is a deceivingly 'simple' action to the user. But under the hood, a ton of stuff happens: we deinitialize a lot of things, we go through _all hardware state_, and we do so in a serial fashion. If just one piece fails to do the right thing, the box might not resume. Still, the user expects this 'simple' thing to just work, all the time. No excuses accepted.

2) Length of code: To get a successful s2ram sequence the kernel runs through tens of thousands of lines of code. Code which never gets executed on a normal box - only if we s2ram. If just one step fails, we get a hung box.

Quoted 8 lines
Quoted 6 lines

3) Debuggability: a lot of s2ram code runs with the console off, making any bugs hard to debug. Furthermore we have no meaningful persistent storage either for kernel bug messages. The RTC trick of PM_DEBUG works but is a very narrow channel of information and it takes a lot of time to debug a bug via that method.

Yep that is an issue.

I'd also like to add #4:

4) One more thing that makes s2ram special is that when the resume path finds hardware often in an even more deinitialized form than during normal bootup. During normal bootup the BIOS/firmware has at least done some minimal bootstrap (to get the kernel loaded), which makes life easier for the kernel.

Quoted 2 lines

At s2ram stage we've got a completely pure hardware init state, with very minimal firmware activation.

This is very true and at least in some cases done on purpose, AFAICS, due to some timing constraints forced on HW vendors by M$, for example.

Quoted 3 lines

So many of the init and deinit problems and bugs we only hit in the s2ram path - which dynamics is again not helpful.

Plus ACPI requires us to do additional things during suspend-resume that are not done on boot-shutdown and which have their own ordering requirements (not necessarily stated directly, but such that we have do discover experimentally). That also change from one BIOS to another.

Quoted 17 lines
Quoted 9 lines
Quoted 5 lines

The combination of these factors really makes up for a perfect storm in terms of kernel technology: we have this very-deceivingly-simple-looking but complex-and-rarely-executed piece of code, which is very hard to debug.

And much of this as you are finding with this piece of code is how the software was designed rather then how the software needed to be.

Well most of the 4 problems above are externalities and cannot go away just by fixing the kernel.

#1 will always be with us.
#3 needs the hardware to change. It's happening, but slowly. #4 will be with us as long as there's non-Linux BIOSes

Quoted 6 lines

#2 is the only thing where we can make a realistic difference, but there's just so much we can do there.

And that still leaves the other three items: each of which is powerful enough of a force to give a bad name to any normal subsystem.

Agreed.

Thanks,
Rafael

#122 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 39 lines · Rafael J. Wysocki
Quoted 38 lines · Ingo Molnar
Quoted 17 lines · Eric W. Biederman
Quoted 17 lines

What makes s2ram fragile is not human failure but the combination of a handful of physical property:

1) Psychology: shutting the lid or pushing the suspend button is a deceivingly 'simple' action to the user. But under the hood, a ton of stuff happens: we deinitialize a lot of things, we go through _all hardware state_, and we do so in a serial fashion. If just one piece fails to do the right thing, the box might not resume. Still, the user expects this 'simple' thing to just work, all the time. No excuses accepted.

2) Length of code: To get a successful s2ram sequence the kernel runs through tens of thousands of lines of code. Code which never gets executed on a normal box - only if we s2ram. If just one step fails, we get a hung box.

Quoted 8 lines
Quoted 6 lines

3) Debuggability: a lot of s2ram code runs with the console off, making any bugs hard to debug. Furthermore we have no meaningful persistent storage either for kernel bug messages. The RTC trick of PM_DEBUG works but is a very narrow channel of information and it takes a lot of time to debug a bug via that method.

Yep that is an issue.

I'd also like to add #4:

4) One more thing that makes s2ram special is that when the resume path finds hardware often in an even more deinitialized form than during normal bootup. During normal bootup the BIOS/firmware has at least done some minimal bootstrap (to get the kernel loaded), which makes life easier for the kernel.

Quoted 6 lines
Quoted 2 lines

At s2ram stage we've got a completely pure hardware init state, with very minimal firmware activation.

This is very true and at least in some cases done on purpose, AFAICS, due to some timing constraints forced on HW vendors by M$, for example.

IMHO i think it's the technically sane thing to do. Personally i trust the quirks of bare metal much more than the combined quirks of firmware _and_ bare metal.

Quoted 9 lines
Quoted 3 lines

So many of the init and deinit problems and bugs we only hit in the s2ram path - which dynamics is again not helpful.

Plus ACPI requires us to do additional things during suspend-resume that are not done on boot-shutdown and which have their own ordering requirements (not necessarily stated directly, but such that we have do discover experimentally). That also change from one BIOS to another.

We could perhaps do a few things here to trigger bugs sooner.

For example at driver init, instead of executing just ->driver_open(), we could execute:

->driver_open()
->driver_suspend()
->driver_resume()

I.e. we'd simulate a suspend+resume mini-step. This makes it sure that the basic driver callbacks are sane. It is also supposed to work because the driver is just being initialized.

This way certain types of bugs would not show up as difficult to debug s2ram regressions - but would show up as 'boot hang' or 'boot crash' bugs.

This does not simulate the "big picture" resume machinery (the dependencies, etc.), nor does it trigger any of the "hardware got really turned off" effects that true resume will trigger - but at least it offloads a portion of the testing space from 's2ram' to 'bootup' testing.

What's your feeling - what percentage of all s2ram regressions in the last year or so could have been triggered this way? Lets assume we had 100 regressions in that timeframe - would it be in the 10 bugs range? Or much lower or much higher?

Ingo

#123 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 63 lines · Ingo Molnar
Quoted 39 lines · Rafael J. Wysocki
Quoted 38 lines · Ingo Molnar
Quoted 17 lines · Eric W. Biederman
Quoted 17 lines

What makes s2ram fragile is not human failure but the combination of a handful of physical property:

1) Psychology: shutting the lid or pushing the suspend button is a deceivingly 'simple' action to the user. But under the hood, a ton of stuff happens: we deinitialize a lot of things, we go through _all hardware state_, and we do so in a serial fashion. If just one piece fails to do the right thing, the box might not resume. Still, the user expects this 'simple' thing to just work, all the time. No excuses accepted.

2) Length of code: To get a successful s2ram sequence the kernel runs through tens of thousands of lines of code. Code which never gets executed on a normal box - only if we s2ram. If just one step fails, we get a hung box.

Quoted 8 lines
Quoted 6 lines

3) Debuggability: a lot of s2ram code runs with the console off, making any bugs hard to debug. Furthermore we have no meaningful persistent storage either for kernel bug messages. The RTC trick of PM_DEBUG works but is a very narrow channel of information and it takes a lot of time to debug a bug via that method.

Yep that is an issue.

I'd also like to add #4:

4) One more thing that makes s2ram special is that when the resume path finds hardware often in an even more deinitialized form than during normal bootup. During normal bootup the BIOS/firmware has at least done some minimal bootstrap (to get the kernel loaded), which makes life easier for the kernel.

Quoted 6 lines
Quoted 2 lines

At s2ram stage we've got a completely pure hardware init state, with very minimal firmware activation.

This is very true and at least in some cases done on purpose, AFAICS, due to some timing constraints forced on HW vendors by M$, for example.

IMHO i think it's the technically sane thing to do. Personally i trust the quirks of bare metal much more than the combined quirks of firmware _and_ bare metal.

Quoted 9 lines
Quoted 3 lines

So many of the init and deinit problems and bugs we only hit in the s2ram path - which dynamics is again not helpful.

Plus ACPI requires us to do additional things during suspend-resume that are not done on boot-shutdown and which have their own ordering requirements (not necessarily stated directly, but such that we have do discover experimentally). That also change from one BIOS to another.

Quoted 8 lines

We could perhaps do a few things here to trigger bugs sooner.

For example at driver init, instead of executing just ->driver_open(), we could execute:

   ->driver_open()
   ->driver_suspend()
   ->driver_resume()

I'm not sure. On PCI we run some code apart from the driver's suspend and resume callbacks, especially in the new framework, and the bus type executes the driver callbacks.

Quoted 7 lines

I.e. we'd simulate a suspend+resume mini-step. This makes it sure that the basic driver callbacks are sane. It is also supposed to work because the driver is just being initialized.

This way certain types of bugs would not show up as difficult to debug s2ram regressions - but would show up as 'boot hang' or 'boot crash' bugs.

There is a testing facility exactly for this (/sys/power/pm_test) that allows you to simulate the entire suspend sequence without suspending as well as some separate pieces of it. Still, it doesn't work very well, because the conditions in which the resume callbacks are being run differ substantially from the conditions right after we get control from the BIOS.

For one example, if ->suspend() puts the device into D3, then your simulated ->resume() will get the device in D3, while the BIOS would probably put it into D0 (at least as far as PCI devices are concerned).

Quoted 10 lines

This does not simulate the "big picture" resume machinery (the dependencies, etc.), nor does it trigger any of the "hardware got really turned off" effects that true resume will trigger - but at least it offloads a portion of the testing space from 's2ram' to 'bootup' testing.

What's your feeling - what percentage of all s2ram regressions in the last year or so could have been triggered this way? Lets assume we had 100 regressions in that timeframe - would it be in the 10 bugs range? Or much lower or much higher?

Very small number of actual bugs with rather a lot of false positives.

IMO there are three basic sources of recent suspend regressions:
1) Arch-dependent changes (x86 mostly) and low-level changes affecting suspend (like PCI bus enumeration, IOMMU etc.), where people didn't realize their modifications would have a broader effect.
2) PM core changes where we weren't sure what was the best way to go (probably I'm to blame for the majority of these).
3) Changes related to graphics (this has always been difficult, but is getting much better now).

Driver regressions, other than the graphics-related, are really a very small fraction.

Well, there still are some known problems unsolved, but that's a different matter.

Thanks,
Rafael

#124 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Ingo Molnar
Quoted 43 lines · Ingo Molnar
Quoted 8 lines · Eric W. Biederman
Quoted 3 lines · Ingo Molnar

I think this aspect has been well-understood during the discussion of this topic and it's just a slightly misleading changelog.

As I was a member of that discussion I did not see that.

Quoted 7 lines

It took me several passes through the patches to realize the goal is to allow drivers to be able to sleep while they are in their late pm shutdown routines.

Why we want this I don't know. But it seems simple enough to implement, and it makes it harder to get the late pm suspend routines wrong, which is always good.

That's not the only goal. The other goal is to further shrink a particular window of suspend fragility: the irqs-disabled stage of the suspend/resume sequence.

Since suspend/resume is a mini-reboot sequence, there's a large amount of code executed - and the variety of code is large as well. We had repeat cases of random drivers re-enabling interrupts and thus breaking other drivers - and these are nasty to debug.

So this patchset disables device IRQs centrally and serializes with pending work - so there's no races with pending IRQs anymore.

The fact that we keep the timer irq running is two-fold: firstly the timer code is special and not really part of the regular suspend/resume sequence.

Drivers want to take timestamps, sometimes they even want to do a small usleep(), etc. Ideally the suspend/resume code is pretty much _the same_ as a regular bootup (and shutdown) code - so we want to provide a similar environment to how drivers initialize and deinitialize, and we want to enable them to share code between bootup/shutdown and suspend/resume agressively.

Quoted 3 lines

So the more generic kernel environment we give these fragile handlers, the better we are off in the end. Since we already had IRQS_TIMER, that was just the natural thing to do.

I am all for sharing code, especially if we can factor if we can find common factors that do the same thing.

I don't know how many times I have found drivers doing something weird in their shutdown routines that they don't know how to get the device out of. The e1000 driver has shown up several times because it likes to suspend the device on shutdown.

The fact that the methods exposed to drivers were only defined to be usable on the s2ram/hibernate path is something I have brought up on more than one occasion as a bad choice.

I'm really not convinced that the rational for separating out the shutdown methods from the remove methods has been very good. That of we don't need to clean up the in-kernel data structures on reboot so why do something extra that can introduce instability.

So having been watching a smaller form of this drama on the reboot path for several years. Having had a device method with fixed semantics, and not the dwm sematics of the historical suspend routing. I expect there is still a ways to go before it is simple and easy for drivers to figure out what they need to implement out of the confusing variety of possible device methods.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

The new suspend code does not rely on truly disabling IRQs on the low level. The purpose is to not get IRQs to drivers
- which might crash/hang/race/misbehave.

Reasonable. I expect one of the problems with drivers getting it wrong is that the interface is too complex for mortal humans to understand.

Quoted 7 lines

The suspend/resume state machine certainly used to be a piece of code that makes a seasoned kernel developer weep in fear.

That has changed drastically in the past few months. The suspend+hibernation logic got unified (at least as far as driver methods go), and all the flow and ordering has been cleaned up and has been made more robust.

I will have to look again. My impression is that overloading a single method is part of what got us into this mess in the first place.

No that I don't see things getting better.

Quoted 17 lines

What makes s2ram fragile is not human failure but the combination of a handful of physical property:

1) Psychology: shutting the lid or pushing the suspend button is a deceivingly 'simple' action to the user. But under the hood, a ton of stuff happens: we deinitialize a lot of things, we go through _all hardware state_, and we do so in a serial fashion. If just one piece fails to do the right thing, the box might not resume. Still, the user expects this 'simple' thing to just work, all the time. No excuses accepted.

2) Length of code: To get a successful s2ram sequence the kernel runs through tens of thousands of lines of code. Code which never gets executed on a normal box - only if we s2ram. If just one step fails, we get a hung box.

Quoted 6 lines

3) Debuggability: a lot of s2ram code runs with the console off, making any bugs hard to debug. Furthermore we have no meaningful persistent storage either for kernel bug messages. The RTC trick of PM_DEBUG works but is a very narrow channel of information and it takes a lot of time to debug a bug via that method.

Yep that is an issue.

Quoted 4 lines

The combination of these factors really makes up for a perfect storm in terms of kernel technology: we have this very-deceivingly-simple-looking but complex-and-rarely-executed piece of code, which is very hard to debug.

And much of this as you are finding with this piece of code is how the software was designed rather then how the software needed to be.

Quoted 6 lines

Even just one of these factors would be enough to make an otherwise healthy subsystem fragile - no wonder s2ram has been a problem ever since it existed in the upstream kernel.

So now we need just one thing: patience and more of the same good stuff that happened lately.

I think there has been some good progress, and so I am happy to be patient. I will still mention on occasion what it seems we are doing wrong. Unfortunately I don't have time to do a lot more than that.

Quoted 19 lines
Quoted 15 lines
Quoted 3 lines

Still, it might make sense to not just use the ->disable sequence but primarily the ->shutdown irqchip method (when it's available in the irqchip).

Disable seems fine to me. This is interesting in the context of all of the irqs that will when masked show up somewhere else (think boot interrupts).

Quoted 6 lines

While we obviously cannot turn off the PIC that delivers timer IRQs at this stage - there's no theoretical reason why the suspend sequence couldnt power down some secondary PICs as well - in some arch code, or maybe even in the generic driver suspend sequence if the device tree is structured carefully enough so that the PIC gets turned off last.

Quoted 3 lines

If the point is simply to prevent deliver of irqs to the drivers I don't see the point of anything more than what the patch does now.

Quoted 6 lines

... except for the usecase i described above. Say some PIC sits on a piece of silicon which gets turned off. I'm not talking about x86 but some custom device. We really dont want that IRQ line to send half of an IRQ message (un-ACK-ed) when it gets turned off. So physically 'suspending' all IRQ lines does make a certain level of long-term sense.

Good point. We will loose both level and edge triggered events that occur between suspending the irqs and restoring them but that is inevitable. So we might as well call shutdown and totally turn off the irqs if we can.

I don't know where in the state machine this is getting called but I would suggest doing this before we shutdown cpus. We are quickly reaching the point where laptops will exceed the 8 core limit, of lowest priority delivery mode. And only in lowest priority delivery mode is it possible to migrate irqs outside of the interrupt handlers.

That plus if we suspend the irqs before shutting down the cpus it means we can safely support more vectors than a single cpu can catch.

I was a little a worried about the shutdown code path because it requires in the worst case acking a level triggered irq when we have it disabled, but looking at ack_apic_level that appears to be a well tested code path. We just can't reprogram the vector.

Quoted 3 lines

There _might_ be one downside: overhead of ->shutdown() methods. With a typical IRQ count on the typical netbook i doubt it's more than ~50 usecs combined.

Eric

#125 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Eric W. Biederman
Quoted 9 lines · Eric W. Biederman
Quoted 3 lines · Ingo Molnar

I think this aspect has been well-understood during the discussion of this topic and it's just a slightly misleading changelog.

As I was a member of that discussion I did not see that.

Quoted 7 lines

It took me several passes through the patches to realize the goal is to allow drivers to be able to sleep while they are in their late pm shutdown routines.

Why we want this I don't know. But it seems simple enough to implement, and it makes it harder to get the late pm suspend routines wrong, which is always good.

To simplify (it's really all in the discussion we had the last few weeks) It boils down to being able to do the proper ACPI calls (which require core interrupts to be on, ie, ACPI uses mutexes, sleeps, etc...) after we have saved and before we restore the PCI config space, in the late suspend or early resume stages of devices.

Cheers,
Ben.

#126 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Eric W. Biederman
Quoted 9 lines · Eric W. Biederman
Quoted 3 lines · Ingo Molnar

I think this aspect has been well-understood during the discussion of this topic and it's just a slightly misleading changelog.

As I was a member of that discussion I did not see that.

Quoted 7 lines

It took me several passes through the patches to realize the goal is to allow drivers to be able to sleep while they are in their late pm shutdown routines.

Why we want this I don't know. But it seems simple enough to implement, and it makes it harder to get the late pm suspend routines wrong, which is always good.

To simplify (it's really all in the discussion we had the last few weeks) It boils down to being able to do the proper ACPI calls (which require core interrupts to be on, ie, ACPI uses mutexes, sleeps, etc...) after we have saved and before we restore the PCI config space, in the late suspend or early resume stages of devices.

Cheers,
Ben.

#127 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

I think this aspect has been well-understood during the discussion of this topic and it's just a slightly misleading changelog.

As I was a member of that discussion I did not see that.

It took me several passes through the patches to realize the goal is to allow drivers to be able to sleep while they are in their late pm shutdown routines.

Why we want this I don't know. But it seems simple enough to implement, and it makes it harder to get the late pm suspend routines wrong, which is always good.

Quoted 3 lines

The new suspend code does not rely on truly disabling IRQs on the low level. The purpose is to not get IRQs to drivers - which might crash/hang/race/misbehave.

Reasonable. I expect one of the problems with drivers getting it wrong is that the interface is too complex for mortal humans to understand.

Quoted 3 lines

Still, it might make sense to not just use the ->disable sequence but primarily the ->shutdown irqchip method (when it's available in the irqchip).

Disable seems fine to me. This is interesting in the context of all of the irqs that will when masked show up somewhere else (think boot interrupts).

Quoted 6 lines

While we obviously cannot turn off the PIC that delivers timer IRQs at this stage - there's no theoretical reason why the suspend sequence couldnt power down some secondary PICs as well
- in some arch code, or maybe even in the generic driver suspend sequence if the device tree is structured carefully enough so that the PIC gets turned off last.

If the point is simply to prevent deliver of irqs to the drivers I don't see the point of anything more than what the patch does now.

Eric

#128 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 27 lines · Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Linus Torvalds

On Sun, 22 Feb 2009, Rafael J. Wysocki wrote:

[--snip--]

Quoted 2 lines

Thanks a lot for your comments, I'll send an updated patch shortly.

The updated patch is appended.

It has been initially tested, but requires more testing, especially with APM, XEN, kexec jump etc.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM: Rework handling of interrupts during suspend-resume (rev. 2)

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Quoted 6 lines

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

I don't have an issue with the code, but I do have an issue with this description of it.

Calling disable especially for ioapics does nothing directly. It simply arranges for the irq to be marked pending and for the irq to be masked if the irq happens.

So what you are doing is arranging so that no interrupts will be delivered to drivers. Not really disabling interrupts at the IO-APIC level.

In addition not all interrupts (even on x86) go through an IO-APIC anymore so describing the patch in terms of an IO-APIC makes it a bit hard to understand what your intent actually is.

Eric

#129 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 28 lines · Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Linus Torvalds

On Sun, 22 Feb 2009, Rafael J. Wysocki wrote:

[--snip--]

Quoted 2 lines

Thanks a lot for your comments, I'll send an updated patch shortly.

The updated patch is appended.

It has been initially tested, but requires more testing, especially with APM, XEN, kexec jump etc.

 arch/x86/kernel/apm_32.c  |   20 ++++++++++++----
 drivers/xen/manage.c      |   32 +++++++++++++++----------
 include/linux/interrupt.h |    3 ++
 include/linux/irq.h       |    1 
 kernel/irq/manage.c       |   57 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec.c            |   10 ++++----
 kernel/power/disk.c       |   46 +++++++++++++++++++++++++++++--------
 kernel/power/main.c       |   20 +++++++++++-----
 8 files changed, 152 insertions(+), 37 deletions(-)

Index: linux-2.6/kernel/irq/manage.c
===================================================================

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 752

Quoted 7 lines
kernel/irq/manage.c +4 -0
@@ -746,3 +746,60 @@ int request_irq(unsigned int irq, irq_ha
 	return retval;
 }
 EXPORT_SYMBOL(request_irq);
+
+#ifdef CONFIG_PM_SLEEP
+/**
+ *	suspend_device_irqs - disable all currently enabled interrupt lines

Code placement nit: please dont put new #ifdef blocks into the core IRQ code, add a kernel/irq/power.c file instead and make the kbuild rule depend on PM_SLEEP.

The new suspend_device_irqs() and resume_device_irqs() doesnt use any manage.c internals so this should work straight away.

Quoted 20 lines
kernel/irq/manage.c +20 -0
+ *
+ *	During system-wide suspend or hibernation device interrupts need to be
+ *	disabled at the chip level and this function is provided for this
+ *	purpose.  It disables all interrupt lines that are enabled at the
+ *	moment and sets the IRQ_SUSPENDED flag for them.
+ */
+void suspend_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		unsigned long flags;
+
+		spin_lock_irqsave(&desc->lock, flags);
+
+		if (!desc->depth && desc->action
+		    && !(desc->action->flags & IRQF_TIMER)) {
+			desc->depth++;
+			desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 782

Quoted 10 lines
kernel/irq/manage.c +10 -0
+			desc->chip->disable(irq);
+		}
+
+		spin_unlock_irqrestore(&desc->lock, flags);
+	}
+
+	for_each_irq_desc(irq, desc) {
+		if (desc->status & IRQ_SUSPENDED)
+			synchronize_irq(irq);
+	}

Optimization/code-flow nit: a possibility might be to do a single loop, i.e. i think it's safe to couple the disable+sync bits [as in 99.99% of the cases there will be no in-execution irq handlers when we execute this.]

Something like:

		int do_sync = 0;

		spin_lock_irqsave(&desc->lock, flags);

		if (!desc->depth && desc->action
		    && !(desc->action->flags & IRQF_TIMER)) {

			desc->depth++;
			desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
			desc->chip->disable(irq);
			do_sync = 1;
		}

		spin_unlock_irqrestore(&desc->lock, flags);

		if (do_sync)
			synchronize_irq(irq);

In fact i'd suggest to factor out this logic into a separate __suspend_irq(irq) / __resume_irq(irq) inline helper functions. (They should be inline for the time being as they are not shared-irq-safe so they shouldnt really be exposed to drivers in such a singular capacity.)

Doing so will also fix the line-break ugliness of the first branch - as in a standalone function the condition fits into a single line.

There's a performance reason as well: especially when we have a lot of IRQ descriptors that will be about twice as fast. (with a large iteration scope this function is cachemiss-limited and doing this passes doubles the cachemiss rate.)

Quoted 10 lines
kernel/irq/manage.c +10 -0
+}
+EXPORT_SYMBOL_GPL(suspend_device_irqs);
+
+/**
+ *	resume_device_irqs - enable interrupts disabled by suspend_device_irqs()
+ *
+ *	Enable all interrupt lines previously disabled by suspend_device_irqs()
+ *	that have the IRQ_SUSPENDED flag set.
+ */
+void resume_device_irqs(void)

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 802

Quoted 10 lines
kernel/irq/manage.c +10 -0
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		if (!(desc->status & IRQ_SUSPENDED))
+			continue;
+		desc->status &= ~IRQ_SUSPENDED;
+		enable_irq(irq);
+	}

Robustness+optimization nit: this will work but could be done in a nicer way: enable_irq() should auto-clear IRQ_SUSPENDED. (We already clear flags there so it's even a tiny bit faster this way.)

We definitely dont want IRQ_SUSPENDED to 'leak' out into an enabled line, should something call enable_irq() on a suspended line. So either make it auto-unsuspend in enable_irq(), or add an extra WARN_ON() to enable_irq(), to make sure IRQ_SUSPENDED is always off by that time.

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 804

Quoted 2 lines
kernel/irq/manage.c +2 -0
+     arch_suspend_disable_irqs();
+     BUG_ON(!irqs_disabled());

Please. We just disabled all devices - a BUG_ON() is a very counter-productive thing to do here - chances are the user will never see anything but a hang. So please turn this into a nice WARN_ONCE().

include/linux/interrupt.h extern int early_irq_init(void); · line 474

Quoted 8 lines
include/linux/interrupt.h +2 -0
@@ -470,4 +470,7 @@ extern int early_irq_init(void);
 extern int arch_early_irq_init(void);
 extern int arch_init_chip_data(struct irq_desc *desc, int cpu);
 
+extern void suspend_device_irqs(void);
+extern void resume_device_irqs(void);

Header cleanliness nit: please dont just throw new prototypes to the tail of headers, but think about where they fit in best, logically.

These two new prototypes should go straight after the normal irq line state management functions:

  extern void disable_irq_nosync(unsigned int irq);
  extern void disable_irq(unsigned int irq);
  extern void enable_irq(unsigned int irq);

Perhaps also with a comment like this:

/*
* Note: dont use these functions in driver code - they are for
* core kernel use only.
*/

Quoted 1 line
include/linux/interrupt.h +1 -0
+++ linux-2.6/kernel/power/main.c

[...]

include/linux/interrupt.h

Quoted 3 lines
include/linux/interrupt.h +3 -0
+
+ Unlock:
+	resume_device_irqs();

Small drive-by style nit: while at it could you please fix the capitalization and the naming of the label (and all labels in this file)? The standard label is "out_unlock". [and "err_unlock" for failure cases - but this isnt a failure case.]

There's 43 such bad label names in kernel/power/*.c, see the output of:

git grep '^ [A-Z][a-z].*:$' kernel/power/

arch/x86/kernel/apm_32.c static int suspend(int vetoable) · line 1198

Quoted 10 lines

Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================

arch/x86/kernel/apm_32.c +4 -0
 
+
+	suspend_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();

hm, this is a very repetitive pattern, all around the various suspend/resume variants. Might make sense to make:

  	device_power_down(PMSG_SUSPEND);

do the irq line disabling plus the local irq disabling automatically. That also means it cannot be forgotten. The symmetric action should happen for PMSG_RESUME.

Is there ever a case where we want a different pattern?

drivers/xen/manage.c static int xen_suspend(void *data) · line 43

Quoted 10 lines

Index: linux-2.6/drivers/xen/manage.c
===================================================================

drivers/xen/manage.c +0 -4
@@ -39,12 +39,6 @@ static int xen_suspend(void *data)
 
-	if (!*cancelled) {
-		xen_irq_resume();
-		xen_console_resume();
-		xen_timer_resume();

This change needs a second look. xen_suspend() is a stop_machine() handler and as such executes on specific CPUs, and your change modifies this. OTOH, i had a look at these handlers and it all looks safe. Jeremy?

drivers/xen/manage.c static void do_suspend(void) · line 127

Quoted 2 lines
drivers/xen/manage.c +2 -0
+resume_devices:
+	resume_device_irqs();

Small style nit: labels should start with a space character. I.e. it should be:

Quoted 27 lines
drivers/xen/manage.c +6 -2
+ resume_devices:
+	resume_device_irqs();
 
+++ linux-2.6/kernel/kexec.c
@@ -1454,7 +1454,7 @@ int kernel_kexec(void)
 		if (error)
 			goto Resume_devices;
 		device_pm_lock();
-		local_irq_disable();
+		suspend_device_irqs();
 		/* At this point, device_suspend() has been called,
 		 * but *not* device_power_down(). We *must*
 		 * device_power_down() now.  Otherwise, drivers for
@@ -1464,8 +1464,9 @@ int kernel_kexec(void)
 		 */
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
-			goto Enable_irqs;
+			goto Resume_irqs;
 
+		local_irq_disable();
 		/* Suspend system devices */
 		error = sysdev_suspend(PMSG_FREEZE);
 		if (error)
@@ -1484,9 +1485,10 @@ int kernel_kexec(void)
 	if (kexec_image->preserve_context) {
 		sysdev_resume();

drivers/xen/manage.c int kernel_kexec(void) · line 1494

Quoted 10 lines
drivers/xen/manage.c +3 -2
  Power_up_devices:
-		device_power_up(PMSG_RESTORE);
- Enable_irqs:
 		local_irq_enable();
+		device_power_up(PMSG_RESTORE);
+ Resume_irqs:
+		resume_device_irqs();
 		device_pm_unlock();
 		enable_nonboot_cpus();
  Resume_devices:

(same comment about label style applies here too.)

include/linux/irq.h typedef void (*irq_flow_handler_t)(unsig · line 71

Quoted 12 lines

Index: linux-2.6/include/linux/irq.h
===================================================================

include/linux/irq.h +1 -0
@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsig
 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
 
 #ifdef CONFIG_IRQ_PER_CPU
 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)

Note, you should probably make PM_SLEEP depend on GENERIC_HARDIRQS - as this change will break the build on all non-genirq architectures. (sparc, alpha, etc.)

Ingo

#130 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 28 lines · Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Linus Torvalds

On Sun, 22 Feb 2009, Rafael J. Wysocki wrote:

[--snip--]

Quoted 2 lines

Thanks a lot for your comments, I'll send an updated patch shortly.

The updated patch is appended.

It has been initially tested, but requires more testing, especially with APM, XEN, kexec jump etc.

 arch/x86/kernel/apm_32.c  |   20 ++++++++++++----
 drivers/xen/manage.c      |   32 +++++++++++++++----------
 include/linux/interrupt.h |    3 ++
 include/linux/irq.h       |    1 
 kernel/irq/manage.c       |   57 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec.c            |   10 ++++----
 kernel/power/disk.c       |   46 +++++++++++++++++++++++++++++--------
 kernel/power/main.c       |   20 +++++++++++-----
 8 files changed, 152 insertions(+), 37 deletions(-)

Index: linux-2.6/kernel/irq/manage.c
===================================================================

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 752

Quoted 7 lines
kernel/irq/manage.c +4 -0
@@ -746,3 +746,60 @@ int request_irq(unsigned int irq, irq_ha
 	return retval;
 }
 EXPORT_SYMBOL(request_irq);
+
+#ifdef CONFIG_PM_SLEEP
+/**
+ *	suspend_device_irqs - disable all currently enabled interrupt lines

Code placement nit: please dont put new #ifdef blocks into the core IRQ code, add a kernel/irq/power.c file instead and make the kbuild rule depend on PM_SLEEP.

The new suspend_device_irqs() and resume_device_irqs() doesnt use any manage.c internals so this should work straight away.

Quoted 20 lines
kernel/irq/manage.c +20 -0
+ *
+ *	During system-wide suspend or hibernation device interrupts need to be
+ *	disabled at the chip level and this function is provided for this
+ *	purpose.  It disables all interrupt lines that are enabled at the
+ *	moment and sets the IRQ_SUSPENDED flag for them.
+ */
+void suspend_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		unsigned long flags;
+
+		spin_lock_irqsave(&desc->lock, flags);
+
+		if (!desc->depth && desc->action
+		    && !(desc->action->flags & IRQF_TIMER)) {
+			desc->depth++;
+			desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 782

Quoted 10 lines
kernel/irq/manage.c +10 -0
+			desc->chip->disable(irq);
+		}
+
+		spin_unlock_irqrestore(&desc->lock, flags);
+	}
+
+	for_each_irq_desc(irq, desc) {
+		if (desc->status & IRQ_SUSPENDED)
+			synchronize_irq(irq);
+	}

Optimization/code-flow nit: a possibility might be to do a single loop, i.e. i think it's safe to couple the disable+sync bits [as in 99.99% of the cases there will be no in-execution irq handlers when we execute this.]

Something like:

		int do_sync = 0;

		spin_lock_irqsave(&desc->lock, flags);

		if (!desc->depth && desc->action
		    && !(desc->action->flags & IRQF_TIMER)) {

			desc->depth++;
			desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
			desc->chip->disable(irq);
			do_sync = 1;
		}

		spin_unlock_irqrestore(&desc->lock, flags);

		if (do_sync)
			synchronize_irq(irq);

In fact i'd suggest to factor out this logic into a separate __suspend_irq(irq) / __resume_irq(irq) inline helper functions. (They should be inline for the time being as they are not shared-irq-safe so they shouldnt really be exposed to drivers in such a singular capacity.)

Doing so will also fix the line-break ugliness of the first branch - as in a standalone function the condition fits into a single line.

There's a performance reason as well: especially when we have a lot of IRQ descriptors that will be about twice as fast. (with a large iteration scope this function is cachemiss-limited and doing this passes doubles the cachemiss rate.)

Quoted 10 lines
kernel/irq/manage.c +10 -0
+}
+EXPORT_SYMBOL_GPL(suspend_device_irqs);
+
+/**
+ *	resume_device_irqs - enable interrupts disabled by suspend_device_irqs()
+ *
+ *	Enable all interrupt lines previously disabled by suspend_device_irqs()
+ *	that have the IRQ_SUSPENDED flag set.
+ */
+void resume_device_irqs(void)

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 802

Quoted 10 lines
kernel/irq/manage.c +10 -0
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		if (!(desc->status & IRQ_SUSPENDED))
+			continue;
+		desc->status &= ~IRQ_SUSPENDED;
+		enable_irq(irq);
+	}

Robustness+optimization nit: this will work but could be done in a nicer way: enable_irq() should auto-clear IRQ_SUSPENDED. (We already clear flags there so it's even a tiny bit faster this way.)

We definitely dont want IRQ_SUSPENDED to 'leak' out into an enabled line, should something call enable_irq() on a suspended line. So either make it auto-unsuspend in enable_irq(), or add an extra WARN_ON() to enable_irq(), to make sure IRQ_SUSPENDED is always off by that time.

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 804

Quoted 2 lines
kernel/irq/manage.c +2 -0
+     arch_suspend_disable_irqs();
+     BUG_ON(!irqs_disabled());

Please. We just disabled all devices - a BUG_ON() is a very counter-productive thing to do here - chances are the user will never see anything but a hang. So please turn this into a nice WARN_ONCE().

include/linux/interrupt.h extern int early_irq_init(void); · line 474

Quoted 8 lines
include/linux/interrupt.h +2 -0
@@ -470,4 +470,7 @@ extern int early_irq_init(void);
 extern int arch_early_irq_init(void);
 extern int arch_init_chip_data(struct irq_desc *desc, int cpu);
 
+extern void suspend_device_irqs(void);
+extern void resume_device_irqs(void);

Header cleanliness nit: please dont just throw new prototypes to the tail of headers, but think about where they fit in best, logically.

These two new prototypes should go straight after the normal irq line state management functions:

  extern void disable_irq_nosync(unsigned int irq);
  extern void disable_irq(unsigned int irq);
  extern void enable_irq(unsigned int irq);

Perhaps also with a comment like this:

/*
* Note: dont use these functions in driver code - they are for
* core kernel use only.
*/

Quoted 1 line
include/linux/interrupt.h +1 -0
+++ linux-2.6/kernel/power/main.c

[...]

include/linux/interrupt.h

Quoted 3 lines
include/linux/interrupt.h +3 -0
+
+ Unlock:
+	resume_device_irqs();

Small drive-by style nit: while at it could you please fix the capitalization and the naming of the label (and all labels in this file)? The standard label is "out_unlock". [and "err_unlock" for failure cases - but this isnt a failure case.]

There's 43 such bad label names in kernel/power/*.c, see the output of:

git grep '^ [A-Z][a-z].*:$' kernel/power/

arch/x86/kernel/apm_32.c static int suspend(int vetoable) · line 1198

Quoted 10 lines

Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================

arch/x86/kernel/apm_32.c +4 -0
 
+
+	suspend_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();

hm, this is a very repetitive pattern, all around the various suspend/resume variants. Might make sense to make:

  	device_power_down(PMSG_SUSPEND);

do the irq line disabling plus the local irq disabling automatically. That also means it cannot be forgotten. The symmetric action should happen for PMSG_RESUME.

Is there ever a case where we want a different pattern?

drivers/xen/manage.c static int xen_suspend(void *data) · line 43

Quoted 10 lines

Index: linux-2.6/drivers/xen/manage.c
===================================================================

drivers/xen/manage.c +0 -4
@@ -39,12 +39,6 @@ static int xen_suspend(void *data)
 
-	if (!*cancelled) {
-		xen_irq_resume();
-		xen_console_resume();
-		xen_timer_resume();

This change needs a second look. xen_suspend() is a stop_machine() handler and as such executes on specific CPUs, and your change modifies this. OTOH, i had a look at these handlers and it all looks safe. Jeremy?

drivers/xen/manage.c static void do_suspend(void) · line 127

Quoted 2 lines
drivers/xen/manage.c +2 -0
+resume_devices:
+	resume_device_irqs();

Small style nit: labels should start with a space character. I.e. it should be:

Quoted 27 lines
drivers/xen/manage.c +6 -2
+ resume_devices:
+	resume_device_irqs();
 
+++ linux-2.6/kernel/kexec.c
@@ -1454,7 +1454,7 @@ int kernel_kexec(void)
 		if (error)
 			goto Resume_devices;
 		device_pm_lock();
-		local_irq_disable();
+		suspend_device_irqs();
 		/* At this point, device_suspend() has been called,
 		 * but *not* device_power_down(). We *must*
 		 * device_power_down() now.  Otherwise, drivers for
@@ -1464,8 +1464,9 @@ int kernel_kexec(void)
 		 */
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
-			goto Enable_irqs;
+			goto Resume_irqs;
 
+		local_irq_disable();
 		/* Suspend system devices */
 		error = sysdev_suspend(PMSG_FREEZE);
 		if (error)
@@ -1484,9 +1485,10 @@ int kernel_kexec(void)
 	if (kexec_image->preserve_context) {
 		sysdev_resume();

drivers/xen/manage.c int kernel_kexec(void) · line 1494

Quoted 10 lines
drivers/xen/manage.c +3 -2
  Power_up_devices:
-		device_power_up(PMSG_RESTORE);
- Enable_irqs:
 		local_irq_enable();
+		device_power_up(PMSG_RESTORE);
+ Resume_irqs:
+		resume_device_irqs();
 		device_pm_unlock();
 		enable_nonboot_cpus();
  Resume_devices:

(same comment about label style applies here too.)

include/linux/irq.h typedef void (*irq_flow_handler_t)(unsig · line 71

Quoted 12 lines

Index: linux-2.6/include/linux/irq.h
===================================================================

include/linux/irq.h +1 -0
@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsig
 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
 
 #ifdef CONFIG_IRQ_PER_CPU
 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)

Note, you should probably make PM_SLEEP depend on GENERIC_HARDIRQS - as this change will break the build on all non-genirq architectures. (sparc, alpha, etc.)

Ingo

#131 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 43 lines · Ingo Molnar
Quoted 28 lines · Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Linus Torvalds

On Sun, 22 Feb 2009, Rafael J. Wysocki wrote:

[--snip--]

Quoted 2 lines

Thanks a lot for your comments, I'll send an updated patch shortly.

The updated patch is appended.

It has been initially tested, but requires more testing, especially with APM, XEN, kexec jump etc.

 arch/x86/kernel/apm_32.c  |   20 ++++++++++++----
 drivers/xen/manage.c      |   32 +++++++++++++++----------
 include/linux/interrupt.h |    3 ++
 include/linux/irq.h       |    1 
 kernel/irq/manage.c       |   57 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec.c            |   10 ++++----
 kernel/power/disk.c       |   46 +++++++++++++++++++++++++++++--------
 kernel/power/main.c       |   20 +++++++++++-----
 8 files changed, 152 insertions(+), 37 deletions(-)

Index: linux-2.6/kernel/irq/manage.c
===================================================================

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 752

Quoted 7 lines
kernel/irq/manage.c +4 -0
@@ -746,3 +746,60 @@ int request_irq(unsigned int irq, irq_ha
 	return retval;
 }
 EXPORT_SYMBOL(request_irq);
+
+#ifdef CONFIG_PM_SLEEP
+/**
+ *	suspend_device_irqs - disable all currently enabled interrupt lines

Code placement nit: please dont put new #ifdef blocks into the core IRQ code, add a kernel/irq/power.c file instead and make the kbuild rule depend on PM_SLEEP.

Quoted 2 lines

The new suspend_device_irqs() and resume_device_irqs() doesnt use any manage.c internals so this should work straight away.

OK, I'll do that.

Quoted 27 lines
Quoted 27 lines
kernel/irq/manage.c +27 -0
+ *
+ *	During system-wide suspend or hibernation device interrupts need to be
+ *	disabled at the chip level and this function is provided for this
+ *	purpose.  It disables all interrupt lines that are enabled at the
+ *	moment and sets the IRQ_SUSPENDED flag for them.
+ */
+void suspend_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		unsigned long flags;
+
+		spin_lock_irqsave(&desc->lock, flags);
+
+		if (!desc->depth && desc->action
+		    && !(desc->action->flags & IRQF_TIMER)) {
+			desc->depth++;
+			desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
+			desc->chip->disable(irq);
+		}
+
+		spin_unlock_irqrestore(&desc->lock, flags);
+	}
+
+	for_each_irq_desc(irq, desc) {
Quoted 8 lines

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 782

Quoted 3 lines
kernel/irq/manage.c +3 -0
+		if (desc->status & IRQ_SUSPENDED)
+			synchronize_irq(irq);
+	}

Optimization/code-flow nit: a possibility might be to do a single loop, i.e. i think it's safe to couple the disable+sync bits [as in 99.99% of the cases there will be no in-execution irq handlers when we execute this.]

Well, Linus suggested to do it in a separate loop. I'm fine with both ways.

Quoted 20 lines

Something like:

		int do_sync = 0;

		spin_lock_irqsave(&desc->lock, flags);

		if (!desc->depth && desc->action
		    && !(desc->action->flags & IRQF_TIMER)) {

			desc->depth++;
			desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
			desc->chip->disable(irq);
			do_sync = 1;
		}

		spin_unlock_irqrestore(&desc->lock, flags);

		if (do_sync)
			synchronize_irq(irq);
Quoted 5 lines

In fact i'd suggest to factor out this logic into a separate __suspend_irq(irq) / __resume_irq(irq) inline helper functions. (They should be inline for the time being as they are not shared-irq-safe so they shouldnt really be exposed to drivers in such a singular capacity.)

Good idea, I'll do it.

Quoted 26 lines

Doing so will also fix the line-break ugliness of the first branch - as in a standalone function the condition fits into a single line.

There's a performance reason as well: especially when we have a lot of IRQ descriptors that will be about twice as fast. (with a large iteration scope this function is cachemiss-limited and doing this passes doubles the cachemiss rate.)

Quoted 17 lines
kernel/irq/manage.c +17 -0
+}
+EXPORT_SYMBOL_GPL(suspend_device_irqs);
+
+/**
+ *	resume_device_irqs - enable interrupts disabled by suspend_device_irqs()
+ *
+ *	Enable all interrupt lines previously disabled by suspend_device_irqs()
+ *	that have the IRQ_SUSPENDED flag set.
+ */
+void resume_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		if (!(desc->status & IRQ_SUSPENDED))
+			continue;
Quoted 8 lines

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 802

Quoted 3 lines
kernel/irq/manage.c +3 -0
+		desc->status &= ~IRQ_SUSPENDED;
+		enable_irq(irq);
+	}

Robustness+optimization nit: this will work but could be done in a nicer way: enable_irq() should auto-clear IRQ_SUSPENDED. (We already clear flags there so it's even a tiny bit faster this way.)

OK

Quoted 6 lines

We definitely dont want IRQ_SUSPENDED to 'leak' out into an enabled line, should something call enable_irq() on a suspended line. So either make it auto-unsuspend in enable_irq(), or add an extra WARN_ON() to enable_irq(), to make sure IRQ_SUSPENDED is always off by that time.

Quoted 7 lines

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 804

Quoted 2 lines
kernel/irq/manage.c +2 -0
+     arch_suspend_disable_irqs();
+     BUG_ON(!irqs_disabled());

Please. We just disabled all devices - a BUG_ON() is a very counter-productive thing to do here - chances are the user will never see anything but a hang. So please turn this into a nice WARN_ONCE().

This is just moving code. Also, the BUG_ON() can only affect powerpc and it's there on purpose AFAICS (Johannes?). Anyway, changing that would be a separate patch.

Quoted 20 lines

include/linux/interrupt.h extern int early_irq_init(void); · line 474

Quoted 8 lines
include/linux/interrupt.h +2 -0
@@ -470,4 +470,7 @@ extern int early_irq_init(void);
 extern int arch_early_irq_init(void);
 extern int arch_init_chip_data(struct irq_desc *desc, int cpu);
 
+extern void suspend_device_irqs(void);
+extern void resume_device_irqs(void);

Header cleanliness nit: please dont just throw new prototypes to the tail of headers, but think about where they fit in best, logically.

These two new prototypes should go straight after the normal irq line state management functions:

  extern void disable_irq_nosync(unsigned int irq);
  extern void disable_irq(unsigned int irq);
  extern void enable_irq(unsigned int irq);
Quoted 6 lines

Perhaps also with a comment like this:

/*
* Note: dont use these functions in driver code - they are for
* core kernel use only.
*/

OK, I'll put them in there.

Quoted 9 lines
Quoted 1 line
include/linux/interrupt.h +1 -0
+++ linux-2.6/kernel/power/main.c

[...]

include/linux/interrupt.h

Quoted 3 lines
include/linux/interrupt.h +3 -0
+
+ Unlock:
+	resume_device_irqs();

Small drive-by style nit: while at it could you please fix the capitalization and the naming of the label (and all labels in this file)?

I don't think they are wrong. They are uniform accross the file and it's clear what they mean.

Quoted 2 lines

The standard label is "out_unlock". [and "err_unlock" for failure cases
- but this isnt a failure case.]

Where exactly is this standard defined?

Quoted 4 lines

There's 43 such bad label names in kernel/power/*.c, see the output of:

git grep '^ [A-Z][a-z].*:$' kernel/power/

If you think they are bad, please send a patch to change them.

Quoted 14 lines

arch/x86/kernel/apm_32.c static int suspend(int vetoable) · line 1198

Quoted 10 lines

Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================

arch/x86/kernel/apm_32.c +4 -0
 
+
+	suspend_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();

hm, this is a very repetitive pattern, all around the various suspend/resume variants. Might make sense to make:

Quoted 7 lines
  	device_power_down(PMSG_SUSPEND);

do the irq line disabling plus the local irq disabling automatically. That also means it cannot be forgotten. The symmetric action should happen for PMSG_RESUME.

Is there ever a case where we want a different pattern?

Even if there's no such case, I prefer to call local_irq_disable() explicitly in here, so that it's clearly known where it happens to anyone reading this code.

Doing the "late" suspend of devices and disabling interrupts on the CPU are separate logical steps.

Quoted 16 lines

drivers/xen/manage.c static int xen_suspend(void *data) · line 43

Quoted 10 lines

Index: linux-2.6/drivers/xen/manage.c
===================================================================

drivers/xen/manage.c +0 -4
@@ -39,12 +39,6 @@ static int xen_suspend(void *data)
 
-	if (!*cancelled) {
-		xen_irq_resume();
-		xen_console_resume();
-		xen_timer_resume();

This change needs a second look. xen_suspend() is a stop_machine() handler and as such executes on specific CPUs, and your change modifies this. OTOH, i had a look at these handlers and it all looks safe. Jeremy?

Quoted 5 lines

drivers/xen/manage.c static void do_suspend(void) · line 127

Quoted 2 lines
drivers/xen/manage.c +2 -0
+resume_devices:
+	resume_device_irqs();

Small style nit: labels should start with a space character. I.e. it should be:

I know, but the second label in there starts without a space character and IMO keeping a uniform coding style i a single file is more important than trying to adjust it to a broader set of rules FWIW. I also think that coding style changes shouldn't be mixed with functional changes as far as reasonably possible.

Quoted 50 lines
Quoted 27 lines
drivers/xen/manage.c +6 -2
+ resume_devices:
+	resume_device_irqs();
 
+++ linux-2.6/kernel/kexec.c
@@ -1454,7 +1454,7 @@ int kernel_kexec(void)
 		if (error)
 			goto Resume_devices;
 		device_pm_lock();
-		local_irq_disable();
+		suspend_device_irqs();
 		/* At this point, device_suspend() has been called,
 		 * but *not* device_power_down(). We *must*
 		 * device_power_down() now.  Otherwise, drivers for
@@ -1464,8 +1464,9 @@ int kernel_kexec(void)
 		 */
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
-			goto Enable_irqs;
+			goto Resume_irqs;
 
+		local_irq_disable();
 		/* Suspend system devices */
 		error = sysdev_suspend(PMSG_FREEZE);
 		if (error)
@@ -1484,9 +1485,10 @@ int kernel_kexec(void)
 	if (kexec_image->preserve_context) {
 		sysdev_resume();

drivers/xen/manage.c int kernel_kexec(void) · line 1494

Quoted 10 lines
drivers/xen/manage.c +3 -2
  Power_up_devices:
-		device_power_up(PMSG_RESTORE);
- Enable_irqs:
 		local_irq_enable();
+		device_power_up(PMSG_RESTORE);
+ Resume_irqs:
+		resume_device_irqs();
 		device_pm_unlock();
 		enable_nonboot_cpus();
  Resume_devices:

(same comment about label style applies here too.)

Quoted 10 lines

Index: linux-2.6/include/linux/irq.h
===================================================================

include/linux/irq.h +1 -0
@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsig
 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
 
Quoted 6 lines

include/linux/irq.h typedef void (*irq_flow_handler_t)(unsig · line 71

Quoted 2 lines
include/linux/irq.h +0 -0
 #ifdef CONFIG_IRQ_PER_CPU
 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)

Note, you should probably make PM_SLEEP depend on GENERIC_HARDIRQS - as this change will break the build on all non-genirq architectures. (sparc, alpha, etc.)

PM_SLEEP depends on ARCH_SUSPEND_POSSIBLE || ARCH_HIBERNATION_POSSIBLE, which I don't think is set on these architectures.

Thanlks a lot for your comments.

Rafael

#132 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Ingo Molnar
Quoted 43 lines · Ingo Molnar
Quoted 28 lines · Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Linus Torvalds

On Sun, 22 Feb 2009, Rafael J. Wysocki wrote:

[--snip--]

Quoted 2 lines

Thanks a lot for your comments, I'll send an updated patch shortly.

The updated patch is appended.

It has been initially tested, but requires more testing, especially with APM, XEN, kexec jump etc.

 arch/x86/kernel/apm_32.c  |   20 ++++++++++++----
 drivers/xen/manage.c      |   32 +++++++++++++++----------
 include/linux/interrupt.h |    3 ++
 include/linux/irq.h       |    1 
 kernel/irq/manage.c       |   57 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec.c            |   10 ++++----
 kernel/power/disk.c       |   46 +++++++++++++++++++++++++++++--------
 kernel/power/main.c       |   20 +++++++++++-----
 8 files changed, 152 insertions(+), 37 deletions(-)

Index: linux-2.6/kernel/irq/manage.c
===================================================================

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 752

Quoted 7 lines
kernel/irq/manage.c +4 -0
@@ -746,3 +746,60 @@ int request_irq(unsigned int irq, irq_ha
 	return retval;
 }
 EXPORT_SYMBOL(request_irq);
+
+#ifdef CONFIG_PM_SLEEP
+/**
+ *	suspend_device_irqs - disable all currently enabled interrupt lines

Code placement nit: please dont put new #ifdef blocks into the core IRQ code, add a kernel/irq/power.c file instead and make the kbuild rule depend on PM_SLEEP.

Quoted 2 lines

The new suspend_device_irqs() and resume_device_irqs() doesnt use any manage.c internals so this should work straight away.

OK, I'll do that.

Quoted 27 lines
Quoted 27 lines
kernel/irq/manage.c +27 -0
+ *
+ *	During system-wide suspend or hibernation device interrupts need to be
+ *	disabled at the chip level and this function is provided for this
+ *	purpose.  It disables all interrupt lines that are enabled at the
+ *	moment and sets the IRQ_SUSPENDED flag for them.
+ */
+void suspend_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		unsigned long flags;
+
+		spin_lock_irqsave(&desc->lock, flags);
+
+		if (!desc->depth && desc->action
+		    && !(desc->action->flags & IRQF_TIMER)) {
+			desc->depth++;
+			desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
+			desc->chip->disable(irq);
+		}
+
+		spin_unlock_irqrestore(&desc->lock, flags);
+	}
+
+	for_each_irq_desc(irq, desc) {
Quoted 8 lines

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 782

Quoted 3 lines
kernel/irq/manage.c +3 -0
+		if (desc->status & IRQ_SUSPENDED)
+			synchronize_irq(irq);
+	}

Optimization/code-flow nit: a possibility might be to do a single loop, i.e. i think it's safe to couple the disable+sync bits [as in 99.99% of the cases there will be no in-execution irq handlers when we execute this.]

Well, Linus suggested to do it in a separate loop. I'm fine with both ways.

Quoted 20 lines

Something like:

		int do_sync = 0;

		spin_lock_irqsave(&desc->lock, flags);

		if (!desc->depth && desc->action
		    && !(desc->action->flags & IRQF_TIMER)) {

			desc->depth++;
			desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
			desc->chip->disable(irq);
			do_sync = 1;
		}

		spin_unlock_irqrestore(&desc->lock, flags);

		if (do_sync)
			synchronize_irq(irq);
Quoted 5 lines

In fact i'd suggest to factor out this logic into a separate __suspend_irq(irq) / __resume_irq(irq) inline helper functions. (They should be inline for the time being as they are not shared-irq-safe so they shouldnt really be exposed to drivers in such a singular capacity.)

Good idea, I'll do it.

Quoted 26 lines

Doing so will also fix the line-break ugliness of the first branch - as in a standalone function the condition fits into a single line.

There's a performance reason as well: especially when we have a lot of IRQ descriptors that will be about twice as fast. (with a large iteration scope this function is cachemiss-limited and doing this passes doubles the cachemiss rate.)

Quoted 17 lines
kernel/irq/manage.c +17 -0
+}
+EXPORT_SYMBOL_GPL(suspend_device_irqs);
+
+/**
+ *	resume_device_irqs - enable interrupts disabled by suspend_device_irqs()
+ *
+ *	Enable all interrupt lines previously disabled by suspend_device_irqs()
+ *	that have the IRQ_SUSPENDED flag set.
+ */
+void resume_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		if (!(desc->status & IRQ_SUSPENDED))
+			continue;
Quoted 8 lines

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 802

Quoted 3 lines
kernel/irq/manage.c +3 -0
+		desc->status &= ~IRQ_SUSPENDED;
+		enable_irq(irq);
+	}

Robustness+optimization nit: this will work but could be done in a nicer way: enable_irq() should auto-clear IRQ_SUSPENDED. (We already clear flags there so it's even a tiny bit faster this way.)

OK

Quoted 6 lines

We definitely dont want IRQ_SUSPENDED to 'leak' out into an enabled line, should something call enable_irq() on a suspended line. So either make it auto-unsuspend in enable_irq(), or add an extra WARN_ON() to enable_irq(), to make sure IRQ_SUSPENDED is always off by that time.

Quoted 7 lines

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 804

Quoted 2 lines
kernel/irq/manage.c +2 -0
+     arch_suspend_disable_irqs();
+     BUG_ON(!irqs_disabled());

Please. We just disabled all devices - a BUG_ON() is a very counter-productive thing to do here - chances are the user will never see anything but a hang. So please turn this into a nice WARN_ONCE().

This is just moving code. Also, the BUG_ON() can only affect powerpc and it's there on purpose AFAICS (Johannes?). Anyway, changing that would be a separate patch.

Quoted 20 lines

include/linux/interrupt.h extern int early_irq_init(void); · line 474

Quoted 8 lines
include/linux/interrupt.h +2 -0
@@ -470,4 +470,7 @@ extern int early_irq_init(void);
 extern int arch_early_irq_init(void);
 extern int arch_init_chip_data(struct irq_desc *desc, int cpu);
 
+extern void suspend_device_irqs(void);
+extern void resume_device_irqs(void);

Header cleanliness nit: please dont just throw new prototypes to the tail of headers, but think about where they fit in best, logically.

These two new prototypes should go straight after the normal irq line state management functions:

  extern void disable_irq_nosync(unsigned int irq);
  extern void disable_irq(unsigned int irq);
  extern void enable_irq(unsigned int irq);
Quoted 6 lines

Perhaps also with a comment like this:

/*
* Note: dont use these functions in driver code - they are for
* core kernel use only.
*/

OK, I'll put them in there.

Quoted 9 lines
Quoted 1 line
include/linux/interrupt.h +1 -0
+++ linux-2.6/kernel/power/main.c

[...]

include/linux/interrupt.h

Quoted 3 lines
include/linux/interrupt.h +3 -0
+
+ Unlock:
+	resume_device_irqs();

Small drive-by style nit: while at it could you please fix the capitalization and the naming of the label (and all labels in this file)?

I don't think they are wrong. They are uniform accross the file and it's clear what they mean.

Quoted 2 lines

The standard label is "out_unlock". [and "err_unlock" for failure cases
- but this isnt a failure case.]

Where exactly is this standard defined?

Quoted 4 lines

There's 43 such bad label names in kernel/power/*.c, see the output of:

git grep '^ [A-Z][a-z].*:$' kernel/power/

If you think they are bad, please send a patch to change them.

Quoted 14 lines

arch/x86/kernel/apm_32.c static int suspend(int vetoable) · line 1198

Quoted 10 lines

Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================

arch/x86/kernel/apm_32.c +4 -0
 
+
+	suspend_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();

hm, this is a very repetitive pattern, all around the various suspend/resume variants. Might make sense to make:

Quoted 7 lines
  	device_power_down(PMSG_SUSPEND);

do the irq line disabling plus the local irq disabling automatically. That also means it cannot be forgotten. The symmetric action should happen for PMSG_RESUME.

Is there ever a case where we want a different pattern?

Even if there's no such case, I prefer to call local_irq_disable() explicitly in here, so that it's clearly known where it happens to anyone reading this code.

Doing the "late" suspend of devices and disabling interrupts on the CPU are separate logical steps.

Quoted 16 lines

drivers/xen/manage.c static int xen_suspend(void *data) · line 43

Quoted 10 lines

Index: linux-2.6/drivers/xen/manage.c
===================================================================

drivers/xen/manage.c +0 -4
@@ -39,12 +39,6 @@ static int xen_suspend(void *data)
 
-	if (!*cancelled) {
-		xen_irq_resume();
-		xen_console_resume();
-		xen_timer_resume();

This change needs a second look. xen_suspend() is a stop_machine() handler and as such executes on specific CPUs, and your change modifies this. OTOH, i had a look at these handlers and it all looks safe. Jeremy?

Quoted 5 lines

drivers/xen/manage.c static void do_suspend(void) · line 127

Quoted 2 lines
drivers/xen/manage.c +2 -0
+resume_devices:
+	resume_device_irqs();

Small style nit: labels should start with a space character. I.e. it should be:

I know, but the second label in there starts without a space character and IMO keeping a uniform coding style i a single file is more important than trying to adjust it to a broader set of rules FWIW. I also think that coding style changes shouldn't be mixed with functional changes as far as reasonably possible.

Quoted 50 lines
Quoted 27 lines
drivers/xen/manage.c +6 -2
+ resume_devices:
+	resume_device_irqs();
 
+++ linux-2.6/kernel/kexec.c
@@ -1454,7 +1454,7 @@ int kernel_kexec(void)
 		if (error)
 			goto Resume_devices;
 		device_pm_lock();
-		local_irq_disable();
+		suspend_device_irqs();
 		/* At this point, device_suspend() has been called,
 		 * but *not* device_power_down(). We *must*
 		 * device_power_down() now.  Otherwise, drivers for
@@ -1464,8 +1464,9 @@ int kernel_kexec(void)
 		 */
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
-			goto Enable_irqs;
+			goto Resume_irqs;
 
+		local_irq_disable();
 		/* Suspend system devices */
 		error = sysdev_suspend(PMSG_FREEZE);
 		if (error)
@@ -1484,9 +1485,10 @@ int kernel_kexec(void)
 	if (kexec_image->preserve_context) {
 		sysdev_resume();

drivers/xen/manage.c int kernel_kexec(void) · line 1494

Quoted 10 lines
drivers/xen/manage.c +3 -2
  Power_up_devices:
-		device_power_up(PMSG_RESTORE);
- Enable_irqs:
 		local_irq_enable();
+		device_power_up(PMSG_RESTORE);
+ Resume_irqs:
+		resume_device_irqs();
 		device_pm_unlock();
 		enable_nonboot_cpus();
  Resume_devices:

(same comment about label style applies here too.)

Quoted 10 lines

Index: linux-2.6/include/linux/irq.h
===================================================================

include/linux/irq.h +1 -0
@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsig
 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
 
Quoted 6 lines

include/linux/irq.h typedef void (*irq_flow_handler_t)(unsig · line 71

Quoted 2 lines
include/linux/irq.h +0 -0
 #ifdef CONFIG_IRQ_PER_CPU
 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)

Note, you should probably make PM_SLEEP depend on GENERIC_HARDIRQS - as this change will break the build on all non-genirq architectures. (sparc, alpha, etc.)

PM_SLEEP depends on ARCH_SUSPEND_POSSIBLE || ARCH_HIBERNATION_POSSIBLE, which I don't think is set on these architectures.

Thanlks a lot for your comments.

Rafael

#133 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 20 lines · Rafael J. Wysocki
Quoted 20 lines

arch/x86/kernel/apm_32.c static int suspend(int vetoable) · line 1198

Quoted 10 lines

Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================

arch/x86/kernel/apm_32.c +4 -0
 
+
+	suspend_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();

hm, this is a very repetitive pattern, all around the various suspend/resume variants. Might make sense to make:

  	device_power_down(PMSG_SUSPEND);

do the irq line disabling plus the local irq disabling automatically. That also means it cannot be forgotten. The symmetric action should happen for PMSG_RESUME.

Quoted 5 lines
Quoted 1 line

Is there ever a case where we want a different pattern?

Even if there's no such case, I prefer to call local_irq_disable() explicitly in here, so that it's clearly known where it happens to anyone reading this code.

That property can be implied in the function name:

   	device_power_down_irq_disable(PMSG_SUSPEND);

Open-coding it, if it looks the same in all the cases just increases the chances that someone somewhere copies them incorrectly.

Ingo

#134 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 20 lines · Rafael J. Wysocki
Quoted 20 lines

arch/x86/kernel/apm_32.c static int suspend(int vetoable) · line 1198

Quoted 10 lines

Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================

arch/x86/kernel/apm_32.c +4 -0
 
+
+	suspend_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();

hm, this is a very repetitive pattern, all around the various suspend/resume variants. Might make sense to make:

  	device_power_down(PMSG_SUSPEND);

do the irq line disabling plus the local irq disabling automatically. That also means it cannot be forgotten. The symmetric action should happen for PMSG_RESUME.

Quoted 5 lines
Quoted 1 line

Is there ever a case where we want a different pattern?

Even if there's no such case, I prefer to call local_irq_disable() explicitly in here, so that it's clearly known where it happens to anyone reading this code.

That property can be implied in the function name:

   	device_power_down_irq_disable(PMSG_SUSPEND);

Open-coding it, if it looks the same in all the cases just increases the chances that someone somewhere copies them incorrectly.

Ingo

#135 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 29 lines · Ingo Molnar
Quoted 20 lines · Rafael J. Wysocki
Quoted 20 lines

arch/x86/kernel/apm_32.c static int suspend(int vetoable) · line 1198

Quoted 10 lines

Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================

arch/x86/kernel/apm_32.c +4 -0
 
+
+	suspend_device_irqs();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();

hm, this is a very repetitive pattern, all around the various suspend/resume variants. Might make sense to make:

  	device_power_down(PMSG_SUSPEND);

do the irq line disabling plus the local irq disabling automatically. That also means it cannot be forgotten. The symmetric action should happen for PMSG_RESUME.

Quoted 5 lines
Quoted 1 line

Is there ever a case where we want a different pattern?

Even if there's no such case, I prefer to call local_irq_disable() explicitly in here, so that it's clearly known where it happens to anyone reading this code.

Quoted 7 lines

That property can be implied in the function name:

   	device_power_down_irq_disable(PMSG_SUSPEND);

Open-coding it, if it looks the same in all the cases just increases the chances that someone somewhere copies them incorrectly.

Well, I see your point, but in that case I'd rather couple the disabling of local interrupts on the CPU with sysdev_suspend and the disabling (or whatever Eric would like to call that) of device interrupts with device_power_down().

Thanks,
Rafael

#136 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Ingo Molnar
Quoted 8 lines

That property can be implied in the function name:

   	device_power_down_irq_disable(PMSG_SUSPEND);

Open-coding it, if it looks the same in all the cases just increases the chances that someone somewhere copies them incorrectly.

No. Some archs need to do "special" things at the irq disable point, leave it open coded in the caller please.

Cheers,
Ben.

#137 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Ingo Molnar
Quoted 8 lines

That property can be implied in the function name:

   	device_power_down_irq_disable(PMSG_SUSPEND);

Open-coding it, if it looks the same in all the cases just increases the chances that someone somewhere copies them incorrectly.

No. Some archs need to do "special" things at the irq disable point, leave it open coded in the caller please.

Cheers,
Ben.

#138 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 10 lines · Rafael J. Wysocki
Quoted 5 lines

drivers/xen/manage.c static void do_suspend(void) · line 127

Quoted 2 lines
drivers/xen/manage.c +2 -0
+resume_devices:
+	resume_device_irqs();

Small style nit: labels should start with a space character. I.e. it should be:

I know, but the second label in there starts without a space character and IMO keeping a uniform coding style i a single file is more important than trying to adjust it to a broader set of rules FWIW. [...]

Even though it's just a very small and insignificant detail (nowhere described in the CodingStyle), barely worth the mention (and i already regret having brought it up at all), what you say is wrong on a conceptual level and that alarms me a bit ;-)

It is exactly these kinds of "my code, my style!" world view that results in a crappy overall kernel style.

For a single file to look consistent is just the first (and required) step, what matters even more is for files to have similar coding patterns, to make the style as helpful to the general kernel developer/reviewer/bug-fixer/maintainer as possible.

"code with a helpful style" here means two things:

1) it should understand and adhere to basic style principles. This is just an (often arbitrary) subset of the infinite set of reasonable style guides. The most common-sense ones are written down in Documentation/CodingStyle. There's a lot of leeway, as long as the basic principle of "be helpful" is understood and followed.

2) it should carry meta information outside of the language syntax and it should build expectations about a code's purpose and general structure.

That is essential so that we can find bugs during review.

If each file has a slightly different style to express labels then that means we insert extra entropy and degrades and obfuscates the true meat of the code and hurts the overall reviewability of the code.

In practical terms: i noticed that weird label - otherwise i would not have commented on it. I noticed it because it had the pattern of a comment block (most comment blocks start with capital letters, and for that good reason).

It was completely unnecessary for me to notice that label - it carries no information about the patch itself. Ergo, it would be better in the long run if code does not raise unnecessary mental exceptions. We have a limited set of exceptions we are able to handle during review, lets make sure we use them sparingly.

Sure, there will always be borderline cases where we'll have to agree to disagree, even if we agree about the general principle.

But this is not one of those cases - having a "Suspend:" capitalized label is not something you added to enhance the basic coding style - it is something very uncommon and self-serving which you added in _spite_ of the general principles i believe. It has no other message beyond "I do this because i can!".

I.e. it is not helpful at all. When it comes to coding style the kernel is not a democracy at all.

Quoted 2 lines

[...] I also think that coding style changes shouldn't be mixed with functional changes as far as reasonably possible.

Sure, you got that drive-by review for free, by virtue of context diffs ;-)

Ingo

#139 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 10 lines · Rafael J. Wysocki
Quoted 5 lines

drivers/xen/manage.c static void do_suspend(void) · line 127

Quoted 2 lines
drivers/xen/manage.c +2 -0
+resume_devices:
+	resume_device_irqs();

Small style nit: labels should start with a space character. I.e. it should be:

I know, but the second label in there starts without a space character and IMO keeping a uniform coding style i a single file is more important than trying to adjust it to a broader set of rules FWIW. [...]

Even though it's just a very small and insignificant detail (nowhere described in the CodingStyle), barely worth the mention (and i already regret having brought it up at all), what you say is wrong on a conceptual level and that alarms me a bit ;-)

It is exactly these kinds of "my code, my style!" world view that results in a crappy overall kernel style.

For a single file to look consistent is just the first (and required) step, what matters even more is for files to have similar coding patterns, to make the style as helpful to the general kernel developer/reviewer/bug-fixer/maintainer as possible.

"code with a helpful style" here means two things:

1) it should understand and adhere to basic style principles. This is just an (often arbitrary) subset of the infinite set of reasonable style guides. The most common-sense ones are written down in Documentation/CodingStyle. There's a lot of leeway, as long as the basic principle of "be helpful" is understood and followed.

2) it should carry meta information outside of the language syntax and it should build expectations about a code's purpose and general structure.

That is essential so that we can find bugs during review.

If each file has a slightly different style to express labels then that means we insert extra entropy and degrades and obfuscates the true meat of the code and hurts the overall reviewability of the code.

In practical terms: i noticed that weird label - otherwise i would not have commented on it. I noticed it because it had the pattern of a comment block (most comment blocks start with capital letters, and for that good reason).

It was completely unnecessary for me to notice that label - it carries no information about the patch itself. Ergo, it would be better in the long run if code does not raise unnecessary mental exceptions. We have a limited set of exceptions we are able to handle during review, lets make sure we use them sparingly.

Sure, there will always be borderline cases where we'll have to agree to disagree, even if we agree about the general principle.

But this is not one of those cases - having a "Suspend:" capitalized label is not something you added to enhance the basic coding style - it is something very uncommon and self-serving which you added in _spite_ of the general principles i believe. It has no other message beyond "I do this because i can!".

I.e. it is not helpful at all. When it comes to coding style the kernel is not a democracy at all.

Quoted 2 lines

[...] I also think that coding style changes shouldn't be mixed with functional changes as far as reasonably possible.

Sure, you got that drive-by review for free, by virtue of context diffs ;-)

Ingo

#140 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 53 lines · Ingo Molnar
Quoted 10 lines · Rafael J. Wysocki
Quoted 5 lines

drivers/xen/manage.c static void do_suspend(void) · line 127

Quoted 2 lines
drivers/xen/manage.c +2 -0
+resume_devices:
+	resume_device_irqs();

Small style nit: labels should start with a space character. I.e. it should be:

I know, but the second label in there starts without a space character and IMO keeping a uniform coding style i a single file is more important than trying to adjust it to a broader set of rules FWIW. [...]

Even though it's just a very small and insignificant detail (nowhere described in the CodingStyle), barely worth the mention (and i already regret having brought it up at all), what you say is wrong on a conceptual level and that alarms me a bit ;-)

It is exactly these kinds of "my code, my style!" world view that results in a crappy overall kernel style.

For a single file to look consistent is just the first (and required) step, what matters even more is for files to have similar coding patterns, to make the style as helpful to the general kernel developer/reviewer/bug-fixer/maintainer as possible.

"code with a helpful style" here means two things:

1) it should understand and adhere to basic style principles. This is just an (often arbitrary) subset of the infinite set of reasonable style guides. The most common-sense ones are written down in Documentation/CodingStyle. There's a lot of leeway, as long as the basic principle of "be helpful" is understood and followed.

2) it should carry meta information outside of the language syntax and it should build expectations about a code's purpose and general structure.

That is essential so that we can find bugs during review.

If each file has a slightly different style to express labels then that means we insert extra entropy and degrades and obfuscates the true meat of the code and hurts the overall reviewability of the code.

In practical terms: i noticed that weird label - otherwise i would not have commented on it. I noticed it because it had the pattern of a comment block (most comment blocks start with capital letters, and for that good reason).

Quoted 5 lines

It was completely unnecessary for me to notice that label - it carries no information about the patch itself. Ergo, it would be better in the long run if code does not raise unnecessary mental exceptions. We have a limited set of exceptions we are able to handle during review, lets make sure we use them sparingly.

Just to clarify, I have nothing against labels that are not capitalized etc., actually I can live with whatever style of labels is considered as appropriate and/or helpful.

However, if specific style of labels was chosen for given file in the past and it is consistent over the entire file, I don't think it should be changed in a patch that does a different thing, regardless of who's maintaining the file or who's written the code in question. It should be changed in a separate patch with a changelog describing why this change is being made. I don't really have the time to write such a patch at the moment and I don't really think it's _that_ important. YMMV.

Quoted 10 lines

Sure, there will always be borderline cases where we'll have to agree to disagree, even if we agree about the general principle.

But this is not one of those cases - having a "Suspend:" capitalized label is not something you added to enhance the basic coding style - it is something very uncommon and self-serving which you added in _spite_ of the general principles i believe. It has no other message beyond "I do this because i can!".

Quoted 8 lines

I.e. it is not helpful at all. When it comes to coding style the kernel is not a democracy at all.

Quoted 2 lines

[...] I also think that coding style changes shouldn't be mixed with functional changes as far as reasonably possible.

Sure, you got that drive-by review for free, by virtue of context diffs ;-)

Well, OK. :-)

Still, IMHO it's more helpful if the comments related to the changes that belong to the patch in question are not mixed with comments related to the coding style of the files being modified. Perhaps I'm picky ...

Thanks,
Rafael

#141 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Ingo Molnar
Quoted 53 lines · Ingo Molnar
Quoted 10 lines · Rafael J. Wysocki
Quoted 5 lines

drivers/xen/manage.c static void do_suspend(void) · line 127

Quoted 2 lines
drivers/xen/manage.c +2 -0
+resume_devices:
+	resume_device_irqs();

Small style nit: labels should start with a space character. I.e. it should be:

I know, but the second label in there starts without a space character and IMO keeping a uniform coding style i a single file is more important than trying to adjust it to a broader set of rules FWIW. [...]

Even though it's just a very small and insignificant detail (nowhere described in the CodingStyle), barely worth the mention (and i already regret having brought it up at all), what you say is wrong on a conceptual level and that alarms me a bit ;-)

It is exactly these kinds of "my code, my style!" world view that results in a crappy overall kernel style.

For a single file to look consistent is just the first (and required) step, what matters even more is for files to have similar coding patterns, to make the style as helpful to the general kernel developer/reviewer/bug-fixer/maintainer as possible.

"code with a helpful style" here means two things:

1) it should understand and adhere to basic style principles. This is just an (often arbitrary) subset of the infinite set of reasonable style guides. The most common-sense ones are written down in Documentation/CodingStyle. There's a lot of leeway, as long as the basic principle of "be helpful" is understood and followed.

2) it should carry meta information outside of the language syntax and it should build expectations about a code's purpose and general structure.

That is essential so that we can find bugs during review.

If each file has a slightly different style to express labels then that means we insert extra entropy and degrades and obfuscates the true meat of the code and hurts the overall reviewability of the code.

In practical terms: i noticed that weird label - otherwise i would not have commented on it. I noticed it because it had the pattern of a comment block (most comment blocks start with capital letters, and for that good reason).

Quoted 5 lines

It was completely unnecessary for me to notice that label - it carries no information about the patch itself. Ergo, it would be better in the long run if code does not raise unnecessary mental exceptions. We have a limited set of exceptions we are able to handle during review, lets make sure we use them sparingly.

Just to clarify, I have nothing against labels that are not capitalized etc., actually I can live with whatever style of labels is considered as appropriate and/or helpful.

However, if specific style of labels was chosen for given file in the past and it is consistent over the entire file, I don't think it should be changed in a patch that does a different thing, regardless of who's maintaining the file or who's written the code in question. It should be changed in a separate patch with a changelog describing why this change is being made. I don't really have the time to write such a patch at the moment and I don't really think it's _that_ important. YMMV.

Quoted 10 lines

Sure, there will always be borderline cases where we'll have to agree to disagree, even if we agree about the general principle.

But this is not one of those cases - having a "Suspend:" capitalized label is not something you added to enhance the basic coding style - it is something very uncommon and self-serving which you added in _spite_ of the general principles i believe. It has no other message beyond "I do this because i can!".

Quoted 8 lines

I.e. it is not helpful at all. When it comes to coding style the kernel is not a democracy at all.

Quoted 2 lines

[...] I also think that coding style changes shouldn't be mixed with functional changes as far as reasonably possible.

Sure, you got that drive-by review for free, by virtue of context diffs ;-)

Well, OK. :-)

Still, IMHO it's more helpful if the comments related to the changes that belong to the patch in question are not mixed with comments related to the coding style of the files being modified. Perhaps I'm picky ...

Thanks,
Rafael

#142 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 11 lines · Rafael J. Wysocki
Quoted 7 lines

kernel/power/main.c void __attribute__ ((weak)) arch_suspend · line 303

Quoted 2 lines
kernel/power/main.c +2 -0
+     arch_suspend_disable_irqs();
+     BUG_ON(!irqs_disabled());

Please. We just disabled all devices - a BUG_ON() is a very counter-productive thing to do here - chances are the user will never see anything but a hang. So please turn this into a nice WARN_ONCE().

This is just moving code. Also, the BUG_ON() can only affect powerpc and it's there on purpose AFAICS (Johannes?). Anyway, changing that would be a separate patch.

It can affect any platform that overrides the weak symbol arch_suspend_disable_irqs(), and I think that if you're writing this low-level code you better have a way to debug. As such, I don't think it needs changing, because you can only ever see that while implementing arch_suspend_disable_irqs(). OTOH, since it can only trigger then, a WARN_ON is probably fine as well since you'll be getting your machine into inconsistent states all the time while implementing this ;)

johannes

#143 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 11 lines · Rafael J. Wysocki
Quoted 7 lines

kernel/power/main.c void __attribute__ ((weak)) arch_suspend · line 303

Quoted 2 lines
kernel/power/main.c +2 -0
+     arch_suspend_disable_irqs();
+     BUG_ON(!irqs_disabled());

Please. We just disabled all devices - a BUG_ON() is a very counter-productive thing to do here - chances are the user will never see anything but a hang. So please turn this into a nice WARN_ONCE().

This is just moving code. Also, the BUG_ON() can only affect powerpc and it's there on purpose AFAICS (Johannes?). Anyway, changing that would be a separate patch.

It can affect any platform that overrides the weak symbol arch_suspend_disable_irqs(), and I think that if you're writing this low-level code you better have a way to debug. As such, I don't think it needs changing, because you can only ever see that while implementing arch_suspend_disable_irqs(). OTOH, since it can only trigger then, a WARN_ON is probably fine as well since you'll be getting your machine into inconsistent states all the time while implementing this ;)

johannes

#144 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 30 lines · Rafael J. Wysocki
Quoted 21 lines
Quoted 21 lines
kernel/irq/manage.c +21 -0
+void suspend_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		unsigned long flags;
+
+		spin_lock_irqsave(&desc->lock, flags);
+
+		if (!desc->depth && desc->action
+		    && !(desc->action->flags & IRQF_TIMER)) {
+			desc->depth++;
+			desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
+			desc->chip->disable(irq);
+		}
+
+		spin_unlock_irqrestore(&desc->lock, flags);
+	}
+
+	for_each_irq_desc(irq, desc) {
Quoted 8 lines

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 782

Quoted 3 lines
kernel/irq/manage.c +3 -0
+		if (desc->status & IRQ_SUSPENDED)
+			synchronize_irq(irq);
+	}

Optimization/code-flow nit: a possibility might be to do a single loop, i.e. i think it's safe to couple the disable+sync bits [as in 99.99% of the cases there will be no in-execution irq handlers when we execute this.]

Quoted 2 lines

Well, Linus suggested to do it in a separate loop. I'm fine with both ways.

Linus, do you have a strong opinion about which variant we should use?

The two approaches are not completely equivalent, the variant suggested by Linus is a bit more 'atomic' - in that it first turns off everything, then looks for everything that needs to be synchronized.

OTOH, it _shouldnt_ make much of a difference on a correctly working system - we ought to be able to disable the irqs one by one and wait on any pending ones on the spot. Maybe if there was some implicit dependency between irq sources it would be more robust to do Linus's version.

Dunno ... no strong feelings either way.

Ingo

#145 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 30 lines · Rafael J. Wysocki
Quoted 21 lines
Quoted 21 lines
kernel/irq/manage.c +21 -0
+void suspend_device_irqs(void)
+{
+	struct irq_desc *desc;
+	int irq;
+
+	for_each_irq_desc(irq, desc) {
+		unsigned long flags;
+
+		spin_lock_irqsave(&desc->lock, flags);
+
+		if (!desc->depth && desc->action
+		    && !(desc->action->flags & IRQF_TIMER)) {
+			desc->depth++;
+			desc->status |= IRQ_DISABLED | IRQ_SUSPENDED;
+			desc->chip->disable(irq);
+		}
+
+		spin_unlock_irqrestore(&desc->lock, flags);
+	}
+
+	for_each_irq_desc(irq, desc) {
Quoted 8 lines

kernel/irq/manage.c int request_irq(unsigned int irq, irq_ha · line 782

Quoted 3 lines
kernel/irq/manage.c +3 -0
+		if (desc->status & IRQ_SUSPENDED)
+			synchronize_irq(irq);
+	}

Optimization/code-flow nit: a possibility might be to do a single loop, i.e. i think it's safe to couple the disable+sync bits [as in 99.99% of the cases there will be no in-execution irq handlers when we execute this.]

Quoted 2 lines

Well, Linus suggested to do it in a separate loop. I'm fine with both ways.

Linus, do you have a strong opinion about which variant we should use?

The two approaches are not completely equivalent, the variant suggested by Linus is a bit more 'atomic' - in that it first turns off everything, then looks for everything that needs to be synchronized.

OTOH, it _shouldnt_ make much of a difference on a correctly working system - we ought to be able to disable the irqs one by one and wait on any pending ones on the spot. Maybe if there was some implicit dependency between irq sources it would be more robust to do Linus's version.

Dunno ... no strong feelings either way.

Ingo

#146 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 3 lines · Ingo Molnar

Linus, do you have a strong opinion about which variant we should use?

Strong? No. I think mine is better just because _if_ another CPU is busy handling an interrupt that we're just now disabling, we'll just go on to the next interrupt. Waiting for them all at the end is always more efficient.

But does it really matter? No. In this case I think we've shut down all other CPU's anyway, so the whole "serialize_irq()" should probably not even be needed.

Linus

#147 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Ingo Molnar

Linus, do you have a strong opinion about which variant we should use?

Quoted 8 lines

Strong? No. I think mine is better just because _if_ another CPU is busy handling an interrupt that we're just now disabling, we'll just go on to the next interrupt. Waiting for them all at the end is always more efficient.

But does it really matter? No. In this case I think we've shut down all other CPU's anyway, so the whole "serialize_irq()" should probably not even be needed.

But we're going to move the shutting down of the other CPUs after this point.

Finally, the sequence is going to be:
- "normal" suspend of devices
- disable device interrupts
- "late" suspend of devices
- _PTS
- disable nonboot CPUs
- local_irq_disable
- sysdev_suspend
[This is because ACPI wants us to put devices into low power states before doing the _PTS, which in turn is supposed to be done before the disabling of nonboot CPUs, and we want to put devices into low power states during "late" suspend. Of course, analogously for the resume part.]

So, I think your version is _really_ better. :-)

Thanks,
Rafael

#148 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Ingo Molnar

Linus, do you have a strong opinion about which variant we should use?

Quoted 8 lines

Strong? No. I think mine is better just because _if_ another CPU is busy handling an interrupt that we're just now disabling, we'll just go on to the next interrupt. Waiting for them all at the end is always more efficient.

But does it really matter? No. In this case I think we've shut down all other CPU's anyway, so the whole "serialize_irq()" should probably not even be needed.

But we're going to move the shutting down of the other CPUs after this point.

Finally, the sequence is going to be:
- "normal" suspend of devices
- disable device interrupts
- "late" suspend of devices
- _PTS
- disable nonboot CPUs
- local_irq_disable
- sysdev_suspend
[This is because ACPI wants us to put devices into low power states before doing the _PTS, which in turn is supposed to be done before the disabling of nonboot CPUs, and we want to put devices into low power states during "late" suspend. Of course, analogously for the resume part.]

So, I think your version is _really_ better. :-)

Thanks,
Rafael

#149 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Quoted 7 lines

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

What impact does this have on wakeup interrupts? Unless you add a check, after masking all interrupt at the CPU, to abort suspend if any wakeup interrupt has IRQ_PENDING set I think you will loose wakeup interrupts (at least for irqs that use default_disable).

Signature · 1 line
Arve Hjønnevåg
#150 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Quoted 7 lines

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

What impact does this have on wakeup interrupts? Unless you add a check, after masking all interrupt at the CPU, to abort suspend if any wakeup interrupt has IRQ_PENDING set I think you will loose wakeup interrupts (at least for irqs that use default_disable).

Signature · 1 line
Arve Hjønnevåg
#151 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 18 lines · Arve Hjønnevåg
Quoted 9 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Quoted 7 lines

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

Quoted 4 lines

What impact does this have on wakeup interrupts? Unless you add a check, after masking all interrupt at the CPU, to abort suspend if any wakeup interrupt has IRQ_PENDING set I think you will loose wakeup interrupts (at least for irqs that use default_disable).

I _think_ they would have to be reenabled after we've called local_irq_disable().

Thanks,
Rafael

#152 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

Quoted 19 lines · Rafael J. Wysocki
Quoted 18 lines · Arve Hjønnevåg
Quoted 16 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

Quoted 7 lines
Quoted 4 lines

What impact does this have on wakeup interrupts? Unless you add a check, after masking all interrupt at the CPU, to abort suspend if any wakeup interrupt has IRQ_PENDING set I think you will loose wakeup interrupts (at least for irqs that use default_disable).

I _think_ they would have to be reenabled after we've called local_irq_disable().

Are you talking about the irq_chip switching from enabled interrupts to wake interrupts? It is not enough for the irq_chip to reenable the hardware interrupt. If the interrupt is edge triggered and occurred after you disabled it, but before local_irq_disable, the only record of it is the IRQ_PENDING flag.

Signature · 1 line
Arve Hjønnevåg
#153 lore raw

Re: [RFC][PATCH 2/2] PM: Rework handling of interrupts during suspend-resume

In reply to Rafael J. Wysocki
Quoted 19 lines · Rafael J. Wysocki
Quoted 18 lines · Arve Hjønnevåg
Quoted 16 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to disable device interrupts (at the IO-APIC level) during suspend or hibernation and enable them during the subsequent resume, respectively, so that the timer interrupts are enabled while "late" suspend callbacks and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device interrupts will be disabled (at the IO-APIC level), with the help of the new helper function, before calling "late" suspend callbacks provided by device drivers and analogously during resume.

Quoted 7 lines
Quoted 4 lines

What impact does this have on wakeup interrupts? Unless you add a check, after masking all interrupt at the CPU, to abort suspend if any wakeup interrupt has IRQ_PENDING set I think you will loose wakeup interrupts (at least for irqs that use default_disable).

I _think_ they would have to be reenabled after we've called local_irq_disable().

Are you talking about the irq_chip switching from enabled interrupts to wake interrupts? It is not enough for the irq_chip to reenable the hardware interrupt. If the interrupt is edge triggered and occurred after you disabled it, but before local_irq_disable, the only record of it is the IRQ_PENDING flag.

Signature · 1 line
Arve Hjønnevåg
#154 lore raw
In reply to Rafael J. Wysocki
Quoted 4 lines · Rafael J. Wysocki

However, x86 currently doesn't set the IRQF_TIMER flag and I need to make it do so before going further in this direction and changing the PCI PM framework to take advantage of the $subject changes, for example.

Actually, you don't.

The modern form of timer interrupt on x86 is the local apic timer, and it doesn't go through the io-apic at all, and is not even visible to the irq subsystem. So it stays enabled through this all.

But for old-style timer interrupts, something like the appended should do it.

Untested, of course, but it looks obvious enough.

Linus

---
 arch/x86/kernel/time_64.c     |    2 +-
 arch/x86/kernel/vmiclock_32.c |    3 ++-
 arch/x86/mach-default/setup.c |    2 +-
 arch/x86/mach-voyager/setup.c |    2 +-
 4 files changed, 5 insertions(+), 4 deletions(-)
arch/x86/kernel/time_64.c +1 -1
index e6e695a..241ec39 100644
@@ -115,7 +115,7 @@ unsigned long __init calibrate_cpu(void)
 
 static struct irqaction irq0 = {
 	.handler	= timer_interrupt,
-	.flags		= IRQF_DISABLED | IRQF_IRQPOLL | IRQF_NOBALANCING,
+	.flags		= IRQF_DISABLED | IRQF_IRQPOLL | IRQF_NOBALANCING | IRQF_TIMER,
 	.mask		= CPU_MASK_NONE,
 	.name		= "timer"
 };
arch/x86/kernel/vmiclock_32.c +2 -1
index bde106c..7a29d5c 100644
@@ -1,3 +1,4 @@
+
 /*
  * VMI paravirtual timer support routines.
  *
@@ -202,7 +203,7 @@ static irqreturn_t vmi_timer_interrupt(int irq, void *dev_id)
 static struct irqaction vmi_clock_action  = {
 	.name 		= "vmi-timer",
 	.handler 	= vmi_timer_interrupt,
-	.flags 		= IRQF_DISABLED | IRQF_NOBALANCING,
+	.flags 		= IRQF_DISABLED | IRQF_NOBALANCING, IRQF_TIMER,
 	.mask 		= CPU_MASK_ALL,
 };
 
arch/x86/mach-default/setup.c +1 -1
index a265a7c..d737542 100644
@@ -96,7 +96,7 @@ void __init trap_init_hook(void)
 
 static struct irqaction irq0  = {
 	.handler = timer_interrupt,
-	.flags = IRQF_DISABLED | IRQF_NOBALANCING | IRQF_IRQPOLL,
+	.flags = IRQF_DISABLED | IRQF_NOBALANCING | IRQF_IRQPOLL, IRQF_TIMER,
 	.mask = CPU_MASK_NONE,
 	.name = "timer"
 };
arch/x86/mach-voyager/setup.c +1 -1
index d914a79..4de9e08 100644
@@ -56,7 +56,7 @@ void __init trap_init_hook(void)
 
 static struct irqaction irq0 = {
 	.handler = timer_interrupt,
-	.flags = IRQF_DISABLED | IRQF_NOBALANCING | IRQF_IRQPOLL,
+	.flags = IRQF_DISABLED | IRQF_NOBALANCING | IRQF_IRQPOLL, IRQF_TIMER,
 	.mask = CPU_MASK_NONE,
 	.name = "timer"
 };
#155 lore raw

Review

Quoted 4 lines
(quoted diff) +3 -0
+	.flags 		= IRQF_DISABLED | IRQF_NOBALANCING, IRQF_TIMER,
+	.flags = IRQF_DISABLED | IRQF_NOBALANCING | IRQF_IRQPOLL, IRQF_TIMER,
+	.flags = IRQF_DISABLED | IRQF_NOBALANCING | IRQF_IRQPOLL, IRQF_TIMER,

s/, IRQF_TIMER/ | IRQF_TIMER

i guess.

Ingo

#156 lore raw
In reply to Linus Torvalds

Review

Quoted 4 lines
(quoted diff) +3 -0
+	.flags 		= IRQF_DISABLED | IRQF_NOBALANCING, IRQF_TIMER,
+	.flags = IRQF_DISABLED | IRQF_NOBALANCING | IRQF_IRQPOLL, IRQF_TIMER,
+	.flags = IRQF_DISABLED | IRQF_NOBALANCING | IRQF_IRQPOLL, IRQF_TIMER,

s/, IRQF_TIMER/ | IRQF_TIMER

i guess.

Ingo

#157 lore raw
Quoted 8 lines · Ingo Molnar

Review

Quoted 4 lines
(quoted diff) +3 -0
+	.flags 		= IRQF_DISABLED | IRQF_NOBALANCING, IRQF_TIMER,
+	.flags = IRQF_DISABLED | IRQF_NOBALANCING | IRQF_IRQPOLL, IRQF_TIMER,
+	.flags = IRQF_DISABLED | IRQF_NOBALANCING | IRQF_IRQPOLL, IRQF_TIMER,

s/, IRQF_TIMER/ | IRQF_TIMER

i guess.

Oops yes. I got one of them right. I guess that's the same one I happened to compile in my config. Duh.

Linus

#158 lore raw
Quoted 3 lines · Linus Torvalds

Oops yes. I got one of them right. I guess that's the same one I happened to compile in my config. Duh.

I committed the trivially fixed version.

I also committed Rafael's patch 1/2 (the one that doesn't actually change anything). Even if we don't do this in 2.6.29, I want to make it easy to test, and get the infrastructure unified.

Linus

#159 lore raw
In reply to Rafael J. Wysocki
Quoted 7 lines · Rafael J. Wysocki

Moreover, the real purpose of these changes is to be able to execute the "late" suspend and "early" resume device callbacks with timer interrupts enabled, so that they can use mutexes etc. However, x86 currently doesn't set the IRQF_TIMER flag and I need to make it do so before going further in this direction and changing the PCI PM framework to take advantage of the $subject changes, for example. So, I need to know how to modify x86 timer code so that the IRQF_TIMER flag is set by it.

How does this sync with the ACPI requirement that the it's late suspend MUST happen with irqs disabled?

Eric

#160 lore raw
Quoted 10 lines · Eric W. Biederman
Quoted 7 lines · Rafael J. Wysocki

Moreover, the real purpose of these changes is to be able to execute the "late" suspend and "early" resume device callbacks with timer interrupts enabled, so that they can use mutexes etc. However, x86 currently doesn't set the IRQF_TIMER flag and I need to make it do so before going further in this direction and changing the PCI PM framework to take advantage of the $subject changes, for example. So, I need to know how to modify x86 timer code so that the IRQF_TIMER flag is set by it.

Quoted 2 lines

How does this sync with the ACPI requirement that the it's late suspend MUST happen with irqs disabled?

If I understand properly what the intention here is, the sysdev suspend and later still happens with hard irqs off.

This is purely the layer between suspend and suspend_late at the driver level that uses the above instead of hard IRQs off in order to be able to properly order the ACPI calls vs. the driver calls.

Ben.

#161 lore raw
In reply to Eric W. Biederman
Quoted 10 lines · Eric W. Biederman
Quoted 7 lines · Rafael J. Wysocki

Moreover, the real purpose of these changes is to be able to execute the "late" suspend and "early" resume device callbacks with timer interrupts enabled, so that they can use mutexes etc. However, x86 currently doesn't set the IRQF_TIMER flag and I need to make it do so before going further in this direction and changing the PCI PM framework to take advantage of the $subject changes, for example. So, I need to know how to modify x86 timer code so that the IRQF_TIMER flag is set by it.

Quoted 2 lines

How does this sync with the ACPI requirement that the it's late suspend MUST happen with irqs disabled?

If I understand properly what the intention here is, the sysdev suspend and later still happens with hard irqs off.

This is purely the layer between suspend and suspend_late at the driver level that uses the above instead of hard IRQs off in order to be able to properly order the ACPI calls vs. the driver calls.

Ben.

#162 lore raw
In reply to Eric W. Biederman
Quoted 3 lines · Eric W. Biederman

How does this sync with the ACPI requirement that the it's late suspend MUST happen with irqs disabled?

All the system device suspend and the actual CPU power-off still happens with CPU interrupts disabled.

It's just that the regular two-phase device suspend code now runs first with interrupts enabled (the regular "->suspend()" callback), and then the second phase runs with the CPU still having interrupts on (and taking timer interrupts), but with the actual device interrupts disabled.

Linus

#163 lore raw
In reply to Rafael J. Wysocki
Quoted 7 lines · Rafael J. Wysocki

Moreover, the real purpose of these changes is to be able to execute the "late" suspend and "early" resume device callbacks with timer interrupts enabled, so that they can use mutexes etc. However, x86 currently doesn't set the IRQF_TIMER flag and I need to make it do so before going further in this direction and changing the PCI PM framework to take advantage of the $subject changes, for example. So, I need to know how to modify x86 timer code so that the IRQF_TIMER flag is set by it.

How does this sync with the ACPI requirement that the it's late suspend MUST happen with irqs disabled?

Eric

#164 lore raw

cover Rework disabling of interrupts during suspend-resume

In reply to Rafael J. Wysocki

Hi,

The following patches modifiy the way in which we handle disabling interrupts during suspend and enabling them during resume. They also change the ordering of the core suspend and hibernation code.

Namely, interrupts are currently disabled on the boot CPU as soon as the nonboot CPUs have been disabled, which doesn't allow device drivers' "late" suspend and "early" resume callbacks to sleep. Among other things this means they cannot execute ACPI AML routines, which leads to problems with suspend-resume of PCI devices, as recently discussed.

1/4 modifies the [suspend|hibernation] and resume code, as well as the other code using the device PM framework, so that device drivers will not receive interrupts during the "late" suspend phase, although interrupts will only be disabled on the CPU right before calling sysdev_suspend() (and analogously during resume). [Ingo, I didn't add your ACK to the patch, because it's changed since you saw it last time.]

2/4 - 4/4 modify the suspend, hibernation and kexec jump code, respectively, so that the "late" phase of suspending devices will happen before the platform "prepare" callback and the disabling of nonboot CPUs (and analogously during resume).

Comments welcome.

Thanks,
Rafael

#165 lore raw

cover Rework disabling of interrupts during suspend-resume

In reply to Rafael J. Wysocki

Hi,

The following patches modifiy the way in which we handle disabling interrupts during suspend and enabling them during resume. They also change the ordering of the core suspend and hibernation code.

Namely, interrupts are currently disabled on the boot CPU as soon as the nonboot CPUs have been disabled, which doesn't allow device drivers' "late" suspend and "early" resume callbacks to sleep. Among other things this means they cannot execute ACPI AML routines, which leads to problems with suspend-resume of PCI devices, as recently discussed.

1/4 modifies the [suspend|hibernation] and resume code, as well as the other code using the device PM framework, so that device drivers will not receive interrupts during the "late" suspend phase, although interrupts will only be disabled on the CPU right before calling sysdev_suspend() (and analogously during resume). [Ingo, I didn't add your ACK to the patch, because it's changed since you saw it last time.]

2/4 - 4/4 modify the suspend, hibernation and kexec jump code, respectively, so that the "late" phase of suspending devices will happen before the platform "prepare" callback and the disabling of nonboot CPUs (and analogously during resume).

Comments welcome.

Thanks,
Rafael

#166 lore raw

RFC PATCH 1/4 PM: Rework handling of interrupts during suspend-resume (rev. 4)

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to prevent device drivers from getting any interrupts (without disabling interrupts on the CPU) during suspend (or hibernation) and to make them start to receive interrupts again during the subsequent resume, respectively. These functions make it possible to keep timer interrupts enabled while the "late" suspend and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device drivers will be prevented from receiving interrupts, with the help of the new helper function, before their "late" suspend callbacks run (and analogously during resume).

In addition, since the device interrups are now disabled before the CPU has turned all interrupts off and the CPU will ACK the interrupts setting the IRQ_PENDING bit for them, check in sysdev_suspend() if any wake-up interrupts are pending and abort suspend if that's the case.

---
 arch/x86/kernel/apm_32.c  |   15 ++++++--
 drivers/base/power/main.c |   20 ++++++-----
 drivers/base/sys.c        |    8 ++++
 drivers/xen/manage.c      |   16 +++++----
 include/linux/interrupt.h |    5 ++
 include/linux/irq.h       |    1 
 kernel/irq/Makefile       |    1 
 kernel/irq/manage.c       |    3 +
 kernel/irq/pm.c           |   78 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec.c            |    8 ++--
 kernel/power/disk.c       |   39 +++++++++++++++++------
 kernel/power/main.c       |   17 ++++++----
 12 files changed, 170 insertions(+), 41 deletions(-)
include/linux/interrupt.h +5 -0
Index: linux-2.6/include/linux/interrupt.h
===================================================================
@@ -106,6 +106,11 @@ extern void disable_irq_nosync(unsigned 
 extern void disable_irq(unsigned int irq);
 extern void enable_irq(unsigned int irq);
 
+/* The following three functions are for the core kernel use only. */
+extern void suspend_device_irqs(void);
+extern void resume_device_irqs(void);
+extern int check_wakeup_irqs(void);
+
 #if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_HARDIRQS)
 
 extern cpumask_var_t irq_default_affinity;
kernel/power/main.c +11 -6
Index: linux-2.6/kernel/power/main.c
===================================================================
@@ -287,17 +287,19 @@ void __attribute__ ((weak)) arch_suspend
  */
 static int suspend_enter(suspend_state_t state)
 {
-	int error = 0;
+	int error;
 
 	device_pm_lock();
-	arch_suspend_disable_irqs();
-	BUG_ON(!irqs_disabled());
 
-	if ((error = device_power_down(PMSG_SUSPEND))) {
+	error = device_power_down(PMSG_SUSPEND);
+	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down\n");
 		goto Done;
 	}
 
+	arch_suspend_disable_irqs();
+	BUG_ON(!irqs_disabled());
+
 	error = sysdev_suspend(PMSG_SUSPEND);
 	if (!error) {
 		if (!suspend_test(TEST_CORE))
@@ -305,11 +307,14 @@ static int suspend_enter(suspend_state_t
 		sysdev_resume();
 	}
 
-	device_power_up(PMSG_RESUME);
- Done:
 	arch_suspend_enable_irqs();
 	BUG_ON(irqs_disabled());
+
+	device_power_up(PMSG_RESUME);
+
+ Done:
 	device_pm_unlock();
+
 	return error;
 }
 
kernel/power/disk.c +29 -10
Index: linux-2.6/kernel/power/disk.c
===================================================================
@@ -214,7 +214,7 @@ static int create_image(int platform_mod
 		return error;
 
 	device_pm_lock();
-	local_irq_disable();
+
 	/* At this point, device_suspend() has been called, but *not*
 	 * device_power_down(). We *must* call device_power_down() now.
 	 * Otherwise, drivers for some devices (e.g. interrupt controllers)
@@ -225,8 +225,11 @@ static int create_image(int platform_mod
 	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down, "
 			"aborting hibernation\n");
-		goto Enable_irqs;
+		goto Unlock;
 	}
+
+	local_irq_disable();
+
 	sysdev_suspend(PMSG_FREEZE);
 	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down, "
@@ -252,12 +255,16 @@ static int create_image(int platform_mod
 	/* NOTE:  device_power_up() is just a resume() for devices
 	 * that suspended with irqs off ... no overall powerup.
 	 */
+
  Power_up_devices:
+	local_irq_enable();
+
 	device_power_up(in_suspend ?
 		(error ? PMSG_RECOVER : PMSG_THAW) : PMSG_RESTORE);
- Enable_irqs:
-	local_irq_enable();
+
+ Unlock:
 	device_pm_unlock();
+
 	return error;
 }
 
@@ -336,13 +343,16 @@ static int resume_target_kernel(void)
 	int error;
 
 	device_pm_lock();
-	local_irq_disable();
+
 	error = device_power_down(PMSG_QUIESCE);
 	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down, "
 			"aborting resume\n");
-		goto Enable_irqs;
+		goto Unlock;
 	}
+
+	local_irq_disable();
+
 	sysdev_suspend(PMSG_QUIESCE);
 	/* We'll ignore saved state, but this gets preempt count (etc) right */
 	save_processor_state();
@@ -366,11 +376,16 @@ static int resume_target_kernel(void)
 	swsusp_free();
 	restore_processor_state();
 	touch_softlockup_watchdog();
+
 	sysdev_resume();
-	device_power_up(PMSG_RECOVER);
- Enable_irqs:
+
 	local_irq_enable();
+
+	device_power_up(PMSG_RECOVER);
+
+ Unlock:
 	device_pm_unlock();
+
 	return error;
 }
 
@@ -447,15 +462,16 @@ int hibernation_platform_enter(void)
 		goto Finish;
 
 	device_pm_lock();
-	local_irq_disable();
+
 	error = device_power_down(PMSG_HIBERNATE);
 	if (!error) {
+		local_irq_disable();
 		sysdev_suspend(PMSG_HIBERNATE);
 		hibernation_ops->enter();
 		/* We should never get here */
 		while (1);
 	}
-	local_irq_enable();
+
 	device_pm_unlock();
 
 	/*
@@ -464,12 +480,15 @@ int hibernation_platform_enter(void)
 	 */
  Finish:
 	hibernation_ops->finish();
+
  Resume_devices:
 	entering_platform_hibernation = false;
 	device_resume(PMSG_RESTORE);
 	resume_console();
+
  Close:
 	hibernation_ops->end();
+
 	return error;
 }
 
arch/x86/kernel/apm_32.c +11 -4
Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================
@@ -1190,8 +1190,10 @@ static int suspend(int vetoable)
 	struct apm_user	*as;
 
 	device_suspend(PMSG_SUSPEND);
-	local_irq_disable();
+
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 
 	local_irq_enable();
@@ -1209,9 +1211,12 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
+
 	device_resume(PMSG_RESUME);
 	queue_event(APM_NORMAL_RESUME, NULL);
 	spin_lock(&user_list_lock);
@@ -1228,8 +1233,9 @@ static void standby(void)
 {
 	int err;
 
-	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 	local_irq_enable();
 
@@ -1239,8 +1245,9 @@ static void standby(void)
 
 	local_irq_disable();
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
 }
 
 static apm_event_t get_event(void)
drivers/xen/manage.c +9 -7
Index: linux-2.6/drivers/xen/manage.c
===================================================================
@@ -39,12 +39,6 @@ static int xen_suspend(void *data)
 
 	BUG_ON(!irqs_disabled());
 
-	err = device_power_down(PMSG_SUSPEND);
-	if (err) {
-		printk(KERN_ERR "xen_suspend: device_power_down failed: %d\n",
-		       err);
-		return err;
-	}
 	err = sysdev_suspend(PMSG_SUSPEND);
 	if (err) {
 		printk(KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n",
@@ -69,7 +63,6 @@ static int xen_suspend(void *data)
 	xen_mm_unpin_all();
 
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 
 	if (!*cancelled) {
 		xen_irq_resume();
@@ -108,6 +101,12 @@ static void do_suspend(void)
 	/* XXX use normal device tree? */
 	xenbus_suspend();
 
+	err = device_power_down(PMSG_SUSPEND);
+	if (err) {
+		printk(KERN_ERR "device_power_down failed: %d\n", err);
+		goto resume_devices;
+	}
+
 	err = stop_machine(xen_suspend, &cancelled, &cpumask_of_cpu(0));
 	if (err) {
 		printk(KERN_ERR "failed to start xen_suspend: %d\n", err);
@@ -120,6 +119,9 @@ static void do_suspend(void)
 	} else
 		xenbus_suspend_cancel();
 
+	device_power_up(PMSG_RESUME);
+
+resume_devices:
 	device_resume(PMSG_RESUME);
 
 	/* Make sure timer events get retriggered on all CPUs */
kernel/kexec.c +4 -4
Index: linux-2.6/kernel/kexec.c
===================================================================
@@ -1454,7 +1454,6 @@ int kernel_kexec(void)
 		if (error)
 			goto Resume_devices;
 		device_pm_lock();
-		local_irq_disable();
 		/* At this point, device_suspend() has been called,
 		 * but *not* device_power_down(). We *must*
 		 * device_power_down() now.  Otherwise, drivers for
@@ -1464,8 +1463,9 @@ int kernel_kexec(void)
 		 */
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
-			goto Enable_irqs;
+			goto Unlock_pm;
 
+		local_irq_disable();
 		/* Suspend system devices */
 		error = sysdev_suspend(PMSG_FREEZE);
 		if (error)
@@ -1484,9 +1484,9 @@ int kernel_kexec(void)
 	if (kexec_image->preserve_context) {
 		sysdev_resume();
  Power_up_devices:
-		device_power_up(PMSG_RESTORE);
- Enable_irqs:
 		local_irq_enable();
+		device_power_up(PMSG_RESTORE);
+ Unlock_pm:
 		device_pm_unlock();
 		enable_nonboot_cpus();
  Resume_devices:
include/linux/irq.h +1 -0
Index: linux-2.6/include/linux/irq.h
===================================================================
@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsig
 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
 
 #ifdef CONFIG_IRQ_PER_CPU
 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)
kernel/irq/pm.c new file +78 -0

78 lines not on this page · raw message

kernel/irq/Makefile +1 -0
Index: linux-2.6/kernel/irq/Makefile
===================================================================
@@ -4,3 +4,4 @@ obj-$(CONFIG_GENERIC_IRQ_PROBE) += autop
 obj-$(CONFIG_PROC_FS) += proc.o
 obj-$(CONFIG_GENERIC_PENDING_IRQ) += migration.o
 obj-$(CONFIG_NUMA_MIGRATE_IRQ_DESC) += numa_migrate.o
+obj-$(CONFIG_PM_SLEEP) += pm.o
kernel/irq/manage.c +2 -1
Index: linux-2.6/kernel/irq/manage.c
===================================================================
@@ -222,8 +222,9 @@ static void __enable_irq(struct irq_desc
 		WARN(1, KERN_WARNING "Unbalanced enable for IRQ %d\n", irq);
 		break;
 	case 1: {
-		unsigned int status = desc->status & ~IRQ_DISABLED;
+		unsigned int status;
 
+		status = desc->status & ~(IRQ_DISABLED | IRQ_SUSPENDED);
 		/* Prevent probing on this irq: */
 		desc->status = status | IRQ_NOPROBE;
 		check_irq_resend(desc, irq);
drivers/base/power/main.c +11 -9
Index: linux-2.6/drivers/base/power/main.c
===================================================================
@@ -23,6 +23,7 @@
 #include <linux/pm.h>
 #include <linux/resume-trace.h>
 #include <linux/rwsem.h>
+#include <linux/interrupt.h>
 
 #include "../base.h"
 #include "power.h"
@@ -305,7 +306,8 @@ static int resume_device_noirq(struct de
  *	Execute the appropriate "noirq resume" callback for all devices marked
  *	as DPM_OFF_IRQ.
  *
- *	Must be called with interrupts disabled and only one CPU running.
+ *	Must be called under dpm_list_mtx.  Device drivers should not receive
+ *	interrupts while it's being executed.
  */
 static void dpm_power_up(pm_message_t state)
 {
@@ -326,14 +328,13 @@ static void dpm_power_up(pm_message_t st
  *	device_power_up - Turn on all devices that need special attention.
  *	@state: PM transition of the system being carried out.
  *
- *	Power on system devices, then devices that required we shut them down
- *	with interrupts disabled.
- *
- *	Must be called with interrupts disabled.
+ *	Call the "early" resume handlers and enable device drivers to receive
+ *	interrupts.
  */
 void device_power_up(pm_message_t state)
 {
 	dpm_power_up(state);
+	resume_device_irqs();
 }
 EXPORT_SYMBOL_GPL(device_power_up);
 
@@ -558,16 +559,17 @@ static int suspend_device_noirq(struct d
  *	device_power_down - Shut down special devices.
  *	@state: PM transition of the system being carried out.
  *
- *	Power down devices that require interrupts to be disabled.
- *	Then power down system devices.
+ *	Prevent device drivers from receiving interrupts and call the "late"
+ *	suspend handlers.
  *
- *	Must be called with interrupts disabled and only one CPU running.
+ *	Must be called under dpm_list_mtx.
  */
 int device_power_down(pm_message_t state)
 {
 	struct device *dev;
 	int error = 0;
 
+	suspend_device_irqs();
 	list_for_each_entry_reverse(dev, &dpm_list, power.entry) {
 		error = suspend_device_noirq(dev, state);
 		if (error) {
@@ -577,7 +579,7 @@ int device_power_down(pm_message_t state
 		dev->power.status = DPM_OFF_IRQ;
 	}
 	if (error)
-		dpm_power_up(resume_event(state));
+		device_power_up(resume_event(state));
 	return error;
 }
 EXPORT_SYMBOL_GPL(device_power_down);
drivers/base/sys.c +8 -0
Index: linux-2.6/drivers/base/sys.c
===================================================================
@@ -22,6 +22,7 @@
 #include <linux/pm.h>
 #include <linux/device.h>
 #include <linux/mutex.h>
+#include <linux/interrupt.h>
 
 #include "base.h"
 
@@ -369,6 +370,13 @@ int sysdev_suspend(pm_message_t state)
 	struct sysdev_driver *drv, *err_drv;
 	int ret;
 
+	pr_debug("Checking wake-up interrupts\n");
+
+	/* Return error code if there are any wake-up interrupts pending */
+	ret = check_wakeup_irqs();
+	if (ret)
+		return ret;
+
 	pr_debug("Suspending System Devices\n");
 
 	list_for_each_entry_reverse(cls, &system_kset->list, kset.kobj.entry) {
#167 lore raw

RFC PATCH 1/4 PM: Rework handling of interrupts during suspend-resume (rev. 4)

In reply to Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to prevent device drivers from getting any interrupts (without disabling interrupts on the CPU) during suspend (or hibernation) and to make them start to receive interrupts again during the subsequent resume, respectively. These functions make it possible to keep timer interrupts enabled while the "late" suspend and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device drivers will be prevented from receiving interrupts, with the help of the new helper function, before their "late" suspend callbacks run (and analogously during resume).

In addition, since the device interrups are now disabled before the CPU has turned all interrupts off and the CPU will ACK the interrupts setting the IRQ_PENDING bit for them, check in sysdev_suspend() if any wake-up interrupts are pending and abort suspend if that's the case.

---
 arch/x86/kernel/apm_32.c  |   15 ++++++--
 drivers/base/power/main.c |   20 ++++++-----
 drivers/base/sys.c        |    8 ++++
 drivers/xen/manage.c      |   16 +++++----
 include/linux/interrupt.h |    5 ++
 include/linux/irq.h       |    1 
 kernel/irq/Makefile       |    1 
 kernel/irq/manage.c       |    3 +
 kernel/irq/pm.c           |   78 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec.c            |    8 ++--
 kernel/power/disk.c       |   39 +++++++++++++++++------
 kernel/power/main.c       |   17 ++++++----
 12 files changed, 170 insertions(+), 41 deletions(-)
include/linux/interrupt.h +5 -0
Index: linux-2.6/include/linux/interrupt.h
===================================================================
@@ -106,6 +106,11 @@ extern void disable_irq_nosync(unsigned 
 extern void disable_irq(unsigned int irq);
 extern void enable_irq(unsigned int irq);
 
+/* The following three functions are for the core kernel use only. */
+extern void suspend_device_irqs(void);
+extern void resume_device_irqs(void);
+extern int check_wakeup_irqs(void);
+
 #if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_HARDIRQS)
 
 extern cpumask_var_t irq_default_affinity;
kernel/power/main.c +11 -6
Index: linux-2.6/kernel/power/main.c
===================================================================
@@ -287,17 +287,19 @@ void __attribute__ ((weak)) arch_suspend
  */
 static int suspend_enter(suspend_state_t state)
 {
-	int error = 0;
+	int error;
 
 	device_pm_lock();
-	arch_suspend_disable_irqs();
-	BUG_ON(!irqs_disabled());
 
-	if ((error = device_power_down(PMSG_SUSPEND))) {
+	error = device_power_down(PMSG_SUSPEND);
+	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down\n");
 		goto Done;
 	}
 
+	arch_suspend_disable_irqs();
+	BUG_ON(!irqs_disabled());
+
 	error = sysdev_suspend(PMSG_SUSPEND);
 	if (!error) {
 		if (!suspend_test(TEST_CORE))
@@ -305,11 +307,14 @@ static int suspend_enter(suspend_state_t
 		sysdev_resume();
 	}
 
-	device_power_up(PMSG_RESUME);
- Done:
 	arch_suspend_enable_irqs();
 	BUG_ON(irqs_disabled());
+
+	device_power_up(PMSG_RESUME);
+
+ Done:
 	device_pm_unlock();
+
 	return error;
 }
 
kernel/power/disk.c +29 -10
Index: linux-2.6/kernel/power/disk.c
===================================================================
@@ -214,7 +214,7 @@ static int create_image(int platform_mod
 		return error;
 
 	device_pm_lock();
-	local_irq_disable();
+
 	/* At this point, device_suspend() has been called, but *not*
 	 * device_power_down(). We *must* call device_power_down() now.
 	 * Otherwise, drivers for some devices (e.g. interrupt controllers)
@@ -225,8 +225,11 @@ static int create_image(int platform_mod
 	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down, "
 			"aborting hibernation\n");
-		goto Enable_irqs;
+		goto Unlock;
 	}
+
+	local_irq_disable();
+
 	sysdev_suspend(PMSG_FREEZE);
 	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down, "
@@ -252,12 +255,16 @@ static int create_image(int platform_mod
 	/* NOTE:  device_power_up() is just a resume() for devices
 	 * that suspended with irqs off ... no overall powerup.
 	 */
+
  Power_up_devices:
+	local_irq_enable();
+
 	device_power_up(in_suspend ?
 		(error ? PMSG_RECOVER : PMSG_THAW) : PMSG_RESTORE);
- Enable_irqs:
-	local_irq_enable();
+
+ Unlock:
 	device_pm_unlock();
+
 	return error;
 }
 
@@ -336,13 +343,16 @@ static int resume_target_kernel(void)
 	int error;
 
 	device_pm_lock();
-	local_irq_disable();
+
 	error = device_power_down(PMSG_QUIESCE);
 	if (error) {
 		printk(KERN_ERR "PM: Some devices failed to power down, "
 			"aborting resume\n");
-		goto Enable_irqs;
+		goto Unlock;
 	}
+
+	local_irq_disable();
+
 	sysdev_suspend(PMSG_QUIESCE);
 	/* We'll ignore saved state, but this gets preempt count (etc) right */
 	save_processor_state();
@@ -366,11 +376,16 @@ static int resume_target_kernel(void)
 	swsusp_free();
 	restore_processor_state();
 	touch_softlockup_watchdog();
+
 	sysdev_resume();
-	device_power_up(PMSG_RECOVER);
- Enable_irqs:
+
 	local_irq_enable();
+
+	device_power_up(PMSG_RECOVER);
+
+ Unlock:
 	device_pm_unlock();
+
 	return error;
 }
 
@@ -447,15 +462,16 @@ int hibernation_platform_enter(void)
 		goto Finish;
 
 	device_pm_lock();
-	local_irq_disable();
+
 	error = device_power_down(PMSG_HIBERNATE);
 	if (!error) {
+		local_irq_disable();
 		sysdev_suspend(PMSG_HIBERNATE);
 		hibernation_ops->enter();
 		/* We should never get here */
 		while (1);
 	}
-	local_irq_enable();
+
 	device_pm_unlock();
 
 	/*
@@ -464,12 +480,15 @@ int hibernation_platform_enter(void)
 	 */
  Finish:
 	hibernation_ops->finish();
+
  Resume_devices:
 	entering_platform_hibernation = false;
 	device_resume(PMSG_RESTORE);
 	resume_console();
+
  Close:
 	hibernation_ops->end();
+
 	return error;
 }
 
arch/x86/kernel/apm_32.c +11 -4
Index: linux-2.6/arch/x86/kernel/apm_32.c
===================================================================
@@ -1190,8 +1190,10 @@ static int suspend(int vetoable)
 	struct apm_user	*as;
 
 	device_suspend(PMSG_SUSPEND);
-	local_irq_disable();
+
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 
 	local_irq_enable();
@@ -1209,9 +1211,12 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
+
 	device_resume(PMSG_RESUME);
 	queue_event(APM_NORMAL_RESUME, NULL);
 	spin_lock(&user_list_lock);
@@ -1228,8 +1233,9 @@ static void standby(void)
 {
 	int err;
 
-	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+
+	local_irq_disable();
 	sysdev_suspend(PMSG_SUSPEND);
 	local_irq_enable();
 
@@ -1239,8 +1245,9 @@ static void standby(void)
 
 	local_irq_disable();
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 	local_irq_enable();
+
+	device_power_up(PMSG_RESUME);
 }
 
 static apm_event_t get_event(void)
drivers/xen/manage.c +9 -7
Index: linux-2.6/drivers/xen/manage.c
===================================================================
@@ -39,12 +39,6 @@ static int xen_suspend(void *data)
 
 	BUG_ON(!irqs_disabled());
 
-	err = device_power_down(PMSG_SUSPEND);
-	if (err) {
-		printk(KERN_ERR "xen_suspend: device_power_down failed: %d\n",
-		       err);
-		return err;
-	}
 	err = sysdev_suspend(PMSG_SUSPEND);
 	if (err) {
 		printk(KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n",
@@ -69,7 +63,6 @@ static int xen_suspend(void *data)
 	xen_mm_unpin_all();
 
 	sysdev_resume();
-	device_power_up(PMSG_RESUME);
 
 	if (!*cancelled) {
 		xen_irq_resume();
@@ -108,6 +101,12 @@ static void do_suspend(void)
 	/* XXX use normal device tree? */
 	xenbus_suspend();
 
+	err = device_power_down(PMSG_SUSPEND);
+	if (err) {
+		printk(KERN_ERR "device_power_down failed: %d\n", err);
+		goto resume_devices;
+	}
+
 	err = stop_machine(xen_suspend, &cancelled, &cpumask_of_cpu(0));
 	if (err) {
 		printk(KERN_ERR "failed to start xen_suspend: %d\n", err);
@@ -120,6 +119,9 @@ static void do_suspend(void)
 	} else
 		xenbus_suspend_cancel();
 
+	device_power_up(PMSG_RESUME);
+
+resume_devices:
 	device_resume(PMSG_RESUME);
 
 	/* Make sure timer events get retriggered on all CPUs */
kernel/kexec.c +4 -4
Index: linux-2.6/kernel/kexec.c
===================================================================
@@ -1454,7 +1454,6 @@ int kernel_kexec(void)
 		if (error)
 			goto Resume_devices;
 		device_pm_lock();
-		local_irq_disable();
 		/* At this point, device_suspend() has been called,
 		 * but *not* device_power_down(). We *must*
 		 * device_power_down() now.  Otherwise, drivers for
@@ -1464,8 +1463,9 @@ int kernel_kexec(void)
 		 */
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
-			goto Enable_irqs;
+			goto Unlock_pm;
 
+		local_irq_disable();
 		/* Suspend system devices */
 		error = sysdev_suspend(PMSG_FREEZE);
 		if (error)
@@ -1484,9 +1484,9 @@ int kernel_kexec(void)
 	if (kexec_image->preserve_context) {
 		sysdev_resume();
  Power_up_devices:
-		device_power_up(PMSG_RESTORE);
- Enable_irqs:
 		local_irq_enable();
+		device_power_up(PMSG_RESTORE);
+ Unlock_pm:
 		device_pm_unlock();
 		enable_nonboot_cpus();
  Resume_devices:
include/linux/irq.h +1 -0
Index: linux-2.6/include/linux/irq.h
===================================================================
@@ -65,6 +65,7 @@ typedef	void (*irq_flow_handler_t)(unsig
 #define IRQ_SPURIOUS_DISABLED	0x00800000	/* IRQ was disabled by the spurious trap */
 #define IRQ_MOVE_PCNTXT		0x01000000	/* IRQ migration from process context */
 #define IRQ_AFFINITY_SET	0x02000000	/* IRQ affinity was set from userspace*/
+#define IRQ_SUSPENDED		0x04000000	/* IRQ has gone through suspend sequence */
 
 #ifdef CONFIG_IRQ_PER_CPU
 # define CHECK_IRQ_PER_CPU(var) ((var) & IRQ_PER_CPU)
kernel/irq/pm.c new file +78 -0

78 lines not on this page · raw message

kernel/irq/Makefile +1 -0
Index: linux-2.6/kernel/irq/Makefile
===================================================================
@@ -4,3 +4,4 @@ obj-$(CONFIG_GENERIC_IRQ_PROBE) += autop
 obj-$(CONFIG_PROC_FS) += proc.o
 obj-$(CONFIG_GENERIC_PENDING_IRQ) += migration.o
 obj-$(CONFIG_NUMA_MIGRATE_IRQ_DESC) += numa_migrate.o
+obj-$(CONFIG_PM_SLEEP) += pm.o
kernel/irq/manage.c +2 -1
Index: linux-2.6/kernel/irq/manage.c
===================================================================
@@ -222,8 +222,9 @@ static void __enable_irq(struct irq_desc
 		WARN(1, KERN_WARNING "Unbalanced enable for IRQ %d\n", irq);
 		break;
 	case 1: {
-		unsigned int status = desc->status & ~IRQ_DISABLED;
+		unsigned int status;
 
+		status = desc->status & ~(IRQ_DISABLED | IRQ_SUSPENDED);
 		/* Prevent probing on this irq: */
 		desc->status = status | IRQ_NOPROBE;
 		check_irq_resend(desc, irq);
drivers/base/power/main.c +11 -9
Index: linux-2.6/drivers/base/power/main.c
===================================================================
@@ -23,6 +23,7 @@
 #include <linux/pm.h>
 #include <linux/resume-trace.h>
 #include <linux/rwsem.h>
+#include <linux/interrupt.h>
 
 #include "../base.h"
 #include "power.h"
@@ -305,7 +306,8 @@ static int resume_device_noirq(struct de
  *	Execute the appropriate "noirq resume" callback for all devices marked
  *	as DPM_OFF_IRQ.
  *
- *	Must be called with interrupts disabled and only one CPU running.
+ *	Must be called under dpm_list_mtx.  Device drivers should not receive
+ *	interrupts while it's being executed.
  */
 static void dpm_power_up(pm_message_t state)
 {
@@ -326,14 +328,13 @@ static void dpm_power_up(pm_message_t st
  *	device_power_up - Turn on all devices that need special attention.
  *	@state: PM transition of the system being carried out.
  *
- *	Power on system devices, then devices that required we shut them down
- *	with interrupts disabled.
- *
- *	Must be called with interrupts disabled.
+ *	Call the "early" resume handlers and enable device drivers to receive
+ *	interrupts.
  */
 void device_power_up(pm_message_t state)
 {
 	dpm_power_up(state);
+	resume_device_irqs();
 }
 EXPORT_SYMBOL_GPL(device_power_up);
 
@@ -558,16 +559,17 @@ static int suspend_device_noirq(struct d
  *	device_power_down - Shut down special devices.
  *	@state: PM transition of the system being carried out.
  *
- *	Power down devices that require interrupts to be disabled.
- *	Then power down system devices.
+ *	Prevent device drivers from receiving interrupts and call the "late"
+ *	suspend handlers.
  *
- *	Must be called with interrupts disabled and only one CPU running.
+ *	Must be called under dpm_list_mtx.
  */
 int device_power_down(pm_message_t state)
 {
 	struct device *dev;
 	int error = 0;
 
+	suspend_device_irqs();
 	list_for_each_entry_reverse(dev, &dpm_list, power.entry) {
 		error = suspend_device_noirq(dev, state);
 		if (error) {
@@ -577,7 +579,7 @@ int device_power_down(pm_message_t state
 		dev->power.status = DPM_OFF_IRQ;
 	}
 	if (error)
-		dpm_power_up(resume_event(state));
+		device_power_up(resume_event(state));
 	return error;
 }
 EXPORT_SYMBOL_GPL(device_power_down);
drivers/base/sys.c +8 -0
Index: linux-2.6/drivers/base/sys.c
===================================================================
@@ -22,6 +22,7 @@
 #include <linux/pm.h>
 #include <linux/device.h>
 #include <linux/mutex.h>
+#include <linux/interrupt.h>
 
 #include "base.h"
 
@@ -369,6 +370,13 @@ int sysdev_suspend(pm_message_t state)
 	struct sysdev_driver *drv, *err_drv;
 	int ret;
 
+	pr_debug("Checking wake-up interrupts\n");
+
+	/* Return error code if there are any wake-up interrupts pending */
+	ret = check_wakeup_irqs();
+	if (ret)
+		return ret;
+
 	pr_debug("Suspending System Devices\n");
 
 	list_for_each_entry_reverse(cls, &system_kset->list, kset.kobj.entry) {
#168 lore raw

Re: [RFC][PATCH 1/4] PM: Rework handling of interrupts during suspend-resume (rev. 4)

Quoted 25 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to prevent device drivers from getting any interrupts (without disabling interrupts on the CPU) during suspend (or hibernation) and to make them start to receive interrupts again during the subsequent resume, respectively.  These functions make it possible to keep timer interrupts enabled while the "late" suspend and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume.  Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device drivers will be prevented from receiving interrupts, with the help of the new helper function, before their "late" suspend callbacks run (and analogously during resume).

In addition, since the device interrups are now disabled before the CPU has turned all interrupts off and the CPU will ACK the interrupts setting the IRQ_PENDING bit for them, check in sysdev_suspend() if any wake-up interrupts are pending and abort suspend if that's the case.

kernel/irq/pm.c line 62

Quoted 9 lines
kernel/irq/pm.c new file +9 -0
+void resume_device_irqs(void)
+{
+       struct irq_desc *desc;
+       int irq;
+
+       for_each_irq_desc(irq, desc)
+               if (desc->status & IRQ_SUSPENDED)
+                       enable_irq(irq);
+}

I think you need to clear IRQ_SUSPENDED here, not in enable_irq.

kernel/irq/manage.c static void __enable_irq(struct irq_desc · line 230

Quoted 11 lines
kernel/irq/manage.c +2 -1
@@ -222,8 +222,9 @@ static void __enable_irq(struct irq_desc
                WARN(1, KERN_WARNING "Unbalanced enable for IRQ %d\n", irq);
                break;
        case 1: {
-               unsigned int status = desc->status & ~IRQ_DISABLED;
+               unsigned int status;
 
+               status = desc->status & ~(IRQ_DISABLED | IRQ_SUSPENDED);
                /* Prevent probing on this irq: */
                desc->status = status | IRQ_NOPROBE;
                check_irq_resend(desc, irq);

This only clears IRQ_SUSPENDED if the interrupt was not disabled elsewhere. If a driver calls interrupt_disable in suspend_late, but calls interrupt_enable lazily, resume_device_irqs will reenable the interrupt even though the driver has a disable reference.

The rest of the patch looks good.

Signature · 1 line
Arve Hjønnevåg
#169 lore raw

Re: [RFC][PATCH 1/4] PM: Rework handling of interrupts during suspend-resume (rev. 4)

In reply to Rafael J. Wysocki
Quoted 25 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to prevent device drivers from getting any interrupts (without disabling interrupts on the CPU) during suspend (or hibernation) and to make them start to receive interrupts again during the subsequent resume, respectively.  These functions make it possible to keep timer interrupts enabled while the "late" suspend and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume.  Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device drivers will be prevented from receiving interrupts, with the help of the new helper function, before their "late" suspend callbacks run (and analogously during resume).

In addition, since the device interrups are now disabled before the CPU has turned all interrupts off and the CPU will ACK the interrupts setting the IRQ_PENDING bit for them, check in sysdev_suspend() if any wake-up interrupts are pending and abort suspend if that's the case.

kernel/irq/pm.c line 62

Quoted 9 lines
kernel/irq/pm.c new file +9 -0
+void resume_device_irqs(void)
+{
+       struct irq_desc *desc;
+       int irq;
+
+       for_each_irq_desc(irq, desc)
+               if (desc->status & IRQ_SUSPENDED)
+                       enable_irq(irq);
+}

I think you need to clear IRQ_SUSPENDED here, not in enable_irq.

kernel/irq/manage.c static void __enable_irq(struct irq_desc · line 230

Quoted 11 lines
kernel/irq/manage.c +2 -1
@@ -222,8 +222,9 @@ static void __enable_irq(struct irq_desc
                WARN(1, KERN_WARNING "Unbalanced enable for IRQ %d\n", irq);
                break;
        case 1: {
-               unsigned int status = desc->status & ~IRQ_DISABLED;
+               unsigned int status;
 
+               status = desc->status & ~(IRQ_DISABLED | IRQ_SUSPENDED);
                /* Prevent probing on this irq: */
                desc->status = status | IRQ_NOPROBE;
                check_irq_resend(desc, irq);

This only clears IRQ_SUSPENDED if the interrupt was not disabled elsewhere. If a driver calls interrupt_disable in suspend_late, but calls interrupt_enable lazily, resume_device_irqs will reenable the interrupt even though the driver has a disable reference.

The rest of the patch looks good.

Signature · 1 line
Arve Hjønnevåg
#170 lore raw

Re: [RFC][PATCH 1/4] PM: Rework handling of interrupts during suspend-resume (rev. 4)

Quoted 26 lines · Arve Hjønnevåg
Quoted 23 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to prevent device drivers from getting any interrupts (without disabling interrupts on the CPU) during suspend (or hibernation) and to make them start to receive interrupts again during the subsequent resume, respectively. These functions make it possible to keep timer interrupts enabled while the "late" suspend and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device drivers will be prevented from receiving interrupts, with the help of the new helper function, before their "late" suspend callbacks run (and analogously during resume).

In addition, since the device interrups are now disabled before the CPU has turned all interrupts off and the CPU will ACK the interrupts setting the IRQ_PENDING bit for them, check in sysdev_suspend() if any wake-up interrupts are pending and abort suspend if that's the case.

Quoted 11 lines

kernel/irq/pm.c line 62

Quoted 9 lines
kernel/irq/pm.c new file +9 -0
+void resume_device_irqs(void)
+{
+       struct irq_desc *desc;
+       int irq;
+
+       for_each_irq_desc(irq, desc)
+               if (desc->status & IRQ_SUSPENDED)
+                       enable_irq(irq);
+}

I think you need to clear IRQ_SUSPENDED here, not in enable_irq.

enable_irq() clears IRQ_SUSPENDED. This has already been discussed btw.

Quoted 7 lines
Quoted 7 lines
kernel/irq/manage.c +1 -1
@@ -222,8 +222,9 @@ static void __enable_irq(struct irq_desc
               WARN(1, KERN_WARNING "Unbalanced enable for IRQ %d\n", irq);
               break;
       case 1: {
-               unsigned int status = desc->status & ~IRQ_DISABLED;
+               unsigned int status;
 
Quoted 9 lines

kernel/irq/manage.c static void __enable_irq(struct irq_desc · line 230

Quoted 4 lines
kernel/irq/manage.c +1 -0
+               status = desc->status & ~(IRQ_DISABLED | IRQ_SUSPENDED);
               /* Prevent probing on this irq: */
               desc->status = status | IRQ_NOPROBE;
               check_irq_resend(desc, irq);

This only clears IRQ_SUSPENDED if the interrupt was not disabled elsewhere. If a driver calls interrupt_disable in suspend_late, but calls interrupt_enable lazily, resume_device_irqs will reenable the interrupt even though the driver has a disable reference.

Then I'd regard the driver as buggy.

Quoted 1 line

The rest of the patch looks good.

I'm glad you like it.

Thanks,
Rafael

#171 lore raw

Re: [RFC][PATCH 1/4] PM: Rework handling of interrupts during suspend-resume (rev. 4)

In reply to Arve Hjønnevåg
Quoted 26 lines · Arve Hjønnevåg
Quoted 23 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to prevent device drivers from getting any interrupts (without disabling interrupts on the CPU) during suspend (or hibernation) and to make them start to receive interrupts again during the subsequent resume, respectively. These functions make it possible to keep timer interrupts enabled while the "late" suspend and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device drivers will be prevented from receiving interrupts, with the help of the new helper function, before their "late" suspend callbacks run (and analogously during resume).

In addition, since the device interrups are now disabled before the CPU has turned all interrupts off and the CPU will ACK the interrupts setting the IRQ_PENDING bit for them, check in sysdev_suspend() if any wake-up interrupts are pending and abort suspend if that's the case.

Quoted 11 lines

kernel/irq/pm.c line 62

Quoted 9 lines
kernel/irq/pm.c new file +9 -0
+void resume_device_irqs(void)
+{
+       struct irq_desc *desc;
+       int irq;
+
+       for_each_irq_desc(irq, desc)
+               if (desc->status & IRQ_SUSPENDED)
+                       enable_irq(irq);
+}

I think you need to clear IRQ_SUSPENDED here, not in enable_irq.

enable_irq() clears IRQ_SUSPENDED. This has already been discussed btw.

Quoted 7 lines
Quoted 7 lines
kernel/irq/manage.c +1 -1
@@ -222,8 +222,9 @@ static void __enable_irq(struct irq_desc
               WARN(1, KERN_WARNING "Unbalanced enable for IRQ %d\n", irq);
               break;
       case 1: {
-               unsigned int status = desc->status & ~IRQ_DISABLED;
+               unsigned int status;
 
Quoted 9 lines

kernel/irq/manage.c static void __enable_irq(struct irq_desc · line 230

Quoted 4 lines
kernel/irq/manage.c +1 -0
+               status = desc->status & ~(IRQ_DISABLED | IRQ_SUSPENDED);
               /* Prevent probing on this irq: */
               desc->status = status | IRQ_NOPROBE;
               check_irq_resend(desc, irq);

This only clears IRQ_SUSPENDED if the interrupt was not disabled elsewhere. If a driver calls interrupt_disable in suspend_late, but calls interrupt_enable lazily, resume_device_irqs will reenable the interrupt even though the driver has a disable reference.

Then I'd regard the driver as buggy.

Quoted 1 line

The rest of the patch looks good.

I'm glad you like it.

Thanks,
Rafael

#172 lore raw

Re: [RFC][PATCH 1/4] PM: Rework handling of interrupts during suspend-resume (rev. 4)

Quoted 39 lines · Rafael J. Wysocki
Quoted 26 lines · Arve Hjønnevåg
Quoted 23 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Introduce two helper functions allowing us to prevent device drivers from getting any interrupts (without disabling interrupts on the CPU) during suspend (or hibernation) and to make them start to receive interrupts again during the subsequent resume, respectively.  These functions make it possible to keep timer interrupts enabled while the "late" suspend and "early" resume callbacks provided by device drivers are being executed.

Use these functions to rework the handling of interrupts during suspend (hibernation) and resume.  Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device drivers will be prevented from receiving interrupts, with the help of the new helper function, before their "late" suspend callbacks run (and analogously during resume).

In addition, since the device interrups are now disabled before the CPU has turned all interrupts off and the CPU will ACK the interrupts setting the IRQ_PENDING bit for them, check in sysdev_suspend() if any wake-up interrupts are pending and abort suspend if that's the case.

Quoted 11 lines

kernel/irq/pm.c line 62

Quoted 9 lines
kernel/irq/pm.c new file +9 -0
+void resume_device_irqs(void)
+{
+       struct irq_desc *desc;
+       int irq;
+
+       for_each_irq_desc(irq, desc)
+               if (desc->status & IRQ_SUSPENDED)
+                       enable_irq(irq);
+}

I think you need to clear IRQ_SUSPENDED here, not in enable_irq.

Quoted 2 lines

enable_irq() clears IRQ_SUSPENDED.  This has already been discussed btw.

I'm if I missed that discussion, but enable_irq cannot know who is calling it and therefore cannot know if IRQ_SUSPENDED should be cleared.

Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
kernel/irq/manage.c +1 -1
@@ -222,8 +222,9 @@ static void __enable_irq(struct irq_desc
                WARN(1, KERN_WARNING "Unbalanced enable for IRQ %d\n", irq);
                break;
        case 1: {
-               unsigned int status = desc->status & ~IRQ_DISABLED;
+               unsigned int status;
 
Quoted 9 lines

kernel/irq/manage.c static void __enable_irq(struct irq_desc · line 230

Quoted 4 lines
kernel/irq/manage.c +1 -0
+               status = desc->status & ~(IRQ_DISABLED | IRQ_SUSPENDED);
                /* Prevent probing on this irq: */
                desc->status = status | IRQ_NOPROBE;
                check_irq_resend(desc, irq);

This only clears IRQ_SUSPENDED if the interrupt was not disabled elsewhere. If a driver calls interrupt_disable in suspend_late, but calls interrupt_enable lazily, resume_device_irqs will reenable the interrupt even though the driver has a disable reference.

Quoted 1 line

Then I'd regard the driver as buggy.

The bug is not in the driver. The driver called disable_irq once. You called disable_irq once, but enable_irq twice.

Signature · 1 line
Arve Hjønnevåg