Skip to contentSkip to conversation

PCI PM: Restore standard config registers of all devices early

DISCUSSION Benjamin Herrenschmidt 10 participants 98 messages Feb 2, 2009 – Mar 5, 2009 1 message not in the archive

Subsystems (10)
lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 9 lines · Linux Kernel Mailing List
Gitweb:     http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=aa8c6c93747f7b55fa11e1624fec8ca33763a805
Commit:     aa8c6c93747f7b55fa11e1624fec8ca33763a805
Parent:     0db29af1e767464d71b89410d61a1e5b668d0370
Author:     Rafael J. Wysocki <[email protected]>
AuthorDate: Fri Jan 16 21:54:43 2009 +0100
Committer:  Jesse Barnes <[email protected]>
CommitDate: Fri Jan 16 12:57:58 2009 -0800

PCI PM: Restore standard config registers of all devices early

This seems to break resume on some PowerBooks (just reported by Andreas).

I'm not yet sure what's up, but I can imagine various problems with the approach. For one, pci_enable_device() is the only place where the arch has a hook to turn back on things like clocks etc... to a device. You aren't supposed to touch at the config space before the arch got a chance to turn things back on. I agree there's somewhat a chicken and egg problem here, so we might need to look more closely at what that restore thing does and maybe add a call to the arch low level enable hook first... Or create a new hook.

I don't know how x86 does but I'm sure there must be some kind ACPI thingy that must be called too before you can touch a device, in case it got powered off by more than just the standard D states (ie, clock stopped on the bus or whole power plane switched off).

I'll try to reproduce and think about it more, but it looks to me that this patch might not be quite the right approach yet.

Cheers,
Ben.

Quoted 364 lines

There is a problem in our handling of suspend-resume of PCI devices that many of them have their standard config registers restored with interrupts enabled and they are put into the full power state with interrupts enabled as well. This may lead to the following scenario:
* an interrupt vector is shared between two or more devices
* one device is resumed earlier and generates an interrupt
* the interrupt handler of another device tries to handle it and attempts to access the device the config space of which hasn't been restored yet and/or which still is in a low power state
* the system crashes as a result

To prevent this from happening we should restore the standard configuration registers of all devices with interrupts disabled and we should put them into the D0 power state right after that. Unfortunately, this cannot be done using the existing pci_set_power_state(), because it can sleep. Also, to do it we have to make sure that the config spaces of all devices were actually saved during suspend.

---
 drivers/pci/pci-driver.c |   91 ++++++++++++++--------------------------------
 drivers/pci/pci.c        |   63 +++++++++++++++++++++++++++++---
 drivers/pci/pci.h        |    6 +++
 include/linux/pci.h      |    5 +++
 4 files changed, 95 insertions(+), 70 deletions(-)
drivers/pci/pci-driver.c +27 -64
index c697f26..9de07b7 100644
@@ -355,17 +355,27 @@ static int pci_legacy_suspend(struct device *dev, pm_message_t state)
 	int i = 0;
 
 	if (drv && drv->suspend) {
+		pci_dev->state_saved = false;
+
 		i = drv->suspend(pci_dev, state);
 		suspend_report_result(drv->suspend, i);
-	} else {
-		pci_save_state(pci_dev);
-		/*
-		 * This is for compatibility with existing code with legacy PM
-		 * support.
-		 */
-		pci_pm_set_unknown_state(pci_dev);
+		if (i)
+			return i;
+
+		if (pci_dev->state_saved)
+			goto Fixup;
+
+		if (WARN_ON_ONCE(pci_dev->current_state != PCI_D0))
+			goto Fixup;
 	}
 
+	pci_save_state(pci_dev);
+	/*
+	 * This is for compatibility with existing code with legacy PM support.
+	 */
+	pci_pm_set_unknown_state(pci_dev);
+
+ Fixup:
 	pci_fixup_device(pci_fixup_suspend, pci_dev);
 
 	return i;
@@ -386,81 +396,34 @@ static int pci_legacy_suspend_late(struct device *dev, pm_message_t state)
 
 static int pci_legacy_resume_early(struct device *dev)
 {
-	int error = 0;
 	struct pci_dev * pci_dev = to_pci_dev(dev);
 	struct pci_driver * drv = pci_dev->driver;
 
-	pci_fixup_device(pci_fixup_resume_early, pci_dev);
-
-	if (drv && drv->resume_early)
-		error = drv->resume_early(pci_dev);
-	return error;
+	return drv && drv->resume_early ?
+			drv->resume_early(pci_dev) : 0;
 }
 
 static int pci_legacy_resume(struct device *dev)
 {
-	int error;
 	struct pci_dev * pci_dev = to_pci_dev(dev);
 	struct pci_driver * drv = pci_dev->driver;
 
 	pci_fixup_device(pci_fixup_resume, pci_dev);
 
-	if (drv && drv->resume) {
-		error = drv->resume(pci_dev);
-	} else {
-		/* restore the PCI config space */
-		pci_restore_state(pci_dev);
-		error = pci_pm_reenable_device(pci_dev);
-	}
-	return error;
+	return drv && drv->resume ?
+			drv->resume(pci_dev) : pci_pm_reenable_device(pci_dev);
 }
 
 /* Auxiliary functions used by the new power management framework */
 
-static int pci_restore_standard_config(struct pci_dev *pci_dev)
-{
-	struct pci_dev *parent = pci_dev->bus->self;
-	int error = 0;
-
-	/* Check if the device's bus is operational */
-	if (!parent || parent->current_state == PCI_D0) {
-		pci_restore_state(pci_dev);
-		pci_update_current_state(pci_dev, PCI_D0);
-	} else {
-		dev_warn(&pci_dev->dev, "unable to restore config, "
-			"bridge %s in low power state D%d\n", pci_name(parent),
-			parent->current_state);
-		pci_dev->current_state = PCI_UNKNOWN;
-		error = -EAGAIN;
-	}
-
-	return error;
-}
-
-static bool pci_is_bridge(struct pci_dev *pci_dev)
-{
-	return !!(pci_dev->subordinate);
-}
-
 static void pci_pm_default_resume_noirq(struct pci_dev *pci_dev)
 {
-	if (pci_restore_standard_config(pci_dev))
-		pci_fixup_device(pci_fixup_resume_early, pci_dev);
+	pci_restore_standard_config(pci_dev);
+	pci_fixup_device(pci_fixup_resume_early, pci_dev);
 }
 
 static int pci_pm_default_resume(struct pci_dev *pci_dev)
 {
-	/*
-	 * pci_restore_standard_config() should have been called once already,
-	 * but it would have failed if the device's parent bridge had not been
-	 * in power state D0 at that time.  Check it and try again if necessary.
-	 */
-	if (pci_dev->current_state == PCI_UNKNOWN) {
-		int error = pci_restore_standard_config(pci_dev);
-		if (error)
-			return error;
-	}
-
 	pci_fixup_device(pci_fixup_resume, pci_dev);
 
 	if (!pci_is_bridge(pci_dev))
@@ -575,11 +538,11 @@ static int pci_pm_resume_noirq(struct device *dev)
 	struct device_driver *drv = dev->driver;
 	int error = 0;
 
+	pci_pm_default_resume_noirq(pci_dev);
+
 	if (pci_has_legacy_pm_support(pci_dev))
 		return pci_legacy_resume_early(dev);
 
-	pci_pm_default_resume_noirq(pci_dev);
-
 	if (drv && drv->pm && drv->pm->resume_noirq)
 		error = drv->pm->resume_noirq(dev);
 
@@ -730,11 +693,11 @@ static int pci_pm_restore_noirq(struct device *dev)
 	struct device_driver *drv = dev->driver;
 	int error = 0;
 
+	pci_pm_default_resume_noirq(pci_dev);
+
 	if (pci_has_legacy_pm_support(pci_dev))
 		return pci_legacy_resume_early(dev);
 
-	pci_pm_default_resume_noirq(pci_dev);
-
 	if (drv && drv->pm && drv->pm->restore_noirq)
 		error = drv->pm->restore_noirq(dev);
 
drivers/pci/pci.c +57 -6
index e491fde..17bd932 100644
@@ -22,7 +22,7 @@
 #include <asm/dma.h>	/* isa_dma_bridge_buggy */
 #include "pci.h"
 
-unsigned int pci_pm_d3_delay = 10;
+unsigned int pci_pm_d3_delay = PCI_PM_D3_WAIT;
 
 #ifdef CONFIG_PCI_DOMAINS
 int pci_domains_supported = 1;
@@ -426,6 +426,7 @@ static inline int platform_pci_sleep_wake(struct pci_dev *dev, bool enable)
  *                           given PCI device
  * @dev: PCI device to handle.
  * @state: PCI power state (D0, D1, D2, D3hot) to put the device into.
+ * @wait: If 'true', wait for the device to change its power state
  *
  * RETURN VALUE:
  * -EINVAL if the requested state is invalid.
@@ -435,7 +436,7 @@ static inline int platform_pci_sleep_wake(struct pci_dev *dev, bool enable)
  * 0 if device's power state has been successfully changed.
  */
 static int
-pci_raw_set_power_state(struct pci_dev *dev, pci_power_t state)
+pci_raw_set_power_state(struct pci_dev *dev, pci_power_t state, bool wait)
 {
 	u16 pmcsr;
 	bool need_restore = false;
@@ -480,8 +481,10 @@ pci_raw_set_power_state(struct pci_dev *dev, pci_power_t state)
 		break;
 	case PCI_UNKNOWN: /* Boot-up */
 		if ((pmcsr & PCI_PM_CTRL_STATE_MASK) == PCI_D3hot
-		 && !(pmcsr & PCI_PM_CTRL_NO_SOFT_RESET))
+		 && !(pmcsr & PCI_PM_CTRL_NO_SOFT_RESET)) {
 			need_restore = true;
+			wait = true;
+		}
 		/* Fall-through: force to D0 */
 	default:
 		pmcsr = 0;
@@ -491,12 +494,15 @@ pci_raw_set_power_state(struct pci_dev *dev, pci_power_t state)
 	/* enter specified state */
 	pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, pmcsr);
 
+	if (!wait)
+		return 0;
+
 	/* Mandatory power management transition delays */
 	/* see PCI PM 1.1 5.6.1 table 18 */
 	if (state == PCI_D3hot || dev->current_state == PCI_D3hot)
 		msleep(pci_pm_d3_delay);
 	else if (state == PCI_D2 || dev->current_state == PCI_D2)
-		udelay(200);
+		udelay(PCI_PM_D2_DELAY);
 
 	dev->current_state = state;
 
@@ -515,7 +521,7 @@ pci_raw_set_power_state(struct pci_dev *dev, pci_power_t state)
 	if (need_restore)
 		pci_restore_bars(dev);
 
-	if (dev->bus->self)
+	if (wait && dev->bus->self)
 		pcie_aspm_pm_state_change(dev->bus->self);
 
 	return 0;
@@ -585,7 +591,7 @@ int pci_set_power_state(struct pci_dev *dev, pci_power_t state)
 	if (state == PCI_D3hot && (dev->dev_flags & PCI_DEV_FLAGS_NO_D3))
 		return 0;
 
-	error = pci_raw_set_power_state(dev, state);
+	error = pci_raw_set_power_state(dev, state, true);
 
 	if (state > PCI_D0 && platform_pci_power_manageable(dev)) {
 		/* Allow the platform to finalize the transition */
@@ -730,6 +736,7 @@ pci_save_state(struct pci_dev *dev)
 	/* XXX: 100% dword access ok here? */
 	for (i = 0; i < 16; i++)
 		pci_read_config_dword(dev, i * 4,&dev->saved_config_space[i]);
+	dev->state_saved = true;
 	if ((i = pci_save_pcie_state(dev)) != 0)
 		return i;
 	if ((i = pci_save_pcix_state(dev)) != 0)
@@ -1374,6 +1381,50 @@ void pci_allocate_cap_save_buffers(struct pci_dev *dev)
 }
 
 /**
+ * pci_restore_standard_config - restore standard config registers of PCI device
+ * @dev: PCI device to handle
+ *
+ * This function assumes that the device's configuration space is accessible.
+ * If the device needs to be powered up, the function will wait for it to
+ * change the state.
+ */
+int pci_restore_standard_config(struct pci_dev *dev)
+{
+	pci_power_t prev_state;
+	int error;
+
+	pci_restore_state(dev);
+	pci_update_current_state(dev, PCI_D0);
+
+	prev_state = dev->current_state;
+	if (prev_state == PCI_D0)
+		return 0;
+
+	error = pci_raw_set_power_state(dev, PCI_D0, false);
+	if (error)
+		return error;
+
+	if (pci_is_bridge(dev)) {
+		if (prev_state > PCI_D1)
+			mdelay(PCI_PM_BUS_WAIT);
+	} else {
+		switch(prev_state) {
+		case PCI_D3cold:
+		case PCI_D3hot:
+			mdelay(pci_pm_d3_delay);
+			break;
+		case PCI_D2:
+			udelay(PCI_PM_D2_DELAY);
+			break;
+		}
+	}
+
+	dev->current_state = PCI_D0;
+
+	return 0;
+}
+
+/**
  * pci_enable_ari - enable ARI forwarding if hardware support it
  * @dev: the PCI device
  */
drivers/pci/pci.h +6 -0
index 1351bb4..26ddf78 100644
@@ -49,6 +49,12 @@ extern void pci_disable_enabled_device(struct pci_dev *dev);
 extern void pci_pm_init(struct pci_dev *dev);
 extern void platform_pci_wakeup_init(struct pci_dev *dev);
 extern void pci_allocate_cap_save_buffers(struct pci_dev *dev);
+extern int pci_restore_standard_config(struct pci_dev *dev);
+
+static inline bool pci_is_bridge(struct pci_dev *pci_dev)
+{
+	return !!(pci_dev->subordinate);
+}
 
 extern int pci_user_read_config_byte(struct pci_dev *dev, int where, u8 *val);
 extern int pci_user_read_config_word(struct pci_dev *dev, int where, u16 *val);
include/linux/pci.h +5 -1
index 80f8b8b..48890cf 100644
@@ -117,6 +117,10 @@ typedef int __bitwise pci_power_t;
 #define PCI_UNKNOWN	((pci_power_t __force) 5)
 #define PCI_POWER_ERROR	((pci_power_t __force) -1)
 
+#define PCI_PM_D2_DELAY	200
+#define PCI_PM_D3_WAIT	10
+#define PCI_PM_BUS_WAIT	50
+
 /** The pci_channel state describes connectivity between the CPU and
  *  the pci device.  If some PCI bus between here and the pci device
  *  has crashed or locked up, this info is reflected here.
@@ -252,6 +256,7 @@ struct pci_dev {
 	unsigned int	ari_enabled:1;	/* ARI forwarding */
 	unsigned int	is_managed:1;
 	unsigned int	is_pcie:1;
+	unsigned int	state_saved:1;
 	pci_dev_flags_t dev_flags;
 	atomic_t	enable_cnt;	/* pci_enable_device has been called */
 
--

To unsubscribe from this list: send the line "unsubscribe git-commits-head" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 5 lines · Benjamin Herrenschmidt

I don't know how x86 does but I'm sure there must be some kind ACPI thingy that must be called too before you can touch a device, in case it got powered off by more than just the standard D states (ie, clock stopped on the bus or whole power plane switched off).

It gets called later in that case.

Quoted 2 lines

I'll try to reproduce and think about it more, but it looks to me that this patch might not be quite the right approach yet.

I think you'll need to move the clock gating thing to be a sysdev suspend/resume event, which gets done really early along with things like core timekeeping etc. We kind of hit that already with the USB driver, where CONFIG_PPC does all kinds of wrong things:

	#ifdef CONFIG_PPC_PMAC
	        if (retval == 0) {
	                /* Disable ASIC clocks for USB */
	                if (machine_is(powermac)) {
	                        struct device_node      *of_node;

	                        of_node = pci_device_to_OF_node(dev);
	                        if (of_node)
	                                pmac_call_feature(PMAC_FTR_USB_ENABLE,
	                                                        of_node, 0, 0);
	                }
	        }
	#endif

inside the hcd-pci.c driver. If that whole thing was a sysdev feature, you wouldn't need that kind of insane "do my own arch-specific thing in a generic driver" thing. AND waking it up would work too.

I'm assuming this is exactly the kind of thing that is now biting you?

Linus

#98 lore raw
In reply to Benjamin Herrenschmidt
Quoted 9 lines · Benjamin Herrenschmidt
Quoted 8 lines · Linux Kernel Mailing List
Gitweb:     http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=aa8c6c93747f7b55fa11e1624fec8ca33763a805
Commit:     aa8c6c93747f7b55fa11e1624fec8ca33763a805
Parent:     0db29af1e767464d71b89410d61a1e5b668d0370
Author:     Rafael J. Wysocki <[email protected]>
AuthorDate: Fri Jan 16 21:54:43 2009 +0100
Committer:  Jesse Barnes <[email protected]>
CommitDate: Fri Jan 16 12:57:58 2009 -0800
Quoted 4 lines
Quoted 1 line

PCI PM: Restore standard config registers of all devices early

This seems to break resume on some PowerBooks (just reported by Andreas).

Yes, I saw the report in Bugzilla.

Quoted 8 lines

I'm not yet sure what's up, but I can imagine various problems with the approach. For one, pci_enable_device() is the only place where the arch has a hook to turn back on things like clocks etc... to a device. You aren't supposed to touch at the config space before the arch got a chance to turn things back on. I agree there's somewhat a chicken and egg problem here, so we might need to look more closely at what that restore thing does and maybe add a call to the arch low level enable hook first... Or create a new hook.

I'd prefer to create a new hook, althouth there may be a problem with ACPI vs interrupts off. In the meantime, I'd like to test some recent fixes on top of this patch and try a couple of debug patches to see what's up (in the Bugzilla if that's not a problem).

Quoted 7 lines

I don't know how x86 does but I'm sure there must be some kind ACPI thingy that must be called too before you can touch a device, in case it got powered off by more than just the standard D states (ie, clock stopped on the bus or whole power plane switched off).

I'll try to reproduce and think about it more, but it looks to me that this patch might not be quite the right approach yet.

Yeah.

Well, the approach is a bit x86-ish ...

Thanks,
Rafael

#3 lore raw
In reply to Linus Torvalds
Quoted 5 lines

I think you'll need to move the clock gating thing to be a sysdev suspend/resume event, which gets done really early along with things like core timekeeping etc. We kind of hit that already with the USB driver, where CONFIG_PPC does all kinds of wrong things:

But I want the gating to be tied to the device. IE. If the device is ever suspended on its own, or the driver removed, I want the clocks off. Ideally, I want USB autosuspend to be able to clock gate or that sort of thing ..

I'd rather hook it up inside pci_set_power_state()...

Quoted 5 lines

inside the hcd-pci.c driver. If that whole thing was a sysdev feature, you wouldn't need that kind of insane "do my own arch-specific thing in a generic driver" thing. AND waking it up would work too.

I'm assuming this is exactly the kind of thing that is now biting you?

I don't know yet what is biting us, though I suppose so. I only saw Andreas report before going to bed yesterday ;-)

I wouldn't need a sysdev if I was going to put the gating under arch control, I could just do that from various arch code I already have there doing bits and pieces. But I like it being tied to the driver, it makes more sense in many ways to have a driver control the clocks of its device. Maybe the best approach is to stick a hook into pci_set_power_state() ... This should really be the very first thing to happen, even before whacking back the BARs.

Cheers,
Ben.

#7 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 4 lines

I don't know how x86 does but I'm sure there must be some kind ACPI thingy that must be called too before you can touch a device, in case it got powered off by more than just the standard D states (ie, clock stopped on the bus or whole power plane switched off).

It gets called later in that case.

And won't you have a potential problem here if ACPI is doing clock gating or turning off the whole power plane ? IE. If that happens, your pci_restore_state() done early with interrupts off will fail as well...

Since the root of that problem seems to be related to interrupts, maybe the right approach is to have drivers disable it on suspend (ie, disable_irq -> disabled at PIC level) and thus it would only get re-enabled when all the drivers on a shared line have re-enabled it... but that would mean drivers have to know they can not synchronously wait for an interrupt from the device to happen at resume() time ...

It's not a trivial problem...

Ben.

#4 lore raw
In reply to Benjamin Herrenschmidt
Quoted 2 lines · Benjamin Herrenschmidt

I'd rather hook it up inside pci_set_power_state()...

Umm. But you already _have_ that. Look at platform_pci_set_power_state(). It gets called before turning the device on (pci_raw_set_power_state(D0)) and after turning the device off (pci_raw_set_power_state(D1+))

Maybe you missed it? It's how ACPI does things.

However, the issue you see is that pci_restore_standard_config() doesn't call it, because at least with ACPI, the ACPI code simply isn't ready to be called with interrupts off. So it looks like you may be looking at the wrong thing, hmm?

Linus

#8 lore raw
In reply to Benjamin Herrenschmidt
Quoted 4 lines · Benjamin Herrenschmidt

And won't you have a potential problem here if ACPI is doing clock gating or turning off the whole power plane ? IE. If that happens, your pci_restore_state() done early with interrupts off will fail as well...

Well, it won't fail any worse than it does anyway.

Quoted 3 lines

Since the root of that problem seems to be related to interrupts, maybe the right approach is to have drivers disable it on suspend (ie, disable_irq -> disabled at PIC level)

No.

We went through this, you apparently didn't follow it.

You cannot disable_irq() at suspend/resume time, because that means that when you have shared interrupts the device can no longer use interrupts in those paths - because they may be disabled by totally different devices.

And a lot of devices WILL NOT WORK without interrupts. Including suspend/resume events. Think something as common as USB.

Quoted 1 line

It's not a trivial problem...

That's the understatement of the year.

The whole reason we want to do the two-phase commit thing where "suspend()" starts the thing and "suspend_late()" finalizes things is exactly all about these inter-connections and interrupts in particular.

I suspect that we could possibly make ACPI happy by actually leaving interrupts "enabled" in the suspend-late (and early-resume) paths, but with all hardware interrupts actually turned off. But that's really just a "let's fool people by turning off interrupts a different way" thing - it in no way really changes any fundamental issues.

Whether you use "disable_irq() over all interrupts" or "local_irq_save -> local_irq_restore" really doesn't change anything. You cannot do this in a single phase, because that means that you randomly disable interrupts too early (or enable them too late) when drivers still _require_ them.

So the only workable way to handle interrupts is one of two:

- the two-phase thing we do. Do a first phase with interrupts enabled, then the actual low-level "turn off" with interrupts disabled (and the reverse on resume), so that device drivers never have to see the case of "interrupt happens with dead device", while still having the _guarantee_ that interrupts work for part of their suspend/resume cycle.

- expecting all drivers to be perfect and handle interrupts correcly in the driver.

Quite frankly, I don't think the second one is workable. It may be the optimal one in theory, but it's never worked for us in practice.

Linus

#9 lore raw
Quoted 6 lines · Linus Torvalds

I suspect that we could possibly make ACPI happy by actually leaving interrupts "enabled" in the suspend-late (and early-resume) paths, but with all hardware interrupts actually turned off. But that's really just a "let's fool people by turning off interrupts a different way" thing - it in no way really changes any fundamental issues.

Btw, I do think that we can make ACPI happy regardless.

We quite commonly call into ACPI during the early boot sequence, when interrupts are disabled for all the same reasons. We don't get the "might_sleep()" warnings, simply because we have these kinds of checks:

	..
        if ((!in_atomic() && !irqs_disabled()) ||
                    system_state != SYSTEM_RUNNING || oops_in_progress)
		return;
	..

ie we know that "system_state != SYSTEM_RUNNING" is a special case where things are allowed to do things that they aren't normally allowed to do.

I suspect that late-suspend/early-resume is just exactly the same. It's a boot, after all. Just let ACPI do its odd things, despite the fact that interrupts are disabled. The fact that doing them while the system is _running_ is invalid doesn't necessarily mean that it is invalid under bootup or suspend/resume.

We might even make it possible to have timers going, if we end up saying "we'll mask all hardware interrupts _except_ for the timer". I'm not sure that is necessarily something we can do portably, though..

Linus

#5 lore raw
In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 2 lines · Benjamin Herrenschmidt

I'd rather hook it up inside pci_set_power_state()...

Umm. But you already _have_ that. Look at platform_pci_set_power_state(). It gets called before turning the device on (pci_raw_set_power_state(D0)) and after turning the device off (pci_raw_set_power_state(D1+))

Maybe you missed it? It's how ACPI does things.

I may well have, it didn't exist when my stuff was written.

Quoted 4 lines

However, the issue you see is that pci_restore_standard_config() doesn't call it, because at least with ACPI, the ACPI code simply isn't ready to be called with interrupts off. So it looks like you may be looking at the wrong thing, hmm?

Well.. yes and no... I still don't see how it can be sane to whack config space back before the device has been turned back on. I wouldn't be surprised if some devices don't grok well their BARs being written while not in D0.

I suppose I can always add another hook inside pci_restore_blah for my specific case but I can also very easily see that can of worms hitting even more badly in tight embedded environments such as handhelds, who use things like fine clock control a lot more extensively.

In fact, even on x86, I'm not sure it's kosher to restore the config space before you called ACPI...

Ben.

#86 lore raw
In reply to Linus Torvalds
Quoted 6 lines

I suspect that we could possibly make ACPI happy by actually leaving interrupts "enabled" in the suspend-late (and early-resume) paths, but with all hardware interrupts actually turned off. But that's really just a "let's fool people by turning off interrupts a different way" thing - it in no way really changes any fundamental issues.

Well, it might be the right approach ... for the simple reason that ACPI doesn't -really- need external interrupts off ... it's a side effect of the interpreter using mutexes etc...

So maybe indeed we should look into doing something like that, in fact, I quite like it:

* suspend:

- suspend devices with IRQs on
- suspend PICs (ie, mask all interrupts). the main kernel clock source should be kept running tho
- do the device suspend "late" thing involving ACPI etc..

* resume:

the other way around.

Do we still need a real suspend "late" with hard IRQs off in that scenario ?

Granted, because we are still effectively scheduling etc... we might have the driver being hit by requests etc... so the suspend_late in the above setup loses the property of being shielded against that...

Quoted 4 lines

Whether you use "disable_irq() over all interrupts" or "local_irq_save -> local_irq_restore" really doesn't change anything. You cannot do this in a single phase, because that means that you randomly disable interrupts too early (or enable them too late) when drivers still _require_ them.

Well, sort-of. IE. local_irq_save() vs. disable_irq() has a relevant difference in our context ... the ability to use mutexes, msleep, etc... which may allow ACPI to operate.

I don't care about ACPI on powerpc, so I'm happy hooking the gating with irqs off ... but the existing hook isn't in the right place imho.

Quoted 2 lines

Quite frankly, I don't think the second one is workable. It may be the optimal one in theory, but it's never worked for us in practice.

True, it's going to be a pain. However, I still think you're going to be bitten somewhere if you whack config space back before you called the appropriate ACPI magic to resume the device, but I may be wrong, maybe nobody will ever implement really aggressive power management in their AML ? :-)

Cheers,
Ben.

#10 lore raw
In reply to Linus Torvalds
Quoted 3 lines

I suspect that late-suspend/early-resume is just exactly the same. It's a boot, after all.

I agree. We know we have turned off all other CPUs, we can't race, etc...

Quoted 8 lines

Just let ACPI do its odd things, despite the fact that interrupts are disabled. The fact that doing them while the system is _running_ is invalid doesn't necessarily mean that it is invalid under bootup or suspend/resume.

We might even make it possible to have timers going, if we end up saying "we'll mask all hardware interrupts _except_ for the timer". I'm not sure that is necessarily something we can do portably, though..

Dunno. I can do it for powerpc easily enough but it's hard to tell for others. The problem with timers running is that it means a slight change of semantics for drivers in the sense that suspend_late now -might- be interrupted by some request hitting the driver from interrupt time ... ie, we had the property that suspend_late was not going to bother with any exclusion vs. normal processing code path, but with timers on, that's no longer the case...

I prefer the hack to get ACPI working early. Sounds saner to me in a wicked way :-) And being able to restore the device via ACPI before touching its config space -at-all- in the kernel side might end up making things more robust for us .. who knows what those AML do tho...

BTW. Somebody happens to know what Windows does in that area ? Does it restore config space ?

Cheers,
Ben.

#6 lore raw
In reply to Benjamin Herrenschmidt
Quoted 23 lines · Benjamin Herrenschmidt
Quoted 10 lines · Linus Torvalds
Quoted 2 lines · Benjamin Herrenschmidt

I'd rather hook it up inside pci_set_power_state()...

Umm. But you already _have_ that. Look at platform_pci_set_power_state(). It gets called before turning the device on (pci_raw_set_power_state(D0)) and after turning the device off (pci_raw_set_power_state(D1+))

Maybe you missed it? It's how ACPI does things.

I may well have, it didn't exist when my stuff was written.

Quoted 4 lines

However, the issue you see is that pci_restore_standard_config() doesn't call it, because at least with ACPI, the ACPI code simply isn't ready to be called with interrupts off. So it looks like you may be looking at the wrong thing, hmm?

Well.. yes and no... I still don't see how it can be sane to whack config space back before the device has been turned back on. I wouldn't be surprised if some devices don't grok well their BARs being written while not in D0.

Quoted 7 lines

I suppose I can always add another hook inside pci_restore_blah for my specific case but I can also very easily see that can of worms hitting even more badly in tight embedded environments such as handhelds, who use things like fine clock control a lot more extensively.

In fact, even on x86, I'm not sure it's kosher to restore the config space before you called ACPI...

It need not be in theory, but I haven't seen any real life example of that.

I think devices without the PM capability are suspicious, so to speak.

Thanks,
Rafael

#11 lore raw
In reply to Linus Torvalds
Quoted 15 lines · Linus Torvalds
Quoted 6 lines · Linus Torvalds

I suspect that we could possibly make ACPI happy by actually leaving interrupts "enabled" in the suspend-late (and early-resume) paths, but with all hardware interrupts actually turned off. But that's really just a "let's fool people by turning off interrupts a different way" thing - it in no way really changes any fundamental issues.

Btw, I do think that we can make ACPI happy regardless.

We quite commonly call into ACPI during the early boot sequence, when interrupts are disabled for all the same reasons. We don't get the "might_sleep()" warnings, simply because we have these kinds of checks:

Quoted 8 lines
	..
        if ((!in_atomic() && !irqs_disabled()) ||
                    system_state != SYSTEM_RUNNING || oops_in_progress)
		return;
	..

ie we know that "system_state != SYSTEM_RUNNING" is a special case where things are allowed to do things that they aren't normally allowed to do.

Yes, which is why I thought it might be a good idea to make the AML interpreter allow is to execute AML with interrupts off, so that we can put devices into low power states and/or put them into D0 with interrupts off.

Then, we'll be able to make ACPI happy (up to some strange ordering expectations of some insane BIOSes maybe) and fix the interrupts issue at the same time.

The idea would be to have a special code path(s) where AML can be executed with interrupts off and a couple of special entry points into the AML interpreter for this purpose.

Thanks,
Rafael

#87 lore raw
In reply to Benjamin Herrenschmidt
Quoted 14 lines · Benjamin Herrenschmidt
Quoted 6 lines

I suspect that we could possibly make ACPI happy by actually leaving interrupts "enabled" in the suspend-late (and early-resume) paths, but with all hardware interrupts actually turned off. But that's really just a "let's fool people by turning off interrupts a different way" thing - it in no way really changes any fundamental issues.

Well, it might be the right approach ... for the simple reason that ACPI doesn't -really- need external interrupts off ... it's a side effect of the interpreter using mutexes etc...

So maybe indeed we should look into doing something like that, in fact, I quite like it:

Quoted 6 lines

* suspend:

- suspend devices with IRQs on
- suspend PICs (ie, mask all interrupts). the main kernel clock source should be kept running tho
- do the device suspend "late" thing involving ACPI etc..

I think it would be easier to make ACPI allow us to run AML with interrupts off.

Quoted 10 lines

* resume:

the other way around.

Do we still need a real suspend "late" with hard IRQs off in that scenario ?

Granted, because we are still effectively scheduling etc... we might have the driver being hit by requests etc... so the suspend_late in the above setup loses the property of being shielded against that...

Well, I don't think we'd need it in this case.

Quoted 12 lines
Quoted 4 lines

Whether you use "disable_irq() over all interrupts" or "local_irq_save -> local_irq_restore" really doesn't change anything. You cannot do this in a single phase, because that means that you randomly disable interrupts too early (or enable them too late) when drivers still _require_ them.

Well, sort-of. IE. local_irq_save() vs. disable_irq() has a relevant difference in our context ... the ability to use mutexes, msleep, etc... which may allow ACPI to operate.

I don't care about ACPI on powerpc, so I'm happy hooking the gating with irqs off ... but the existing hook isn't in the right place imho.

Quoted 8 lines
Quoted 2 lines

Quite frankly, I don't think the second one is workable. It may be the optimal one in theory, but it's never worked for us in practice.

True, it's going to be a pain. However, I still think you're going to be bitten somewhere if you whack config space back before you called the appropriate ACPI magic to resume the device, but I may be wrong, maybe nobody will ever implement really aggressive power management in their AML ? :-)

In theory, if the device supports native PCI PM, it should be safe to put it into D0 using that and restore the config registers.

Still, there may be some strange planar devices that need AML to power them up. We'll see. :-)

Thanks,
Rafael

#12 lore raw
In reply to Rafael J. Wysocki
Quoted 5 lines · Rafael J. Wysocki

Yes, which is why I thought it might be a good idea to make the AML interpreter allow is to execute AML with interrupts off, so that we can put devices into low power states and/or put them into D0 with interrupts off.

Quoted 7 lines

Then, we'll be able to make ACPI happy (up to some strange ordering expectations of some insane BIOSes maybe) and fix the interrupts issue at the same time.

The idea would be to have a special code path(s) where AML can be executed with interrupts off and a couple of special entry points into the AML interpreter for this purpose.

Well, I don't think anybody wants to (or necessarily has the ability) touch ACPI internals.

How about this kind of approach:

- step#1: split "sysdev" device suspend/resume from the general "device_power_on/off()" step

Rationale: sysdev's include things like the actual interrupt controller etc, and we almost certainly really do want to have hard interrupts disabled over that whole thing.

I'm appending a suggested trivial patch for this.

- step#2: make suspend/resume actually do the three phases:

a) existing device->suspend() (interrupts on, everything live)

b) disable_device_irq()'s: things are live, but device interrupts are turned off by essentially looping over the irq_desc_ptr[] table.

c) existing device->suspend_late(). System is kind of live, but all interrupts have been shut off at the source. But we really could try to keep timers etc alive.

d) disable CPU interrupts.

e) sysdev_suspend(). This turns off the interrupt controller etc

*suspend cpu*

and then do the resume in reverse order.

The reason step#1 is needed is just because we have that irq handling difference between the two. And if timers are running, we really can pretty much run all normal code (turning timers off does hurt anything that uses timeouts like msleep() or schedule_timeout()). Otherwise step#1 isn't important in itself.

Hmm? It doesn't look too painful, and it would mean that we could go back to a perfectly regular pci_set_power_state() in the early-resume codepath.

Ingo: how do we walk over all the interrupt descriptors in todays world? The whole sparse-vs-nonsparse makes things a bit more complex.

Linus

---

Author: Linus Torvalds <[email protected]>
Date:   Mon Feb 2 13:36:19 2009 -0800

Split up sysdev_[suspend|resume] from device_power_[down|up]

This just moves the sysdev_suspend/resume from the callee to the callers, with no real change in semantics.

---
 arch/x86/kernel/apm_32.c  |    4 ++++
 drivers/base/power/main.c |    3 ---
 drivers/xen/manage.c      |    8 ++++++++
 kernel/kexec.c            |    7 +++++++
 kernel/power/disk.c       |   11 +++++++++++
 kernel/power/main.c       |    8 ++++++--
 6 files changed, 36 insertions(+), 5 deletions(-)
arch/x86/kernel/apm_32.c +4 -0
index 98807bb..67021ad 100644
@@ -1192,6 +1192,7 @@ static int suspend(int vetoable)
 	device_suspend(PMSG_SUSPEND);
 	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+	sysdev_syspend();
 
 	local_irq_enable();
 
@@ -1208,6 +1209,7 @@ static int suspend(int vetoable)
 	if (err != APM_SUCCESS)
 		apm_error("suspend", err);
 	err = (err == APM_SUCCESS) ? 0 : -EIO;
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 	local_irq_enable();
 	device_resume(PMSG_RESUME);
@@ -1228,6 +1230,7 @@ static void standby(void)
 
 	local_irq_disable();
 	device_power_down(PMSG_SUSPEND);
+	sysdev_suspend();
 	local_irq_enable();
 
 	err = set_system_power_state(APM_STATE_STANDBY);
@@ -1235,6 +1238,7 @@ static void standby(void)
 		apm_error("standby", err);
 
 	local_irq_disable();
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 	local_irq_enable();
 }
drivers/base/power/main.c +0 -3
index 670c9d6..2d14f4a 100644
@@ -333,7 +333,6 @@ static void dpm_power_up(pm_message_t state)
  */
 void device_power_up(pm_message_t state)
 {
-	sysdev_resume();
 	dpm_power_up(state);
 }
 EXPORT_SYMBOL_GPL(device_power_up);
@@ -577,8 +576,6 @@ int device_power_down(pm_message_t state)
 		}
 		dev->power.status = DPM_OFF_IRQ;
 	}
-	if (!error)
-		error = sysdev_suspend(state);
 	if (error)
 		dpm_power_up(resume_event(state));
 	return error;
drivers/xen/manage.c +8 -0
index 9b91617..fb183a0 100644
@@ -45,6 +45,13 @@ static int xen_suspend(void *data)
 		       err);
 		return err;
 	}
+	err = sysdev_suspend();
+	if (err) {
+		printk(KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n",
+			err);
+		device_power_up(PMSG_RESUME);
+		return err;
+	}
 
 	xen_mm_pin_all();
 	gnttab_suspend();
@@ -61,6 +68,7 @@ static int xen_suspend(void *data)
 	gnttab_resume();
 	xen_mm_unpin_all();
 
+	sysdev_resume();
 	device_power_up(PMSG_RESUME);
 
 	if (!*cancelled) {
kernel/kexec.c +7 -0
index 8a6d7b0..b780288 100644
@@ -1465,6 +1465,11 @@ int kernel_kexec(void)
 		error = device_power_down(PMSG_FREEZE);
 		if (error)
 			goto Enable_irqs;
+
+		/* Suspend system devices */
+		error = sysdev_suspend();
+		if (error)
+			goto Power_up_devices;
 	} else
 #endif
 	{
@@ -1477,6 +1482,8 @@ int kernel_kexec(void)
 
 #ifdef CONFIG_KEXEC_JUMP
 	if (kexec_image->preserve_context) {
+		sysdev_resume();
+ Power_up_devices:
 		device_power_up(PMSG_RESTORE);
  Enable_irqs:
 		local_irq_enable();
kernel/power/disk.c +11 -0
index 432ee57..07dc9a2 100644
@@ -227,6 +227,12 @@ static int create_image(int platform_mode)
 			"aborting hibernation\n");
 		goto Enable_irqs;
 	}
+	sysdev_suspend();
+	if (error) {
+		printk(KERN_ERR "PM: Some devices failed to power down, "
+			"aborting hibernation\n");
+		goto Power_up_devices;
+	}
 
 	if (hibernation_test(TEST_CORE))
 		goto Power_up;
@@ -242,9 +248,11 @@ static int create_image(int platform_mode)
 	if (!in_suspend)
 		platform_leave(platform_mode);
  Power_up:
+	sysdev_resume();
 	/* NOTE:  device_power_up() is just a resume() for devices
 	 * that suspended with irqs off ... no overall powerup.
 	 */
+ Power_up_devices:
 	device_power_up(in_suspend ?
 		(error ? PMSG_RECOVER : PMSG_THAW) : PMSG_RESTORE);
  Enable_irqs:
@@ -335,6 +343,7 @@ static int resume_target_kernel(void)
 			"aborting resume\n");
 		goto Enable_irqs;
 	}
+	sysdev_suspend();
 	/* We'll ignore saved state, but this gets preempt count (etc) right */
 	save_processor_state();
 	error = restore_highmem();
@@ -357,6 +366,7 @@ static int resume_target_kernel(void)
 	swsusp_free();
 	restore_processor_state();
 	touch_softlockup_watchdog();
+	sysdev_resume();
 	device_power_up(PMSG_RECOVER);
  Enable_irqs:
 	local_irq_enable();
@@ -440,6 +450,7 @@ int hibernation_platform_enter(void)
 	local_irq_disable();
 	error = device_power_down(PMSG_HIBERNATE);
 	if (!error) {
+		sysdev_suspend();
 		hibernation_ops->enter();
 		/* We should never get here */
 		while (1);
kernel/power/main.c +6 -2
index 2399888..7709556 100644
@@ -298,8 +298,12 @@ static int suspend_enter(suspend_state_t state)
 		goto Done;
 	}
 
-	if (!suspend_test(TEST_CORE))
-		error = suspend_ops->enter(state);
+	error = sysdev_suspend();
+	if (!error) {
+		if (!suspend_test(TEST_CORE))
+			error = suspend_ops->enter(state);
+		sysdev_resume();
+	}
 
 	device_power_up(PMSG_RESUME);
  Done:
#13 lore raw
Quoted 3 lines · Linus Torvalds

Ingo: how do we walk over all the interrupt descriptors in todays world? The whole sparse-vs-nonsparse makes things a bit more complex.

Stage#2 would look something like this, except I haven't written the "[disable|enable]_device_interrupts()" for the simple reason that I'd like somebody else to do it ;)

But it would basically be just

	for (irq = 1; irq < NR_IRQS; irq++)
		[disable|enable]_irq(irq);

with maybe some day some way of marking system interrupts separately (ie timers).

Linus

---
 kernel/power/main.c |   13 ++++++++-----
 1 files changed, 8 insertions(+), 5 deletions(-)
kernel/power/main.c +8 -5
index 7709556..f54acb6 100644
@@ -290,14 +290,15 @@ static int suspend_enter(suspend_state_t state)
 	int error = 0;
 
 	device_pm_lock();
-	arch_suspend_disable_irqs();
-	BUG_ON(!irqs_disabled());
-
+	disable_device_interrupts();
 	if ((error = device_power_down(PMSG_SUSPEND))) {
 		printk(KERN_ERR "PM: Some devices failed to power down\n");
 		goto Done;
 	}
 
+	arch_suspend_disable_irqs();
+	BUG_ON(!irqs_disabled());
+
 	error = sysdev_suspend();
 	if (!error) {
 		if (!suspend_test(TEST_CORE))
@@ -305,10 +306,12 @@ static int suspend_enter(suspend_state_t state)
 		sysdev_resume();
 	}
 
-	device_power_up(PMSG_RESUME);
- Done:
 	arch_suspend_enable_irqs();
 	BUG_ON(irqs_disabled());
+
+	device_power_up(PMSG_RESUME);
+ Done:
+	enable_device_interrupts();
 	device_pm_unlock();
 	return error;
 }
#88 lore raw
In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki

I think it would be easier to make ACPI allow us to run AML with interrupts off.

Well, I'd agree, except I have this strong memory of us having known bugs with ACPI turning hard-interrupts on again. Similarly, it uses mutexes etc that simply don't work with interrupts off and/or may turn them on again thanks to scheduling.

"Fixing" that seems not very easy. ACPI has a bad habit of being _really_ hard to fix in this area.

I do agree that _if_ we can just fix ACPI, we wouldn't have these issues, and we should just call it with interrupts disabled with our existing code. But my previous email was a "maybe we can do it like this" kind of thing, which might allow us to use ACPI with none of the irq-off issues.

Linus

#85 lore raw
In reply to Rafael J. Wysocki
Quoted 11 lines · Rafael J. Wysocki

Yes, which is why I thought it might be a good idea to make the AML interpreter allow is to execute AML with interrupts off, so that we can put devices into low power states and/or put them into D0 with interrupts off.

Then, we'll be able to make ACPI happy (up to some strange ordering expectations of some insane BIOSes maybe) and fix the interrupts issue at the same time.

The idea would be to have a special code path(s) where AML can be executed with interrupts off and a couple of special entry points into the AML interpreter for this purpose.

Right. That way, I can also use the same hooks for my clock gating and be confident they'll be called at the right time.

Now, we need some ACPI folks to tell us why this won't work :-)

Ben.

#14 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 9 lines · Rafael J. Wysocki

Yes, which is why I thought it might be a good idea to make the AML interpreter allow is to execute AML with interrupts off, so that we can put devices into low power states and/or put them into D0 with interrupts off.

Then, we'll be able to make ACPI happy (up to some strange ordering expectations of some insane BIOSes maybe) and fix the interrupts issue at the same time.

Quoted 6 lines
Quoted 3 lines

The idea would be to have a special code path(s) where AML can be executed with interrupts off and a couple of special entry points into the AML interpreter for this purpose.

Well, I don't think anybody wants to (or necessarily has the ability) touch ACPI internals.

That need not be very difficult AFAICS.

Quoted 8 lines

How about this kind of approach:

- step#1: split "sysdev" device suspend/resume from the general "device_power_on/off()" step

Rationale: sysdev's include things like the actual interrupt controller etc, and we almost certainly really do want to have hard interrupts disabled over that whole thing.

Fine by me.

Quoted 1 line

I'm appending a suggested trivial patch for this.

Looks correct.

Quoted 7 lines

- step#2: make suspend/resume actually do the three phases:

	a) existing device->suspend() (interrupts on, everything live)

	b) disable_device_irq()'s: things are live, but device interrupts 
	   are turned off by essentially looping over the irq_desc_ptr[] 
	   table. 

Well, do we actually need to turn off all device interrupts?

Shared interrupts are the source of the problem, so perhaps we can only disable interrupts of devices that use interrupt pins at this point (MSI/MSI-X need not be disabled, for example, and the timer interrupts most probably too)?

Quoted 5 lines

c) existing device->suspend_late(). System is kind of live, but all interrupts have been shut off at the source. But we really could try to keep timers etc alive.

d) disable CPU interrupts.

At what point do we disable the other CPUs?

Quoted 6 lines

e) sysdev_suspend(). This turns off the interrupt controller etc

*suspend cpu*

and then do the resume in reverse order.

Quoted 7 lines

The reason step#1 is needed is just because we have that irq handling difference between the two. And if timers are running, we really can pretty much run all normal code (turning timers off does hurt anything that uses timeouts like msleep() or schedule_timeout()). Otherwise step#1 isn't important in itself.

Hmm? It doesn't look too painful,

Well, it means reworking the entire suspend sequence (again) or we will break assumptions made by some existing drivers (interrupts off during suspend_late and resume_early). And that affects all drivers, not only PCI.

Quoted 2 lines

and it would mean that we could go back to a perfectly regular pci_set_power_state() in the early-resume codepath.

I first would like to understand what _exactly_ breaks on the iBook reported to have problems.

Thanks,
Rafael

#89 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

I think it would be easier to make ACPI allow us to run AML with interrupts off.

Well, I'd agree, except I have this strong memory of us having known bugs with ACPI turning hard-interrupts on again. Similarly, it uses mutexes etc that simply don't work with interrupts off and/or may turn them on again thanks to scheduling.

Quoted 7 lines

"Fixing" that seems not very easy. ACPI has a bad habit of being _really_ hard to fix in this area.

I do agree that _if_ we can just fix ACPI, we wouldn't have these issues, and we should just call it with interrupts disabled with our existing code. But my previous email was a "maybe we can do it like this" kind of thing, which might allow us to use ACPI with none of the irq-off issues.

Well, I think both will be difficult, this way or another. :-)

I'll have a look at the ACPI thing.

Thanks,
Rafael

#73 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

Well, I don't think anybody wants to (or necessarily has the ability) touch ACPI internals.

Somebody has a rough idea of what underlying kernel support ACPI uses ? I know from discussions with Len that it's mostly mutexes... timers too ?

It might end up being trivial to make it safe to call early with interrupts off with the trick you mentioned, since it's really just the same as boot as you mentioned.. ie, it's actually safe to take mutexes etc... since we -know- no other CPU is running and we aren't scheduling etc...

Ben.

#90 lore raw
In reply to Rafael J. Wysocki
Quoted 15 lines · Rafael J. Wysocki
Quoted 14 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

I think it would be easier to make ACPI allow us to run AML with interrupts off.

Well, I'd agree, except I have this strong memory of us having known bugs with ACPI turning hard-interrupts on again. Similarly, it uses mutexes etc that simply don't work with interrupts off and/or may turn them on again thanks to scheduling.

"Fixing" that seems not very easy. ACPI has a bad habit of being _really_ hard to fix in this area.

Quoted 8 lines
Quoted 4 lines

I do agree that _if_ we can just fix ACPI, we wouldn't have these issues, and we should just call it with interrupts disabled with our existing code. But my previous email was a "maybe we can do it like this" kind of thing, which might allow us to use ACPI with none of the irq-off issues.

Well, I think both will be difficult, this way or another. :-)

I'll have a look at the ACPI thing.

Generally speaking, we'd need to run acpi_evaluate_object() with interrupts off.

There are two apparent problems with that, from a quick look:

* The ACPI_MTX_INTERPRETER mutex needs to be acquired, but we know we won't need that mutex with interrupts off, so presumably we can work around this.

* Memory allocations with GFP_KERNEL are made, which is even worse, because we really shouldn't do that during suspend _at_ _all_, even during the regular ->suspend() with interrupts on, because there's not guarantee that swap will will be available at that time. So, for the sake of correctness, we should get rid of the GFP_KERNEL from the ACPI code paths executed during suspend-resume anyway.

Thanks,
Rafael

#94 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

I think it would be easier to make ACPI allow us to run AML with interrupts off.

Well, I'd agree, except I have this strong memory of us having known bugs with ACPI turning hard-interrupts on again. Similarly, it uses mutexes etc that simply don't work with interrupts off and/or may turn them on again thanks to scheduling.

Quoted 7 lines

"Fixing" that seems not very easy. ACPI has a bad habit of being _really_ hard to fix in this area.

I do agree that _if_ we can just fix ACPI, we wouldn't have these issues, and we should just call it with interrupts disabled with our existing code. But my previous email was a "maybe we can do it like this" kind of thing, which might allow us to use ACPI with none of the irq-off issues.

Len, what's your take here ? How much of that stuff is burried deep and how much is nicely split in a "helper" layer we could more easily fix ?

I'm adding Ingo to the CC as he might have more ideas on how best to just make the mutexes work & not complain rather than touching ACPI itself... again, just like boot, might just be a matter of instructing the mutexes/lockdep to shut up and ignore in_atomic() in those "special" phases such as late suspend and early resume().

That would help me for something else that broke recently too ... I have a special "hook" in radeonfb that my arch calls to resume it -very- early (interrupts off, I haven't even re-enabled the L2 cache). This is very useful to help debugging problems at resume since without that you basically don't see a thing and we have no serial port on most of these machines.

However, that started breaking recently due to fb_set_suspend() calling into various bits of infrastructure that is no longer safe to call in atomic context.

Here too, in fact, those -would- be safe since it's mostly a matter of teaching things like mutex of kmalloc that we are not in standard SYSTEM_RUNNING state, and thus mutex can just pretty much ignore the problem of being in atomic state and kmalloc/gfp could automatically degrade to GFP_ATOMIC (*)

So it might just all be a matter of making might_sleep() shut up in late suspend/early resume, and possibly msleep() silently turn into mdelay or something like that. Just make sure we don't actually try to schedule (and possibly BUG_ON if we actually end up blocking on a mutex, we should not).

Len, do you think that would work with ACPI or it's more convoluted than that ?

(*) There are reasons to think that kmalloc/gfp should both silently turn into GFP_NOIO always while the suspend process is started, but that's somewhat a different subject. Rafael, did we ever act on that ? It's an old discussion we had but I don't know if we actually implemented anything.

IE. Without that, afaik, a driver that hasn't suspend yet might end up being blocked in some allocation somewhere due to an attempt to page things out on to an already sleeping device. That driver might be in such blockage while holding one of its internal mutexes or other thing that will cause it's own suspend routine later on to screw up. etc etc etc... In general, best to avoid having to teach drivers that in suspend-land, non-atomic, allocations may block for ever. Better to make them all atomic magically.

Cheers,
Ben.

#74 lore raw
In reply to Benjamin Herrenschmidt
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 3 lines · Linus Torvalds

Well, I don't think anybody wants to (or necessarily has the ability) touch ACPI internals.

Somebody has a rough idea of what underlying kernel support ACPI uses ? I know from discussions with Len that it's mostly mutexes... timers too ?

Please see my last message in this thread (just sent).

Quoted 5 lines

It might end up being trivial to make it safe to call early with interrupts off with the trick you mentioned, since it's really just the same as boot as you mentioned.. ie, it's actually safe to take mutexes etc... since we -know- no other CPU is running and we aren't scheduling etc...

But we can't tell no one is holding the mutex in question, AFAICS.

I'm afraid we'd really need a special "no mutexes, no GFP_KERNEL allocations" code path for that.

Thanks,
Rafael

#15 lore raw
In reply to Rafael J. Wysocki
Quoted 11 lines · Rafael J. Wysocki
Quoted 4 lines
	b) disable_device_irq()'s: things are live, but device interrupts 
	   are turned off by essentially looping over the irq_desc_ptr[] 
	   table. 

Well, do we actually need to turn off all device interrupts?

Shared interrupts are the source of the problem, so perhaps we can only disable interrupts of devices that use interrupt pins at this point (MSI/MSI-X need not be disabled, for example, and the timer interrupts most probably too)?

We could try that, yes.

Quoted 3 lines
Quoted 1 line

d) disable CPU interrupts.

At what point do we disable the other CPUs?

I left it out, because I don't much care or think it matters. So take your pick. I'd suggest keeping the current setup, and literally just insert the new point between "device_power_off()" and "sysdev_suspend()", with _zero_ other changes.

Quoted 3 lines

Well, it means reworking the entire suspend sequence (again) or we will break assumptions made by some existing drivers (interrupts off during suspend_late and resume_early). And that affects all drivers, not only PCI.

No it doesn't.

No changes AT ALL to the suspend sequence. We do everything in the same
order: look at my patch. The only difference is that instead of doing that "cli" we do the "for_each_irq(disable_irq)" instead (and do the 'cli' a bit later).

ZERO effect on drivers. The calling convention is 100% the same as far as the driver is concerned: ->suspend() is called with interrupts on and a fully working machine, and ->suspend_late() called with interrupts off.

The only difference is the _mechanism_ of turning interrupts off. NOTHING else.

Quoted 2 lines

I first would like to understand what _exactly_ breaks on the iBook reported to have problems.

I bet it's code like the USB one:

	int usb_hcd_pci_resume(struct pci_dev *dev)
	{

	#ifdef CONFIG_PPC_PMAC
	        /* Reenable ASIC clocks for USB */
	        if (machine_is(powermac)) {
	                struct device_node *of_node;

	                of_node = pci_device_to_OF_node(dev);
	                if (of_node)
	                        pmac_call_feature(PMAC_FTR_USB_ENABLE,
	                                                of_node, 0, 1);
	        }
	#endif
	..
	        retval = pci_enable_device(dev);  

and now pci_enable_device() calls pci_raw_set_power_state(), which does:

	        if (dev->current_state == state) {
	                /* we're already there */
	                return 0;
		} else ..

which means that it doesn't actually _do_ anything, because it thinks that 'current_state' was already PCI_D0. But if the device was totally turned off, that's wrong.

(background: pci_restore_standard_config() will have done pci_raw_set_power_state(PCI_D0) with the device clocks off, which wouldn't actualyl have _done_ anythign to the device, but then it does

	dev->current_state = PCI_D0;

Maybe the simplest thing to do migth be to replace that with a

	pci_update_current_state(dev, PCI_D0);

instead, to try to read back the state explicitly in pci_restore_standard_config()).

Best test:

Ben, does this trivial patch make any difference for those powermacs?

Linus

---
 drivers/pci/pci.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/pci/pci.c +1 -1
index 17bd932..97e1c38 100644
@@ -1419,7 +1419,7 @@ int pci_restore_standard_config(struct pci_dev *dev)
 		}
 	}
 
-	dev->current_state = PCI_D0;
+	pci_update_current_state(dev, PCI_D0);
 
 	return 0;
 }
#95 lore raw
In reply to Benjamin Herrenschmidt
Quoted 52 lines · Benjamin Herrenschmidt
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

I think it would be easier to make ACPI allow us to run AML with interrupts off.

Well, I'd agree, except I have this strong memory of us having known bugs with ACPI turning hard-interrupts on again. Similarly, it uses mutexes etc that simply don't work with interrupts off and/or may turn them on again thanks to scheduling.

Quoted 7 lines

"Fixing" that seems not very easy. ACPI has a bad habit of being _really_ hard to fix in this area.

I do agree that _if_ we can just fix ACPI, we wouldn't have these issues, and we should just call it with interrupts disabled with our existing code. But my previous email was a "maybe we can do it like this" kind of thing, which might allow us to use ACPI with none of the irq-off issues.

Len, what's your take here ? How much of that stuff is burried deep and how much is nicely split in a "helper" layer we could more easily fix ?

I'm adding Ingo to the CC as he might have more ideas on how best to just make the mutexes work & not complain rather than touching ACPI itself... again, just like boot, might just be a matter of instructing the mutexes/lockdep to shut up and ignore in_atomic() in those "special" phases such as late suspend and early resume().

That would help me for something else that broke recently too ... I have a special "hook" in radeonfb that my arch calls to resume it -very- early (interrupts off, I haven't even re-enabled the L2 cache). This is very useful to help debugging problems at resume since without that you basically don't see a thing and we have no serial port on most of these machines.

However, that started breaking recently due to fb_set_suspend() calling into various bits of infrastructure that is no longer safe to call in atomic context.

Here too, in fact, those -would- be safe since it's mostly a matter of teaching things like mutex of kmalloc that we are not in standard SYSTEM_RUNNING state, and thus mutex can just pretty much ignore the problem of being in atomic state and kmalloc/gfp could automatically degrade to GFP_ATOMIC (*)

So it might just all be a matter of making might_sleep() shut up in late suspend/early resume, and possibly msleep() silently turn into mdelay or something like that. Just make sure we don't actually try to schedule (and possibly BUG_ON if we actually end up blocking on a mutex, we should not).

Quoted 8 lines

Len, do you think that would work with ACPI or it's more convoluted than that ?

(*) There are reasons to think that kmalloc/gfp should both silently turn into GFP_NOIO always while the suspend process is started, but that's somewhat a different subject. Rafael, did we ever act on that ? It's an old discussion we had but I don't know if we actually implemented anything.

We have the ->prepare(), ->complete() callbacks that, among other things, can be used for allocating and freeing memory with GFP_KERNEL safely.

Quoted 6 lines

IE. Without that, afaik, a driver that hasn't suspend yet might end up being blocked in some allocation somewhere due to an attempt to page things out on to an already sleeping device. That driver might be in such blockage while holding one of its internal mutexes or other thing that will cause it's own suspend routine later on to screw up. etc etc etc...

Yes, that's possible in theory, never observed in practice from what I can tell.

Quoted 3 lines

In general, best to avoid having to teach drivers that in suspend-land, non-atomic, allocations may block for ever. Better to make them all atomic magically.

Hm, atomic allocations may cause other problems to happen (ie. fail easily).

Thanks,
Rafael

#16 lore raw
In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Rafael J. Wysocki
Quoted 4 lines
	b) disable_device_irq()'s: things are live, but device interrupts 
	   are turned off by essentially looping over the irq_desc_ptr[] 
	   table. 
Quoted 8 lines
Quoted 6 lines

Well, do we actually need to turn off all device interrupts?

Shared interrupts are the source of the problem, so perhaps we can only disable interrupts of devices that use interrupt pins at this point (MSI/MSI-X need not be disabled, for example, and the timer interrupts most probably too)?

We could try that, yes.

So, perhaps we can just loop over the interrupt links and disable them all at this point?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

d) disable CPU interrupts.

At what point do we disable the other CPUs?

I left it out, because I don't much care or think it matters. So take your pick. I'd suggest keeping the current setup, and literally just insert the new point between "device_power_off()" and "sysdev_suspend()", with _zero_ other changes.

Ah, OK.

Quoted 10 lines
Quoted 3 lines

Well, it means reworking the entire suspend sequence (again) or we will break assumptions made by some existing drivers (interrupts off during suspend_late and resume_early). And that affects all drivers, not only PCI.

No it doesn't.

No changes AT ALL to the suspend sequence. We do everything in the same
order: look at my patch. The only difference is that instead of doing that "cli" we do the "for_each_irq(disable_irq)" instead (and do the 'cli' a bit later).

Sounds good.

Quoted 67 lines

ZERO effect on drivers. The calling convention is 100% the same as far as the driver is concerned: ->suspend() is called with interrupts on and a fully working machine, and ->suspend_late() called with interrupts off.

The only difference is the _mechanism_ of turning interrupts off. NOTHING else.

Quoted 2 lines

I first would like to understand what _exactly_ breaks on the iBook reported to have problems.

I bet it's code like the USB one:

	int usb_hcd_pci_resume(struct pci_dev *dev)
	{

	#ifdef CONFIG_PPC_PMAC
	        /* Reenable ASIC clocks for USB */
	        if (machine_is(powermac)) {
	                struct device_node *of_node;

	                of_node = pci_device_to_OF_node(dev);
	                if (of_node)
	                        pmac_call_feature(PMAC_FTR_USB_ENABLE,
	                                                of_node, 0, 1);
	        }
	#endif
	..
	        retval = pci_enable_device(dev);  

and now pci_enable_device() calls pci_raw_set_power_state(), which does:

	        if (dev->current_state == state) {
	                /* we're already there */
	                return 0;
		} else ..

which means that it doesn't actually _do_ anything, because it thinks that 'current_state' was already PCI_D0. But if the device was totally turned off, that's wrong.

(background: pci_restore_standard_config() will have done pci_raw_set_power_state(PCI_D0) with the device clocks off, which wouldn't actualyl have _done_ anythign to the device, but then it does

	dev->current_state = PCI_D0;

Maybe the simplest thing to do migth be to replace that with a

	pci_update_current_state(dev, PCI_D0);

instead, to try to read back the state explicitly in pci_restore_standard_config()).

Best test:

Ben, does this trivial patch make any difference for those powermacs?

Linus

---
 drivers/pci/pci.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/pci/pci.c +0 -0
index 17bd932..97e1c38 100644

drivers/pci/pci.c int pci_restore_standard_config(struct pci_dev *dev) · line 1422

Quoted 5 lines
drivers/pci/pci.c +1 -1
@@ -1419,7 +1419,7 @@ int pci_restore_standard_config(struct pci_dev *dev)
 		}
 	}
 
-	dev->current_state = PCI_D0;
+	pci_update_current_state(dev, PCI_D0);

Good idea anyway.

drivers/pci/pci.c int pci_restore_standard_config(struct pci_dev *dev) · line 1425

Quoted 2 lines
drivers/pci/pci.c +0 -0
 	return 0;
 }

Or perhaps put

current_state = PCI_UNKNOWN;

under that 'if (machine_is(powermac))' ?

Thanks,
Rafael

#82 lore raw
In reply to Linus Torvalds
Quoted 2 lines · Linus Torvalds

Ingo: how do we walk over all the interrupt descriptors in todays world? The whole sparse-vs-nonsparse makes things a bit more complex.

this:

        struct irq_desc *desc;
        int irq;

        for_each_irq_desc(i, desc) {
	...
	}

should do the trick.

Ingo

#17 lore raw
In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki

So, perhaps we can just loop over the interrupt links and disable them all at this point?

Well, we could, but I think it's actually inferior. Why? Because timer interrupts can come over those too.

So we're really better off just looping over our own software irq_desc arrays. That has the feature that

(a) we can easily mark the individual irqs if we want to - in ways that we can never do if we end up looping over some hardware concept like the IO-APIC inputs.

IOW, we can easily add a magic flag for registering a timer interrupt (in fact, I guess we already have IRQF_TIMER), but we can also expand on this in the future with other purely software interrupt flags - ie a driver that knows it is irq-safe could just tell us so, and we wouldn't need to disable that irq if there are only irq-safe drivers registered on it.

(b) it's portable, and handles re-sending the interrupts correctly (unlike just turning the interrupts off), and doesn't have any issues with worrying about polarity etc complicating issues.

Looping over the irq links is really quite hard. Sure, we can do the ACPI thing, but that's platform-specific, _and_ it's not actually guaranteed to hit everything anyway (ie it likely only disables the mapped ones, not necessarily at all directly connected ones - IDE and USB tend to be direct irq's because they are often on the same chip as the irq controller).

Quoted 5 lines

drivers/pci/pci.c int pci_restore_standard_config(struct pci_dev *dev) · line 1419

Quoted 3 lines
drivers/pci/pci.c +1 -1
@@ -1419,7 +1419,7 @@ int pci_restore_standard_config(struct pci_dev *dev)
-	dev->current_state = PCI_D0;
+	pci_update_current_state(dev, PCI_D0);

Good idea anyway.

Yeah, it probably makes sense to do this regardless of any PPC issues. Exactly because we don't call the platform code.

Quoted 5 lines

Or perhaps put

current_state = PCI_UNKNOWN;

under that 'if (machine_is(powermac))' ?

Yes. That would be ok, but in _theory_ ACPI could hit the same thing, and then the pci_restore_standard_config() really needs to do that anyway.

Linus

#91 lore raw
In reply to Rafael J. Wysocki
Quoted 2 lines · Rafael J. Wysocki

* The ACPI_MTX_INTERPRETER mutex needs to be acquired, but we know we won't need that mutex with interrupts off, so presumably we can work around this.

Or just make the mutex code not care when in the late suspend/early resume. I think that's the right approach and involves no change to ACPI itself.

Quoted 6 lines

* Memory allocations with GFP_KERNEL are made, which is even worse, because we really shouldn't do that during suspend _at_ _all_, even during the regular ->suspend() with interrupts on, because there's not guarantee that swap will will be available at that time. So, for the sake of correctness, we should get rid of the GFP_KERNEL from the ACPI code paths executed during suspend-resume anyway.

Well, as I said, this is a problem with more than just ACPI and again, I don't think the fix is to be done in ACPI itself. The buddy and slab should basically stick NOIO or ATOMIC to any allocation done after we started suspending the system.

Cheers,
Ben.

#18 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

IOW, we can easily add a magic flag for registering a timer interrupt (in fact, I guess we already have IRQF_TIMER)

Side note: we may have IRQF_TIMER (and some timers even set it), but since it's not actually _used_ for anything it should come as no surprise that it's not set by the x86 code, for example (nor by a lot of other architectures).

But we can already notice system interrupts automatically by them being registered by "setup_irq()" instead of "request_irq()". And on x86, many of the really core interrupts aren't actually seen as device interrupts at all, but are directly taken by the kernel through local apic stuff. So they wouldn't get disabled anyway (example: local APIC timer interrupts and the inter-processor IPI).

So it should be pretty easy to do a reasonable job with the information that we already have available.

Linus

#19 lore raw
In reply to Linus Torvalds
Quoted 27 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

So, perhaps we can just loop over the interrupt links and disable them all at this point?

Well, we could, but I think it's actually inferior. Why? Because timer interrupts can come over those too.

So we're really better off just looping over our own software irq_desc arrays. That has the feature that

(a) we can easily mark the individual irqs if we want to - in ways that we can never do if we end up looping over some hardware concept like the IO-APIC inputs.

IOW, we can easily add a magic flag for registering a timer interrupt (in fact, I guess we already have IRQF_TIMER), but we can also expand on this in the future with other purely software interrupt flags - ie a driver that knows it is irq-safe could just tell us so, and we wouldn't need to disable that irq if there are only irq-safe drivers registered on it.

(b) it's portable, and handles re-sending the interrupts correctly (unlike just turning the interrupts off), and doesn't have any issues with worrying about polarity etc complicating issues.

Quoted 6 lines

Looping over the irq links is really quite hard. Sure, we can do the ACPI thing, but that's platform-specific, _and_ it's not actually guaranteed to hit everything anyway (ie it likely only disables the mapped ones, not necessarily at all directly connected ones - IDE and USB tend to be direct irq's because they are often on the same chip as the irq controller).

OK

Quoted 8 lines
Quoted 5 lines

drivers/pci/pci.c int pci_restore_standard_config(struct pci_dev *dev) · line 1419

Quoted 3 lines
drivers/pci/pci.c +1 -1
@@ -1419,7 +1419,7 @@ int pci_restore_standard_config(struct pci_dev *dev)
-	dev->current_state = PCI_D0;
+	pci_update_current_state(dev, PCI_D0);

Good idea anyway.

Yeah, it probably makes sense to do this regardless of any PPC issues. Exactly because we don't call the platform code.

OK, I'll send a (yet another) patch to Jesse for that.

Quoted 8 lines
Quoted 5 lines

Or perhaps put

current_state = PCI_UNKNOWN;

under that 'if (machine_is(powermac))' ?

Yes. That would be ok, but in _theory_ ACPI could hit the same thing, and then the pci_restore_standard_config() really needs to do that anyway.

OK

BTW, on the PMAC the problematic driver appears to be the radeon driver, according to Ben, and the breakage is not related to USB.

Thanks,
Rafael

#92 lore raw
In reply to Benjamin Herrenschmidt
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 7 lines

* Memory allocations with GFP_KERNEL are made, which is even worse, because we really shouldn't do that during suspend _at_ _all_, even during the regular ->suspend() with interrupts on, because there's not guarantee that swap will will be available at that time. So, for the sake of correctness, we should get rid of the GFP_KERNEL from the ACPI code paths executed during suspend-resume anyway.

Quoted 4 lines

Well, as I said, this is a problem with more than just ACPI and again, I don't think the fix is to be done in ACPI itself. The buddy and slab should basically stick NOIO or ATOMIC to any allocation done after we started suspending the system.

We already do have magic code for things like PF_MEMALLOC (recursion), or TIF_MEMDIE (oom) or "(rt_task(p)) && !in_interrupt()" (give RT tasks higher priorities) in the page allocator.

Having a special case for NOT_RUNNING wouldn't be _pretty_, but it certainly isn't totally unreasonable either.

Linus

#75 lore raw
In reply to Rafael J. Wysocki
Quoted 5 lines

But we can't tell no one is holding the mutex in question, AFAICS.

I'm afraid we'd really need a special "no mutexes, no GFP_KERNEL allocations" code path for that.

No, the mutex shouldn't be held already, if it is, you're probably already in deep trouble. IE, you probably want to enfore that anyway, ie, it wouldn't be very sane to suspend the machine while ACPI was already in the -middle- of interpreting something anyway.

IE, you should have something to ensure, before you turn interrupts off, that nobody else is inside the AML interpreter. You already know there are no other CPUs, so it's just a matter of making sure no other process has scheduled while holding that mutex.

The easy way to do that is to do something like taking the mutex yourself and then setting a flag so that the intepreter stops trying to take it or release it itself, maybe just using the global system state.

Then release the mutex on resume.

All of these are issues that exist today. IE. Regardless of that powermac problem, which is unrelated (see other posts), I think these things need to be sorted cleanly or suspend will not be as rock solid as it could/should be. IE. It's several order of magnitude better than it was, I agree, but I believe we have here a few reasonably simple things we can/should do to make it more robust.

Cheers,
Ben.

#20 lore raw
In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki

BTW, on the PMAC the problematic driver appears to be the radeon driver, according to Ben, and the breakage is not related to USB.

Hmm. atyfb_base.c has the same kind of things with magic PMAC code, but it doesn't follow the USB pattern - it just replaces "pci_set_power_state()" _entirely_.

It seems a very interesting suspend routine, btw. It doesn't seem to do any of the pci_save_state() at all, just re-initializes from scratch. Maybe it is unhappy with the PM layer deciding to try to restore state since it clearly didn't..

Linus

#76 lore raw
In reply to Benjamin Herrenschmidt
Quoted 16 lines · Benjamin Herrenschmidt
Quoted 5 lines

But we can't tell no one is holding the mutex in question, AFAICS.

I'm afraid we'd really need a special "no mutexes, no GFP_KERNEL allocations" code path for that.

No, the mutex shouldn't be held already, if it is, you're probably already in deep trouble. IE, you probably want to enfore that anyway, ie, it wouldn't be very sane to suspend the machine while ACPI was already in the -middle- of interpreting something anyway.

IE, you should have something to ensure, before you turn interrupts off, that nobody else is inside the AML interpreter. You already know there are no other CPUs, so it's just a matter of making sure no other process has scheduled while holding that mutex.

Quoted 5 lines

The easy way to do that is to do something like taking the mutex yourself and then setting a flag so that the intepreter stops trying to take it or release it itself, maybe just using the global system state.

Then release the mutex on resume.

Yes, that should work.

Quoted 6 lines

All of these are issues that exist today. IE. Regardless of that powermac problem, which is unrelated (see other posts), I think these things need to be sorted cleanly or suspend will not be as rock solid as it could/should be. IE. It's several order of magnitude better than it was, I agree, but I believe we have here a few reasonably simple things we can/should do to make it more robust.

Agreed.

Thanks,
Rafael

#77 lore raw
In reply to Benjamin Herrenschmidt
Quoted 11 lines · Benjamin Herrenschmidt

IE, you should have something to ensure, before you turn interrupts off, that nobody else is inside the AML interpreter. You already know there are no other CPUs, so it's just a matter of making sure no other process has scheduled while holding that mutex.

The easy way to do that is to do something like taking the mutex yourself and then setting a flag so that the intepreter stops trying to take it or release it itself, maybe just using the global system state.

Then release the mutex on resume.

Why do you think this improves on anything?

Basically, it turns the mutex into a non-entity - but if your whole argument is that it might as well be a non-entity because nobody else can take it anyway, then why not just leave it around?

IOW, if your argument boils down to "there can be no contention", then you might as well say "just use the mutex, it will never block".

So the only thing you really need is to just disable the _debugging_ code that mutexes have (if they get built with debugging in the first place).

I can't find the bothersome code anyway: I do find

	DEBUG_LOCKS_WARN_ON(in_interrupt());

but that's just saying that you shouldn't be using mutexes from interrupts, not from irq-off segments. There's probably something I'm missing, like the preempt_check_resched() causing a schedule event with irq's disabled, and the "might_sleep()" thing. But the latter should already be disabled by the "system_state != SYSTEM_RUNNING" thing.

Oh, except we don't seem to have a SYSTEM_SUSPEND thing. Is that what people are complaining about?

Just do

	system_state = SYSTEM_SUSPEND;
	..
	system_state = SYSTEM_RUNNING;

around the irqs-off section in suspending if so.

Linus

#32 lore raw
In reply to Linus Torvalds
Quoted 4 lines

(background: pci_restore_standard_config() will have done pci_raw_set_power_state(PCI_D0) with the device clocks off, which wouldn't actualyl have _done_ anythign to the device, but then it does

Actually, we are lucky we don't even crash with a machine check when hitting an unclocked device ...

I found a problem with radeonfb (patch sent separately) that explains the breakage on one of my recent machines, but I wouldn't be surprised if some other models of powerbooks/ibooks have also problem with USB as you mentioned. IE. I have to check whether the ones doing PCI D state can also use the Apple specific clock stuff. We might be lucky and they are exclusive from each other.

Also, the nice thing with turning clocks off on these Apple ASICs is that it doesn't lose the config space content. So we may just get lucky here.

Quoted 1 line

Ben, does this trivial patch make any difference for those powermacs?

On my test setup, the problem is radeonfb and I just made a patch to fix it. But it's using some standard PCI USB, not the older fancy Apple stuff. I'll check on an older machine tongight when I get back home, I don't have one here.

Cheers,
Ben.

Quoted 18 lines

Linus

---
 drivers/pci/pci.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/pci/pci.c +1 -1
index 17bd932..97e1c38 100644
@@ -1419,7 +1419,7 @@ int pci_restore_standard_config(struct pci_dev *dev)
 		}
 	}
 
-	dev->current_state = PCI_D0;
+	pci_update_current_state(dev, PCI_D0);
 
 	return 0;
 }
#96 lore raw
In reply to Rafael J. Wysocki
Quoted 9 lines
Quoted 6 lines

(*) There are reasons to think that kmalloc/gfp should both silently turn into GFP_NOIO always while the suspend process is started, but that's somewhat a different subject. Rafael, did we ever act on that ? It's an old discussion we had but I don't know if we actually implemented anything.

We have the ->prepare(), ->complete() callbacks that, among other things, can be used for allocating and freeing memory with GFP_KERNEL safely.

First, that's assuming drivers will be smart enough to figure that out. They won't, believe me.

Then, as I said, this doesn't work in practice because there is no way drivers and/or underlying subsystems will start "remembering" when they are within a prepare/complete pair, and suddenly do allocations differently. That's just going to break.

It's the allocator itself that -must- degrade to NOIO or ATOMIC, or we'll just never get it right imho.

Quoted 2 lines

Yes, that's possible in theory, never observed in practice from what I can tell.

Sure, neither did I, though I could manufacture a case. IE, you'll need memory pressure at suspend time to start having problems with allocations blocking on disk access. So I'm sure most of the time we never hit it ... until we do one day and don't know why the whole system deadlocked somewhere in suspend or resume.

Here we have a few low hanging fruits like that we know about, that can make the difference between suspend/resume works always vs. it works -most- of the time, unless I happened to have been under heavy memory pressure while playing an MP3 on the train a 3rd day of the month....

IE. We don't want that rare case to break, because you can be sure that that once out of 1000 times where it does is going to piss me off real bad and probably just the day I typed something for 2h in a row without saving etc...

Quoted 1 line

Hm, atomic allocations may cause other problems to happen (ie. fail easily).

That's true, but at the end of the day, if the choice is between deadlock and failure, pick your medicine...

Maybe we could do it differently and have the allocators basically just give up if they are going to trigger IOs or something but at the end of the day, we need to make it robust, not just "will work most of the time".

Ben.

#21 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

BTW, on the PMAC the problematic driver appears to be the radeon driver, according to Ben, and the breakage is not related to USB.

Hmm. atyfb_base.c has the same kind of things with magic PMAC code, but it doesn't follow the USB pattern - it just replaces "pci_set_power_state()" _entirely_.

radeonfb_pm.c is the culprit here, I haven't looked at atyfb (mach64) yet, but it has similar issues yes. I just attached a patch to the BZ cleaning up radeonfb. I can have a look at doing a similar cleanup to atyfb and aty128fb next.

Quoted 4 lines

It seems a very interesting suspend routine, btw. It doesn't seem to do any of the pci_save_state() at all, just re-initializes from scratch. Maybe it is unhappy with the PM layer deciding to try to restore state since it clearly didn't..

Well, I wrote that so ... :-) bear in mind that this code was mostly written way before the PCI layer knew how to save and restore state. Also, atyfb has some cases of chips that don't support PCI D states but use private MMIO register to control suspend/resume.

Anyway, my proposed radeonfb patch is at:

http://bugzilla.kernel.org/attachment.cgi?id=20085

I'll look at cleaning up atyfb and aty128fb later today if needed.

Cheers,
Ben.

#78 lore raw
In reply to Linus Torvalds
Quoted 9 lines

Why do you think this improves on anything?

Basically, it turns the mutex into a non-entity - but if your whole argument is that it might as well be a non-entity because nobody else can take it anyway, then why not just leave it around?

IOW, if your argument boils down to "there can be no contention", then you might as well say "just use the mutex, it will never block".

Oh sure, you all argument is fine, there's just one little nit for which I think it made sense to take the mutex that way...

The possible problem I see is on the "boundary" when you are about to get interrupts off. You want to make sure that any other kernel thread / process that was inside ACPI is completed before you do that.

Taking the mutex will do that for you.

Then you can disable interrupts and just ignore the mutex until you re-enable them.

Now, of course, in a perfect world, anything that can poke ACPI would have got some suspend calls before hand and won't be around touching it but I wouldn't bet my life on this.

Cheers,
Ben.

#22 lore raw
In reply to Benjamin Herrenschmidt
Quoted 6 lines · Benjamin Herrenschmidt

Anyway, my proposed radeonfb patch is at:

http://bugzilla.kernel.org/attachment.cgi?id=20085

I'll look at cleaning up atyfb and aty128fb later today if needed.

Ok.

Note how the PCI layer (currently) only saves the low _16_ dwords of config space (64 bytes). The radeonfb code that did it by hand saved the whole 64 dwords (256 bytes). It _probably_ doesn't matter, but..

The reason the PCI layer only does 64 bytes is that that was the really old PCI config space model - the rest was undefined and apparently a few cards reportedly even crashed when accessing it (but who knows, that may be urban folklore).

But if that patch works for you, it's clearly already better than _not_ applying the patch, so..

Linus

#23 lore raw
Quoted 5 lines

Note how the PCI layer (currently) only saves the low _16_ dwords of config space (64 bytes). The radeonfb code that did it by hand saved the whole 64 dwords (256 bytes). It _probably_ doesn't matter, but..

Quoted 7 lines

The reason the PCI layer only does 64 bytes is that that was the really old PCI config space model - the rest was undefined and apparently a few cards reportedly even crashed when accessing it (but who knows, that may be urban folklore).

But if that patch works for you, it's clearly already better than _not_ applying the patch, so..

Radeons don't do much with config space... the worst we may miss I suppose is subsystem vendor/device... Maybe I'll add something to explicitely save and restore it or X might get upset. I'll have a look.

At some point, that whole code will migrate out of radeonfb into the new radeon DRM with kernel mode setting in which case even X should stop caring ... mostly.

Cheers,
Ben.

#24 lore raw
Quoted 9 lines · Benjamin Herrenschmidt
Quoted 9 lines

Note how the PCI layer (currently) only saves the low _16_ dwords of config space (64 bytes). The radeonfb code that did it by hand saved the whole 64 dwords (256 bytes). It _probably_ doesn't matter, but..

The reason the PCI layer only does 64 bytes is that that was the really old PCI config space model - the rest was undefined and apparently a few cards reportedly even crashed when accessing it (but who knows, that may be urban folklore).

Quoted 6 lines
Quoted 2 lines

But if that patch works for you, it's clearly already better than _not_ applying the patch, so..

Radeons don't do much with config space... the worst we may miss I suppose is subsystem vendor/device... Maybe I'll add something to explicitely save and restore it or X might get upset. I'll have a look.

Actually, subsystem stuff is below 0x40 so it should be fine too.

Cheers,
Ben.

#25 lore raw
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 4 lines · Benjamin Herrenschmidt

Radeons don't do much with config space... the worst we may miss I suppose is subsystem vendor/device... Maybe I'll add something to explicitely save and restore it or X might get upset. I'll have a look.

Actually, subsystem stuff is below 0x40 so it should be fine too.

The things above 0x40 tend to be:

- capabilities (values and next-pointers)

The PCI layer will save a random couple of these (read: the ones it cares about)

- random non-architected values specific to that chip. And sometimes these are important. Like ISA interrupt routing information for cardbus controllers. Or timing values set up by the BIOS.

but in 99% of all cases the stuff isn't really anything special, or is just fine at its cold-reset default values. But I could in theory see that some graphics card could hide things like DRAM timings in there, that get initialized on POST, but not (obviously) on a STR cycle.

So it could go either way. Totally unimportant or hugely important. Most cards don't seem to care, and saving just the low 64 bytes seems to work for almost all drivers.

Linus

#33 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

(background: pci_restore_standard_config() will have done pci_raw_set_power_state(PCI_D0) with the device clocks off, which wouldn't
actualyl have _done_ anythign to the device, but then it does

Quoted 8 lines
        dev->current_state = PCI_D0;

Maybe the simplest thing to do migth be to replace that with a

        pci_update_current_state(dev, PCI_D0);

instead, to try to read back the state explicitly in pci_restore_standard_config()).

Note that I notice on various powerbook models a bunch of things like

<some pci device> : restoring config space at offset XX (was: 0xffffffff, writin 0 ...)

I'm not sure if that will do any good :-)

The reason is that some devices (here sungem, ide-pmac and the firewire OHCI) -do- get clock gated until either you hit some of my magic pmac_feature calls in the driver or the driver calls pci_enable_device() (I have a hook there).

So far, it appears to be harmless ... at least on the one machine I have here, but it shows that we are doing something wrong.

So to recapitulate our entire discussion, let me know if I'm off the mark here:

- We agree that in general, the arch must have a chance to turn things on
before we do anything such as restoring the config space. This includes my pmac clock gating stuff and ACPI

- The problem with ACPI is that it needs interrupts because it uses mutexes,
GFP_KERNEL, etc.... There are two approaches to solving that problem. We can either make the mutexes not warn etc... or we can use a loop of disable_irq() instead of local_irq_save() between our normal and our "late" phase of PM.

- There are some arguments for the fact that making mutexes not warn and
GFP_KERNEL silently degrade to at least GFP_NOIO is a good thing to do at that stage in the suspend process (and the GFP_KERNEL -> NOIO should probably happen even earlier) so we might get ACPI for free there.

Now, here's just a few things to throw into the discussion:

- I think sysdevs should still have real IRQs off (local_irq_off) since that
includes shutting down the PIC itself, the timers, etc...

- Doing the disable_irq() loop instead of local_irq_save() will introduce one
subtle change to driver semantics, ie, they -might- be entered while in their suspend_late() via some timer interrupt (ie, IO scheduler acting on a timer, network stuff, tty output for the serial driver, etc...) while we had a pretty strong guarantee with the local_irq_save() case that nothing would happen at that stage, this isn't true anymore. That means more chances for drivers to get it wrong...

- Because of the two things above, I tend to favor myself the option of
making mutexes not warn etc...

- There are some additional reasons where generalizing that to the "late"
suspend and early resume stages could be useful, such as allowing early video resume of fbdev/fbcon or the new KMS etc... while those things may internally use mutexes or do GFP_KERNEL allocs etc... One example hitting me right now is my AGP PM code that I need to run early to bring the video back properly, which calls agp_collect_device_status() which calls pci_get_class() which does a kzalloc(...,GFP_KERNEL). Another one is the fb_set_suspend() itself uses the fb_notifier which is a blocking notifier. This is annoying because this notifier isn't used only for suspend/resume, for for other things that can legitimately be blocking, so making it an atomic notifier isn't really the right options, etc...

For those reason, I think we need to look at the few changes to the scheduler/mutex code, might_sleep() and allocators, so that:

- During the main suspend/resume phase: GFP_KERNEL -> GFP_NOIO

- During the "irq off" phase:
* GFP_KERNEL/NOIO -> GFP_ATOMIC
* msleep -> mdelay
* might_sleep() doesn't warn
* actual blocking on a mutex errors out verbosely

That would allow a whole chunk of code to be used in the later phase as if it was boot time, including probably ACPI. Of course, we still want to be careful, we aren't going to allow explicit scheduling, ie, things like sysdev's should still be written with the idea that it -is- atomic context, but at least cases such as the ones we've exposed above would be much more manageable.

Quoted 1 line

From there, we can then rework PCI to properly call the arch stuff first thing

on power on and be happy with it.

Comments ?

Cheers,
Ben.

#34 lore raw
Quoted 5 lines

Note that I notice on various powerbook models a bunch of things like

<some pci device> : restoring config space at offset XX (was: 0xffffffff, writin 0 ...)

Quoted 8 lines

I'm not sure if that will do any good :-)

The reason is that some devices (here sungem, ide-pmac and the firewire OHCI) -do- get clock gated until either you hit some of my magic pmac_feature calls in the driver or the driver calls pci_enable_device() (I have a hook there).

So far, it appears to be harmless ... at least on the one machine I have here, but it shows that we are doing something wrong.

Another little nit here...

The new code will basically make the core bring back everything to D0 on resume. That's a huge power surge and often not needed at all...

If we set aside the shared interrupt problem for a moment, why should waking up from sleep suddenly D0 everything in my system including stuff that might have self suspended due to not being used (wifi killswitched, audio off, etc...) and is still not going to be used until I actually decide to do so (aka the user)...

I suppose the drivers can then turn those things back off right away but it's still a bit fishy...

Of course, drivers that can do that sort of dynamic power management must already be ready to cope with interrupts coming from a shared line while asleep...

Maybe we need some flags in the drivers telling the core "I know what I'm doing and I'm ready to take shared interrupts after suspended, but please don't mess with my device behind my back"... but then, even stupid driver writers will abuse the flags ... hard.

Cheers,
Ben.

#26 lore raw
In reply to Linus Torvalds
Quoted 12 lines · Linus Torvalds
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 4 lines · Benjamin Herrenschmidt

Radeons don't do much with config space... the worst we may miss I suppose is subsystem vendor/device... Maybe I'll add something to explicitely save and restore it or X might get upset. I'll have a look.

Actually, subsystem stuff is below 0x40 so it should be fine too.

The things above 0x40 tend to be:

Quoted 8 lines

- capabilities (values and next-pointers)

The PCI layer will save a random couple of these (read: the ones it cares about)

- random non-architected values specific to that chip. And sometimes these are important. Like ISA interrupt routing information for cardbus controllers. Or timing values set up by the BIOS.

Right, I've seen such things, but in our case, all the radeons I support in there should be ok with just 64 bytes.

Cheers,
Ben.

#35 lore raw
In reply to Benjamin Herrenschmidt
Quoted 3 lines · Benjamin Herrenschmidt

The new code will basically make the core bring back everything to D0 on resume. That's a huge power surge and often not needed at all...

Well, we could easily do it only if it was D0 _before_ resume.

But quite frankly, before I worry about "huge power surge" issues, I want to feel like the architecture allows for clueless driver writers, and still giving us reliable suspend/resume.

Once we have that reliability, and only _then_, do I care at all about power issues. Make it work first, and not result in a dead machine if there is unlucky shared interrupt timings.

Linus

#36 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Benjamin Herrenschmidt

The new code will basically make the core bring back everything to D0 on resume. That's a huge power surge and often not needed at all...

Well, we could easily do it only if it was D0 _before_ resume.

Quoted 7 lines

But quite frankly, before I worry about "huge power surge" issues, I want to feel like the architecture allows for clueless driver writers, and still giving us reliable suspend/resume.

Once we have that reliability, and only _then_, do I care at all about power issues. Make it work first, and not result in a dead machine if there is unlucky shared interrupt timings.

Right.

Ben.

#79 lore raw
In reply to Linus Torvalds
Quoted 27 lines · Linus Torvalds
Quoted 11 lines · Benjamin Herrenschmidt

IE, you should have something to ensure, before you turn interrupts off, that nobody else is inside the AML interpreter. You already know there are no other CPUs, so it's just a matter of making sure no other process has scheduled while holding that mutex.

The easy way to do that is to do something like taking the mutex yourself and then setting a flag so that the intepreter stops trying to take it or release it itself, maybe just using the global system state.

Then release the mutex on resume.

Why do you think this improves on anything?

Basically, it turns the mutex into a non-entity - but if your whole argument is that it might as well be a non-entity because nobody else can take it anyway, then why not just leave it around?

IOW, if your argument boils down to "there can be no contention", then you might as well say "just use the mutex, it will never block".

So the only thing you really need is to just disable the _debugging_ code that mutexes have (if they get built with debugging in the first place).

I can't find the bothersome code anyway: I do find

Quoted 7 lines
	DEBUG_LOCKS_WARN_ON(in_interrupt());

but that's just saying that you shouldn't be using mutexes from interrupts, not from irq-off segments. There's probably something I'm missing, like the preempt_check_resched() causing a schedule event with irq's disabled, and the "might_sleep()" thing. But the latter should already be disabled by the "system_state != SYSTEM_RUNNING" thing.

Mutexes should work just fine in irqs-off sections - they'll safely save/restore interrupts, even the debug variants.

We used to have code in the mutex code that unconditionally enabled interrupts (a spin_unlock_irq() iirc) - but we fixed that pretty early on because it surprised some early boot code. Maybe this is the case you remember?

Ingo

#27 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

BTW, on the PMAC the problematic driver appears to be the radeon driver, according to Ben, and the breakage is not related to USB.

Quoted 8 lines

Hmm. atyfb_base.c has the same kind of things with magic PMAC code, but it doesn't follow the USB pattern - it just replaces "pci_set_power_state()" _entirely_.

It seems a very interesting suspend routine, btw. It doesn't seem to do any of the pci_save_state() at all, just re-initializes from scratch. Maybe it is unhappy with the PM layer deciding to try to restore state since it clearly didn't..

In fact, atyfb is also busted in -rc3 for different reasons though probably by the same patch. I just dug an old powerbook with a mach64 and it crashes on resume with a machine check in there. (Among 2 or 3 other problems introduced by recent kernels, such as pci_get_* now kmallocs with GFP_KERNEL internally which makes it WARN when I use it in my low level suspend/resume code to whack the memory controller, etc... this one is going to bite others I reckon, or IDE having some interrupt problems on resume).

Adding a pci_save_state() to atyfb_pci_suspend() and a pci_set_power_state() + pci_restore_state() at the beginning of atyfb_pci_resume() fixes the machine check here.

Now where it gets funny is that I've added code to read the BAR and command register content before, between, and after those calls and print it and .. they are sane... Until i discovered that what happens is that the new generic code seems to actually blast 0 all over my config space if I don't call pci_save_state() in suspend(). I suppose I was missing a "mandatory" call here... but the core should be more robust, ie it shouldn't erase the config space of something because a driver "forgot" to call pci_save_state() !

I'll do a patch fixing atyfb. I'll check if aty128fb tomorrow when paulus brings a machine with such a chip in, but it looks like it will have the same problem.

Cheers,
Ben.

#37 lore raw
In reply to Benjamin Herrenschmidt
Quoted 25 lines · Benjamin Herrenschmidt
Quoted 5 lines · Linus Torvalds

(background: pci_restore_standard_config() will have done pci_raw_set_power_state(PCI_D0) with the device clocks off, which wouldn't
actualyl have _done_ anythign to the device, but then it does

Quoted 8 lines
        dev->current_state = PCI_D0;

Maybe the simplest thing to do migth be to replace that with a

        pci_update_current_state(dev, PCI_D0);

instead, to try to read back the state explicitly in pci_restore_standard_config()).

Note that I notice on various powerbook models a bunch of things like

<some pci device> : restoring config space at offset XX (was: 0xffffffff, writin 0 ...)

I'm not sure if that will do any good :-)

The reason is that some devices (here sungem, ide-pmac and the firewire OHCI) -do- get clock gated until either you hit some of my magic pmac_feature calls in the driver or the driver calls pci_enable_device() (I have a hook there).

Quoted 8 lines

So far, it appears to be harmless ... at least on the one machine I have here, but it shows that we are doing something wrong.

So to recapitulate our entire discussion, let me know if I'm off the mark here:

- We agree that in general, the arch must have a chance to turn things on
before we do anything such as restoring the config space. This includes my pmac clock gating stuff and ACPI

Well, in fact that depends on whether or not the $subject patch causes more breakage to happen. If it doesn't, then this probably is only a theoretical issue.

That said, since we appear to be able to fix it, we should do that anyway IMO.

Quoted 9 lines

- The problem with ACPI is that it needs interrupts because it uses mutexes,
GFP_KERNEL, etc.... There are two approaches to solving that problem. We can either make the mutexes not warn etc... or we can use a loop of disable_irq() instead of local_irq_save() between our normal and our "late" phase of PM.

- There are some arguments for the fact that making mutexes not warn and
GFP_KERNEL silently degrade to at least GFP_NOIO is a good thing to do at that stage in the suspend process (and the GFP_KERNEL -> NOIO should probably happen even earlier) so we might get ACPI for free there.

I like the "loop of disable_irq()" idea, with a few modifications (see below).

Quoted 4 lines

Now, here's just a few things to throw into the discussion:

- I think sysdevs should still have real IRQs off (local_irq_off) since that
includes shutting down the PIC itself, the timers, etc...

Agreed. That's quite easy to implement, though.

Quoted 7 lines

- Doing the disable_irq() loop instead of local_irq_save() will introduce one
subtle change to driver semantics, ie, they -might- be entered while in their suspend_late() via some timer interrupt (ie, IO scheduler acting on a timer, network stuff, tty output for the serial driver, etc...) while we had a pretty strong guarantee with the local_irq_save() case that nothing would happen at that stage, this isn't true anymore. That means more chances for drivers to get it wrong...

Fortunately, only a few drivers implement suspend_late, so that should be easy to audit.

Also, the driver core code will probably need some modifications, because it was written with the assumption that interrupts will be off in certain places, but that's not a very big deal.

Quoted 15 lines

- Because of the two things above, I tend to favor myself the option of
making mutexes not warn etc...

- There are some additional reasons where generalizing that to the "late"
suspend and early resume stages could be useful, such as allowing early video resume of fbdev/fbcon or the new KMS etc... while those things may internally use mutexes or do GFP_KERNEL allocs etc... One example hitting me right now is my AGP PM code that I need to run early to bring the video back properly, which calls agp_collect_device_status() which calls pci_get_class() which does a kzalloc(...,GFP_KERNEL). Another one is the fb_set_suspend() itself uses the fb_notifier which is a blocking notifier. This is annoying because this notifier isn't used only for suspend/resume, for for other things that can legitimately be blocking, so making it an atomic notifier isn't really the right options, etc...

Quoted 4 lines

For those reason, I think we need to look at the few changes to the scheduler/mutex code, might_sleep() and allocators, so that:

- During the main suspend/resume phase: GFP_KERNEL -> GFP_NOIO

That will have to be done anyway at one point IMO.

Quoted 5 lines

- During the "irq off" phase:
* GFP_KERNEL/NOIO -> GFP_ATOMIC
* msleep -> mdelay
* might_sleep() doesn't warn
* actual blocking on a mutex errors out verbosely

That should not be necessary if we do the "loop of disable_irq()" version (except for the GFP_KERNEL -> GFP_NOIO thing).

Quoted 10 lines

That would allow a whole chunk of code to be used in the later phase as if it was boot time, including probably ACPI. Of course, we still want to be careful, we aren't going to allow explicit scheduling, ie, things like sysdev's should still be written with the idea that it -is- atomic context, but at least cases such as the ones we've exposed above would be much more manageable.

Quoted 1 line

From there, we can then rework PCI to properly call the arch stuff first thing

on power on and be happy with it.

Comments ?

As I said, I tend to prefer the "loop of disable_irq()" approach, because it would allow us to preserve the current ordering of ACPI operations. Namely, if we do:

suspend devices (normal suspend)
loop of disable_irq()
late suspend of devices
_PTS
disable nonboot CPUs
local_irq_disable()
sysdev suspend
enter sleep state
get control from the BIOS
sysdev resume
(*)
local_irq_enable()
enable nonboot CPUs
_WAK
early resume of devices
loop of enable_irq()
resume devices (normal resume)

the ordering of _PTS with respect to putting devices into low power states and disabling the nonboot CPUs will be the same as it is now and the same applies to _WAK and putting devices into D0 etc. (I really _really_ wouldn't like to change this ordering, since this alone is likely to break things badly).

Now, there's one subtle problem with resume in this picture. Namely, before running the "early resume of devices" we have to make sure that the interrupts will be masked. However, masking MSI-X, for example, means writing into the memory space of the device, so we can't do it at this point. Of course, we can assume that MSI/MSI-X will be masked when we get control from the BIOS (moreover, they are not shareable, so we can just ignore them at this point), but still we'll have to mask the other interrupts before doing the local_irq_enable() on resume - marked by the (*) above. This appears to be doable, though.

Thanks,
Rafael

#80 lore raw
In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

Mutexes should work just fine in irqs-off sections - they'll safely save/restore interrupts, even the debug variants.

mutex_lock() has might_sleep() which will generate a splat in atomic contexts.

That said, afaik the acpi code uses semaphores.

#81 lore raw
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Ingo Molnar

Mutexes should work just fine in irqs-off sections - they'll safely save/restore interrupts, even the debug variants.

mutex_lock() has might_sleep() which will generate a splat in atomic contexts.

That said, afaik the acpi code uses semaphores.

even the might_sleep() should be OK with Linus's system_state change to the suspend code:

void __might_sleep(char *file, int line)
{

#ifdef in_atomic
        static unsigned long prev_jiffy;        /* ratelimiting */

        if ((!in_atomic() && !irqs_disabled()) ||
                    system_state != SYSTEM_RUNNING || oops_in_progress)

	Ingo
#28 lore raw
In reply to Benjamin Herrenschmidt
Quoted 9 lines · Benjamin Herrenschmidt

Now where it gets funny is that I've added code to read the BAR and command register content before, between, and after those calls and print it and .. they are sane... Until i discovered that what happens is that the new generic code seems to actually blast 0 all over my config space if I don't call pci_save_state() in suspend(). I suppose I was missing a "mandatory" call here... but the core should be more robust, ie it shouldn't erase the config space of something because a driver "forgot" to call pci_save_state() !

You've found a bug somewhere.

We _should_ be saving things, the legacy code does something like this:

        if (drv && drv->suspend) {
                pci_dev->state_saved = false;

                i = drv->suspend(pci_dev, state);
                suspend_report_result(drv->suspend, i);
                if (i)
                        return i;

                if (pci_dev->state_saved)
                        goto Fixup;

                if (WARN_ON_ONCE(pci_dev->current_state != PCI_D0))
                        goto Fixup;
        }

        pci_save_state(pci_dev);

ie if your ->suspend function doesn't use pci_save_state() itself (which sets that "state_saved" flag to true), then the generic code will do it for you.

Also, on the resume path, we actually have

        if (pci_dev->state_saved)
                pci_restore_standard_config(pci_dev);

so I wonder how the heck you got that blast of all zeroes - because we clearly shouldn't be trying to restore any unsaved state!

So if you can figure out how it does all that...

Linus

#31 lore raw
In reply to Benjamin Herrenschmidt
Quoted 27 lines · Benjamin Herrenschmidt
Quoted 4 lines · Linus Torvalds
Quoted 2 lines · Rafael J. Wysocki

BTW, on the PMAC the problematic driver appears to be the radeon driver, according to Ben, and the breakage is not related to USB.

Quoted 8 lines

Hmm. atyfb_base.c has the same kind of things with magic PMAC code, but it doesn't follow the USB pattern - it just replaces "pci_set_power_state()" _entirely_.

It seems a very interesting suspend routine, btw. It doesn't seem to do any of the pci_save_state() at all, just re-initializes from scratch. Maybe it is unhappy with the PM layer deciding to try to restore state since it clearly didn't..

In fact, atyfb is also busted in -rc3 for different reasons though probably by the same patch. I just dug an old powerbook with a mach64 and it crashes on resume with a machine check in there. (Among 2 or 3 other problems introduced by recent kernels, such as pci_get_* now kmallocs with GFP_KERNEL internally which makes it WARN when I use it in my low level suspend/resume code to whack the memory controller, etc... this one is going to bite others I reckon, or IDE having some interrupt problems on resume).

Adding a pci_save_state() to atyfb_pci_suspend() and a pci_set_power_state() + pci_restore_state() at the beginning of atyfb_pci_resume() fixes the machine check here.

Quoted 8 lines

Now where it gets funny is that I've added code to read the BAR and command register content before, between, and after those calls and print it and .. they are sane... Until i discovered that what happens is that the new generic code seems to actually blast 0 all over my config space if I don't call pci_save_state() in suspend(). I suppose I was missing a "mandatory" call here... but the core should be more robust, ie it shouldn't erase the config space of something because a driver "forgot" to call pci_save_state() !

Whoa, I don't think we actually zero the contents in the suspend/resume core, but if the device goes into D3 the config space contents may be lost, maybe that's what happening here?

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#38 lore raw

Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Rafael J. Wysocki
Quoted 2 lines · Rafael J. Wysocki
Quoted 1 line · Benjamin Herrenschmidt

On Mon, 2009-02-02 at 15:18 -0800, Linus Torvalds wrote:

[--snip--]

Quoted 29 lines
Quoted 1 line

Comments ?

As I said, I tend to prefer the "loop of disable_irq()" approach, because it would allow us to preserve the current ordering of ACPI operations. Namely, if we do:

suspend devices (normal suspend)
loop of disable_irq()
late suspend of devices
_PTS
disable nonboot CPUs
local_irq_disable()
sysdev suspend
enter sleep state
get control from the BIOS
sysdev resume
(*)
local_irq_enable()
enable nonboot CPUs
_WAK
early resume of devices
loop of enable_irq()
resume devices (normal resume)

the ordering of _PTS with respect to putting devices into low power states and disabling the nonboot CPUs will be the same as it is now and the same applies to _WAK and putting devices into D0 etc. (I really _really_ wouldn't like to change this ordering, since this alone is likely to break things badly).

Quoted 9 lines

Now, there's one subtle problem with resume in this picture. Namely, before running the "early resume of devices" we have to make sure that the interrupts will be masked. However, masking MSI-X, for example, means writing into the memory space of the device, so we can't do it at this point. Of course, we can assume that MSI/MSI-X will be masked when we get control from the BIOS (moreover, they are not shareable, so we can just ignore them at this point), but still we'll have to mask the other interrupts before doing the local_irq_enable() on resume - marked by the (*) above. This appears to be doable, though.

Having reconsidered it, I think that the "loop of disable_irq()" may be problematic due to MSI/MSI-X and devices that are put into D3 during the "normal" suspend. That is, we shouldn't try to mask MSI/MSI-X for devices in D3 (especially MSI-X, since that involves writing to the device's memory space). This implies that devices in D3 should be avoided in the "loop of disable_irq()", but that could be tricky if we loop over struct irq_desc objects.

Still, we can modify pci_pm_suspend() (and the other PCI callbacks analogously) so that it masks the interrupt of the device right before returning to the caller if the device has not been put into a low power state before. After that all devices will either be in low power states, so they won't be able to generate interrupts, or have their interrupts masked. In the latter case the core can then put them into low power states in suspend_late().

To summarize, I'd like to do the following:

suspend devices (normal suspend, mask interrupts for devices still in D0)
late suspend of devices (devices cannot generate interrupts)
_PTS
disable nonboot CPUs
local_irq_disable()
sysdev suspend
enter sleep state
get control from the BIOS
sysdev resume
(*)
local_irq_enable()
enable nonboot CPUs
_WAK
early resume of devices {
- devices cannot generate interrupts
- devices are being put into D0
- standard config registers are being restored
}
resume devices (unmask device interrupts, normal resume)

The "early resume of devices" step can effectively unmask MSI/MSI-X, but that shouldn't matter, since they are not shared anyway.

IOW, I would just move the late suspend of devices before _PTS and the early resume of devices after _WAK, with the additional disabling/enabling of device interrups (all PCI devices must enter the late suspend phase either in a low power state, or with their interrupts masked; analogously for resume).

Thanks,
Rafael

#71 lore raw
In reply to Rafael J. Wysocki
Quoted 23 lines · Rafael J. Wysocki

As I said, I tend to prefer the "loop of disable_irq()" approach, because it would allow us to preserve the current ordering of ACPI operations. Namely, if we do:

suspend devices (normal suspend)
loop of disable_irq()
late suspend of devices
_PTS
disable nonboot CPUs
local_irq_disable()
sysdev suspend
enter sleep state
get control from the BIOS
sysdev resume
(*)
local_irq_enable()
enable nonboot CPUs
_WAK
early resume of devices
loop of enable_irq()
resume devices (normal resume)

Quoted 4 lines

the ordering of _PTS with respect to putting devices into low power states and disabling the nonboot CPUs will be the same as it is now and the same applies to _WAK and putting devices into D0 etc. (I really _really_ wouldn't like to change this ordering, since this alone is likely to break things badly).

Yes.

Also, make the "loop of disable/enable_irq()" phase be a helper function that also sets system_state to SYSTEM_SUSPENDING/SYSTEM_RUNNING respectively, and it should all be pretty clean, and the changes really should be pretty minimal.

Quoted 4 lines

Now, there's one subtle problem with resume in this picture. Namely, before running the "early resume of devices" we have to make sure that the interrupts will be masked. However, masking MSI-X, for example, means writing into the memory space of the device, so we can't do it at this point.

I really don't think it matters. Why? We simply don't care.

All MSI-X things will still have to go through the regular irq layer, so the disable/enable_irq part, so even though we've done the "local_irq_enable()", we just don't care.

Linus

#39 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Rafael J. Wysocki
Quoted 5 lines · Rafael J. Wysocki

Having reconsidered it, I think that the "loop of disable_irq()" may be problematic due to MSI/MSI-X and devices that are put into D3 during the "normal" suspend. That is, we shouldn't try to mask MSI/MSI-X for devices in D3

Rafael, you seem to be confused about what "disable_irq()" does.

It does not touch the driver hardware AT ALL. It literally just touches the interrupt controller, and even that only indirectly.

What disable_irq() does it literally:

	void disable_irq_nosync(unsigned int irq)
	{
	        struct irq_desc *desc = irq_to_desc(irq);
	        unsigned long flags;
  
	        if (!desc)
	                return;

	        spin_lock_irqsave(&desc->lock, flags);
	        if (!desc->depth++) {
	                desc->status |= IRQ_DISABLED;
	                desc->chip->disable(irq);
	        }
	        spin_unlock_irqrestore(&desc->lock, flags);
	}

and then it does a "synchronize_irq()" to wait to make sure that there are no pending ones.

And in many cases, even the

	desc->chip->disable(irq);

doesn't actually _do_ anything - we'll quite possibly continue to take the interrupt, and only when the interrupt happens will it see the "oh, IRQ_DISABLED is set" thing, and do something about it.

So don't worry about putting devices in D3 - disable_irq() will not care AT ALL whether the device is alive or not, and will never try to touch it anyway.

Linus

#40 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

Quoted 4 lines · Linus Torvalds

So don't worry about putting devices in D3 - disable_irq() will not care AT ALL whether the device is alive or not, and will never try to touch it anyway.

Btw, this is very much the case for MSI irq's in particular. If you ask to _mask_ them, it will go to the look up the device list and try to mask them (quite frankly, that sounds insane to me, but whatever), but that not what the irq layer does for the simple "disable()" case.

It will literally just set the flag, and then even if an interrupt happens afterwards, it will just ->ack it, and then call the ->end thing - and doesn't need to do anything else in the whole "disable" path because MSI's are obviously edge-triggered.

And the ACK is a pure (x2/io)apic thing, and again doesn't actually touch the device itself, only the irq controller itself.

So while it is possible in theory that some irq controller ends up trying to access the device for disable_irq, I don't think it's ever true in reality.

But it's possible that I overlooked some really odd case, of course.

Linus

#42 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Linus Torvalds
Quoted 32 lines · Linus Torvalds
Quoted 4 lines · Rafael J. Wysocki

Having reconsidered it, I think that the "loop of disable_irq()" may be problematic due to MSI/MSI-X and devices that are put into D3 during the "normal" suspend. That is, we shouldn't try to mask MSI/MSI-X for devices in D3

Rafael, you seem to be confused about what "disable_irq()" does.

It does not touch the driver hardware AT ALL. It literally just touches the interrupt controller, and even that only indirectly.

What disable_irq() does it literally:

	void disable_irq_nosync(unsigned int irq)
	{
	        struct irq_desc *desc = irq_to_desc(irq);
	        unsigned long flags;

	        if (!desc)
	                return;

	        spin_lock_irqsave(&desc->lock, flags);
	        if (!desc->depth++) {
	                desc->status |= IRQ_DISABLED;
	                desc->chip->disable(irq);
	        }
	        spin_unlock_irqrestore(&desc->lock, flags);
	}

and then it does a "synchronize_irq()" to wait to make sure that there are no pending ones.

Quoted 7 lines

And in many cases, even the

	desc->chip->disable(irq);

doesn't actually _do_ anything - we'll quite possibly continue to take the interrupt, and only when the interrupt happens will it see the "oh, IRQ_DISABLED is set" thing, and do something about it.

But won't ->disable point at ->mask in the MSI case (msi_chip doesn't have a ->disable, so the IRQ core will make ->disable point at ->mask)?

And in mask we do go poke at PCI regs (msi_set_mask_bits) to mask interrupts if possible (though if there's no mask bit in the legacy MSI case we don't do anything).

But again, these interrupts won't be shared, so maybe it doesn't matter, and maybe the IRQ_DISABLED flag will keep any drivers from seeing post suspend interrupts anyway...

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#41 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Linus Torvalds
Quoted 16 lines · Linus Torvalds
Quoted 4 lines · Linus Torvalds

So don't worry about putting devices in D3 - disable_irq() will not care AT ALL whether the device is alive or not, and will never try to touch it anyway.

Btw, this is very much the case for MSI irq's in particular. If you ask to _mask_ them, it will go to the look up the device list and try to mask them (quite frankly, that sounds insane to me, but whatever), but that not what the irq layer does for the simple "disable()" case.

It will literally just set the flag, and then even if an interrupt happens afterwards, it will just ->ack it, and then call the ->end thing - and doesn't need to do anything else in the whole "disable" path because MSI's are obviously edge-triggered.

Quoted 8 lines

And the ACK is a pure (x2/io)apic thing, and again doesn't actually touch the device itself, only the irq controller itself.

So while it is possible in theory that some irq controller ends up trying to access the device for disable_irq, I don't think it's ever true in reality.

But it's possible that I overlooked some really odd case, of course.

Sidenote: one future usecase that moves towards this might be threaded IRQ handlers - but it's not a real problem.

For them to run well we want to split IRQ processing into two bits: the "fast and atomic" bit, which just goes in, checks which hw generated the IRQ, then disables that hw via its device, and then wakes up the irq kernel thread to do all the fancy stuff.

[ at this point that 'complete the IRQ' processing might even be done on
another CPU - if we decide that the irq was requested from a given CPU. So there's even performance advantage to such a splitup. We have not yet built up much local CPU state so we can migrate the work cheaply. ]

But it's not an issue: this too will be driven centrally and disable_irq() will obviously inhibit irqs to all handlers on the chain [to stay compatible] and not touch the device hw in the suspend case.

So the 'disable the device irq itself' functionality is hidden behind the irq flow machinery - and never executed once you've done a widespread disable_irq() calls on (almost) all devices.

Ingo

#43 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Jesse Barnes
Quoted 3 lines · Jesse Barnes

But won't ->disable point at ->mask in the MSI case (msi_chip doesn't have a ->disable, so the IRQ core will make ->disable point at ->mask)?

No. I clarified this in the next message, but basically it boils down to a simple case: "for edge-triggered interrupts, it's actually stupid to mask things, because it's both simpler and _cheaper_ to just take change that the interrupt might happen once" (especially since it almost never happens).

In fact, for an edge-triggered interrupt it is often a _bug_ to mask the interrupt source, because you often lose the interrupt. So masking is not only complex and expensive, it's also often _wrong_. Instead, what disable_irq() does is to take the interrupt, but instead of calling the interrupt handlers (it's disabled!) it just sets a flag.

And that is a big _correctness_ issue, because the setting of the flag is how we know to resend the interrupt when we enable things again - even though the hardware itself actually lost the edge.

[ Of course, you can hope that the PCI device itself doesn't lose it, and
sees the edge again when unmasking, but the fact that many interrupt controllers get this fundamental masking issue wrong means that I'd almost bet that lots of PCI devices get it wrong too! ]

Quoted 3 lines

And in mask we do go poke at PCI regs (msi_set_mask_bits) to mask interrupts if possible (though if there's no mask bit in the legacy MSI case we don't do anything).

I actually think that's a bug. I think it's just horribly stupid for to ->mask to go down to the device layer (for all the same reasons that we don't do it for disable), but it doesn't much matter. We only use it for "shutdown" of the irq controller - we should do the same thing we do for irq_disable and just set a flag.

The only reason to actually _mask_ an interrupt is if it is level-triggered and can cause screaming. But even then it's often better to just wait for the interrupt to happen (and mask it then), because 99% of the time the interrupt obviously never happens.

But we've never had reason to optimize ->mask/->unmask.

That said, we've also never had much reason to _care_ deeply, so it's also possible that we do mask things over some path. I didn't actually walk _all_ the paths, and the logic for irq handling has changed enough over the years that I don't know all the paths any more. Maybe we do that explicit mask in some path I missed. We _shouldn't_, but who knows..

Linus

#44 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

Quoted 6 lines · Linus Torvalds

That said, we've also never had much reason to _care_ deeply, so it's also possible that we do mask things over some path. I didn't actually walk _all_ the paths, and the logic for irq handling has changed enough over the years that I don't know all the paths any more. Maybe we do that explicit mask in some path I missed. We _shouldn't_, but who knows..

Ok, so I decided to actually try to walk it all. Better look at the actual code.

Hmm. The _normal_ simple irq handler does this the way I described, but for some reason the "handle_edge_irq()" does not. And the reason is actually a buglet: it needs to mask things for the "recursive interrupt" case.

But that literally just looks like a small implementation detail (the code decided to share the code for IRQ_INPROGRESS and IRQ_DISABLED). We should fix it, so that you _can_ disable irqs and not have to worry about this all.

I'm really not sure why that handle_edge_irq thing uses "ack_and_mask()" instead of just "desc->chip->ack()"? I'm also totally flummoxed as to why it feels it needs to go all the way out to the device to mask things, instead of just masking at an apic level, which is much simpler and faster (especially since masking should never happen in practice anyway).

Linus

#45 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

Quoted 21 lines · Linus Torvalds
Quoted 6 lines · Linus Torvalds

That said, we've also never had much reason to _care_ deeply, so it's also possible that we do mask things over some path. I didn't actually walk _all_ the paths, and the logic for irq handling has changed enough over the years that I don't know all the paths any more. Maybe we do that explicit mask in some path I missed. We _shouldn't_, but who knows..

Ok, so I decided to actually try to walk it all. Better look at the actual code.

Hmm. The _normal_ simple irq handler does this the way I described, but for some reason the "handle_edge_irq()" does not. And the reason is actually a buglet: it needs to mask things for the "recursive interrupt" case.

But that literally just looks like a small implementation detail (the code decided to share the code for IRQ_INPROGRESS and IRQ_DISABLED). We should fix it, so that you _can_ disable irqs and not have to worry about this all.

Quoted 5 lines

I'm really not sure why that handle_edge_irq thing uses "ack_and_mask()" instead of just "desc->chip->ack()"? I'm also totally flummoxed as to why it feels it needs to go all the way out to the device to mask things, instead of just masking at an apic level, which is much simpler and faster (especially since masking should never happen in practice anyway).

Hm, do you mean mask_ack_irq()? The ->mask_ack() irqchip method is just a small tweak normally: if we get an irq while the irq was disabled, we can mark it pending and masks it for real.

It's optional for a PIC implementation to provide it and the generic code does it via ->mask() + ->ack() if the PIC implementation keeps it NULL.

[ In theory this also solves screaming level-triggered irqs that advertise
themselves as edge-triggered [due to firmware/BIOS bug - these do happen] and then keep spamming the system. ]

I have not done a deep audit, normally (on x86 at least) ->mask_ack() should not touch any lowlevel device bits (only the interrupt controller bits). Have you found a case where it does?

That would be arguably broken i think - we should not touch lowlevel device bits from the current generation of PIC code really, there's just no point.

Ingo

#55 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

Hmm. The _normal_ simple irq handler does this the way I described, but for some reason the "handle_edge_irq()" does not. And the reason is actually a buglet: it needs to mask things for the "recursive interrupt" case.

Btw, just to clarify: none of this happens at the actual "irq_disable()"
time: it only happens if you get an interrupt _while_ it's disabled. Which obviously shouldn't happen in the shutdown/wakeup path anyway for MSI, since the interrupts aren't shared, but it would be good to just be extra safe.

I do suspect we could/should just get rid of the msi masking entirely, but that may be too scary a step.

For the current suspend/resume situation, maybe it's enough to know that it shouldn't be happening anyway, and even if it _does_ happen on a device that has been shut down, it's just not going to do anything. Sure, it's doing that "writel/readl", but if it gets lost, who really cares? Nobody.

Linus

#46 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Ingo Molnar
Quoted 8 lines · Ingo Molnar
Quoted 6 lines

I'm really not sure why that handle_edge_irq thing uses "ack_and_mask()" instead of just "desc->chip->ack()"? I'm also totally flummoxed as to why it feels it needs to go all the way out to the device to mask things, instead of just masking at an apic level, which is much simpler and faster (especially since masking should never happen in practice anyway).

Hm, do you mean mask_ack_irq()?

Yes.

Quoted 3 lines
		 The ->mask_ack() irqchip method is just a 

small tweak normally: if we get an irq while the irq was disabled, we can mark it pending and masks it for real.

No, I know why mask_ack_irq() does what it does and I agree with it. What I was really reacting to was that handle_edge_irq() calls it at _all_. IOW, I'm talking about this code:

	handle_edge_irq(unsigned int irq, struct irq_desc *desc)
	...
	        if (unlikely((desc->status & (IRQ_INPROGRESS | IRQ_DISABLED)) ||
	                    !desc->action)) {
	                desc->status |= (IRQ_PENDING | IRQ_MASKED);
	                mask_ack_irq(desc, irq);
	..

where the masking part seems a bit pointless. And in the case of MSI, it causes us to go all the way out to the device, which sounds pretty expensive too.

So if you have high enough interrupt load that you get another interrupt on another CPU while handling one, I think the "mask_ack_irq()" likely makes things go slower. It also causes us to have that horrible:

                /*
                 * When another irq arrived while we were handling
                 * one, we could have masked the irq.
                 * Renable it, if it was not disabled in meantime.
                 */
                if (unlikely((desc->status &
                               (IRQ_PENDING | IRQ_MASKED | IRQ_DISABLED)) ==
                              (IRQ_PENDING | IRQ_MASKED))) {
                        desc->chip->unmask(irq);
                        desc->status &= ~IRQ_MASKED;
                }

code there in the middle of the loop to handle the interrupt.

That function has another oddity too: it does

                if (unlikely(!action)) {
			desc->chip->mask(irq);
                        goto out_unlock;
		}

but the thing is, if somebody unregistered the interrupt, then it should have been masked to begin with. Why would be need to mask it in the interrupt handler?

Maybe I'm missing something really subtle, but my reaction would be that the function _should_ look just something like the appended.

But no, I'm really not going to commit this. Consider this patch to be a question ("why does it do that?") rather than a suggestion ("we should do this").

Linus

---
 kernel/irq/chip.c |   20 +++-----------------
 1 files changed, 3 insertions(+), 17 deletions(-)
kernel/irq/chip.c +3 -17
index 7de11bd..ad852ce 100644
@@ -468,8 +468,8 @@ handle_edge_irq(unsigned int irq, struct irq_desc *desc)
 	 */
 	if (unlikely((desc->status & (IRQ_INPROGRESS | IRQ_DISABLED)) ||
 		    !desc->action)) {
-		desc->status |= (IRQ_PENDING | IRQ_MASKED);
-		mask_ack_irq(desc, irq);
+		desc->status |= IRQ_PENDING;
+		desc->chip->ack(irq);
 		desc = irq_remap_to_desc(irq, desc);
 		goto out_unlock;
 	}
@@ -486,22 +486,8 @@ handle_edge_irq(unsigned int irq, struct irq_desc *desc)
 		struct irqaction *action = desc->action;
 		irqreturn_t action_ret;
 
-		if (unlikely(!action)) {
-			desc->chip->mask(irq);
+		if (unlikely(!action))
 			goto out_unlock;
-		}
-
-		/*
-		 * When another irq arrived while we were handling
-		 * one, we could have masked the irq.
-		 * Renable it, if it was not disabled in meantime.
-		 */
-		if (unlikely((desc->status &
-			       (IRQ_PENDING | IRQ_MASKED | IRQ_DISABLED)) ==
-			      (IRQ_PENDING | IRQ_MASKED))) {
-			desc->chip->unmask(irq);
-			desc->status &= ~IRQ_MASKED;
-		}
 
 		desc->status &= ~IRQ_PENDING;
 		spin_unlock(&desc->lock);
#47 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

Quoted 28 lines · Linus Torvalds
Quoted 8 lines · Ingo Molnar
Quoted 6 lines

I'm really not sure why that handle_edge_irq thing uses "ack_and_mask()" instead of just "desc->chip->ack()"? I'm also totally flummoxed as to why it feels it needs to go all the way out to the device to mask things, instead of just masking at an apic level, which is much simpler and faster (especially since masking should never happen in practice anyway).

Hm, do you mean mask_ack_irq()?

Yes.

Quoted 3 lines
		 The ->mask_ack() irqchip method is just a 

small tweak normally: if we get an irq while the irq was disabled, we can mark it pending and masks it for real.

No, I know why mask_ack_irq() does what it does and I agree with it. What I was really reacting to was that handle_edge_irq() calls it at _all_. IOW, I'm talking about this code:

	handle_edge_irq(unsigned int irq, struct irq_desc *desc)
	...
	        if (unlikely((desc->status & (IRQ_INPROGRESS | IRQ_DISABLED)) ||
	                    !desc->action)) {
	                desc->status |= (IRQ_PENDING | IRQ_MASKED);
	                mask_ack_irq(desc, irq);
	..
Quoted 3 lines

where the masking part seems a bit pointless. And in the case of MSI, it causes us to go all the way out to the device, which sounds pretty expensive too.

hm, i agree that your patched version looks much simpler.

There's two reasons we have this variant:

- huge bikeshed painting thread in the early days of the genirq patchset, with folks expressing concern about our original plans to keep edge-triggered unmasked _always_. (which your patch does too) So we just went with the path of least resistence and used this hybride.

- the screaming-irq observation i had - do you consider that valid?:

Quoted 3 lines
Quoted 3 lines

[ In theory this also solves screaming level-triggered irqs that
advertise themselves as edge-triggered [due to firmware/BIOS bug - these do happen] and then keep spamming the system. ]

I wanted to have a pretty much interchangeable flow method between edge and level triggered - so that the BIOS cannot screw us by enumerating an irq as edge-triggered while it's level-triggered.

Especially for legacy x86 irqs in the low <16 range the trigger mode can be influenced by chipset settings and might not always be what we think it is.

That's my rough recollection - Thomas, is that correct and do you have anything to add here?

Ingo

#48 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

Quoted 6 lines · Ingo Molnar

- the screaming-irq observation i had - do you consider that valid?:

Quoted 3 lines
Quoted 3 lines

[ In theory this also solves screaming level-triggered irqs that
advertise themselves as edge-triggered [due to firmware/BIOS bug - these do happen] and then keep spamming the system. ]

Quoted 7 lines

I wanted to have a pretty much interchangeable flow method between edge and level triggered - so that the BIOS cannot screw us by enumerating an irq as edge-triggered while it's level-triggered.

Especially for legacy x86 irqs in the low <16 range the trigger mode can be influenced by chipset settings and might not always be what we think it is.

For apic and MSI based methods that's not a big issue: the trigger mode is explicitly set by us, so if there's a mismatch it's a kernel bug.

And even for legacy ISA i8259a.c it should be fine after all, as we initialize it via:

        set_irq_chip_and_handler_name(irq, &i8259A_chip, handle_level_irq,

Which is a screaming-safe sequence. (and the x86 i8259 PIC does not lose edges.)

That still leaves other architectures ... but i think the argument is a lot weaker than i thought it to be.

Could you send a Signed-off-by so that i can queue it up and test it a bit? If it does not blow up on x86 in practice then we should be fine, and it avoids the MSI ->mask() stupidity as well.

Ingo

#49 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Ingo Molnar
Quoted 10 lines · Ingo Molnar

- the screaming-irq observation i had - do you consider that valid?:

Quoted 3 lines
Quoted 3 lines

[ In theory this also solves screaming level-triggered irqs that
advertise themselves as edge-triggered [due to firmware/BIOS bug - these do happen] and then keep spamming the system. ]

I wanted to have a pretty much interchangeable flow method between edge and level triggered - so that the BIOS cannot screw us by enumerating an irq as edge-triggered while it's level-triggered.

Yes, if we can't be 100% sure it's really edge-triggered, I guess the mask thing is really worth it. So maybe "handle_edge_irq()" is actually doing everything right.

Of course, with MSI, we can fundamentally really be sure that it's edge-triggered (since it's literally a packet on the PCI bus that generates it), and that actually brings up another possibility: assuming handle_edge_irq() is doing the correct "safe" thing, maybe the answer is to just get rid of the MSI "mask()" operation as being unnecessary, and catch it at that level.

NOTE! From a correctness standpoint I think this is all irrelevant. Even if we have turned off the power of some device, the msi irq masking isn't going to hurt (apart from _possibly_ causing a machine check, but that's nothing new - architectures that enable machine checks on accesses to non-responding PCI hardware have to handle those anyway).

So I wouldn't worry too much. I think this is interesting mostly from a performance standpoint - MSI interrupts are supposed to be fast, and under heavy interrupt load I could easily see something like

- cpu1: handles interrupt, has acked it, calls down to the handler

- the handler clears the original irq source, but another packet (or disk completion) happens almost immediately

- cpu2 takes the second interrupt, but it's still IRQ_INPROGRESS, so it masks.

- cpu1 gets back and unmasks etc and now really handles it because of IRQ_PENDING.

Note how the mask/unmask were all just costly extra overhead over the PCI bus. If we're talking something like high-performance 10Gbit ethernet (or even maybe fast SSD disks), driver writers actually do count PCI cycles, because a single PCI read can be several hundred ns, and if you take a thousand interrupts per second, it does add up.

Of course, ethernet tends to do things like interrupt mitigation to avoid this, but that has its own downsides (longer latencies) and isn't really considered optimal in some RT environments (wall street trading kind of things).

I really don't know how big an issue this all is. It probably isn't really noticeable.

Linus

#50 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

Quoted 12 lines · Linus Torvalds

So I wouldn't worry too much. I think this is interesting mostly from a performance standpoint - MSI interrupts are supposed to be fast, and under heavy interrupt load I could easily see something like

- cpu1: handles interrupt, has acked it, calls down to the handler

- the handler clears the original irq source, but another packet (or disk completion) happens almost immediately

- cpu2 takes the second interrupt, but it's still IRQ_INPROGRESS, so it masks.

Quoted 8 lines

- cpu1 gets back and unmasks etc and now really handles it because of IRQ_PENDING.

Note how the mask/unmask were all just costly extra overhead over the PCI bus. If we're talking something like high-performance 10Gbit ethernet (or even maybe fast SSD disks), driver writers actually do count PCI cycles, because a single PCI read can be several hundred ns, and if you take a thousand interrupts per second, it does add up.

In practice MSI (and in particular MSI-X) irq sources tend to be bound to a single CPU on modern x86 hardware. The kernel does not do IRQ balancing anymore, nor does the hardware. We have a slow irq-balancer daemon (irqbalanced) in user-space. So singular IRQ sources, especially when they are MSI, tend to be 99.9% on the same CPU. Changing affinity is possible and has to always work reliably, but it is a performance slowpath.

An increasing trend is to have multiple irqs per device (multiple descriptor rings, split rx and tx rings with separate irq sources): and each IRQ can get balanced to a separate CPU. But those irqs cannot interact on a ->mask() level as each IRQ has its separate irq_desc.

The most advanced way of balancing IRQs is not widespread yet: it is where devices actually interpret the payload and send completions dynamically to differing CPUs - depending on things like the TCP/IP hash value or a in-descriptor "target CPU". That way we could get completion on the CPU where the work was submitted from. (and where the data structures are the most cache-localized)

That principle works both for networking and for other IO transports - but we have little support for it yet. It would work really well for workloads where one physical device is shared by many CPUs.

(A lesser method that approximates this is the use of lots of submission/completion rings per device and their binding to cpus - but that can never really approach the number of CPUs really possible in a system.)

And in this most advanced mode of MSI IRQs, and if MSI devices had the ability to direct IRQs to a specific CPU (they dont have that right now AFAICT), we'd run into the overhead scenarios you describe above, and your edge-triggered flow is the most performant one.

Ingo

#69 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki
Quoted 9 lines

Now, there's one subtle problem with resume in this picture. Namely, before running the "early resume of devices" we have to make sure that the interrupts will be masked. However, masking MSI-X, for example, means writing into the memory space of the device, so we can't do it at this point. Of course, we can assume that MSI/MSI-X will be masked when we get control from the BIOS (moreover, they are not shareable, so we can just ignore them at this point), but still we'll have to mask the other interrupts before doing the local_irq_enable() on resume - marked by the (*) above. This appears to be doable, though.

Which is why I prefer making mutex/semaphores/allocations "safe" to use in that late suspend phase with IRQs off.

It sounds like a less invasive thing, simpler, change, allowing to move the ACPI stuff back to where it belongs, and it would help solving other problems such as the problems I exposed with video resume, which I'm trying to do -very- early (ie, before sysdev's even).

In fact, as I may have said elsewhere, I'm also being bitten by the PCI layer doing kmalloc(...GFP_KERNEL) all over the place nowadays including in things like pci_get_device() which are hurting some memory controller code I have that runs in late suspend (I could refactor that code to do the pci_get_* earlier, it's just one more thing..).

Quoted 8 lines

Having reconsidered it, I think that the "loop of disable_irq()" may be problematic due to MSI/MSI-X and devices that are put into D3 during the "normal" suspend. That is, we shouldn't try to mask MSI/MSI-X for devices in D3 (especially MSI-X, since that involves writing to the device's memory space). This implies that devices in D3 should be avoided in the "loop of disable_irq()", but that could be tricky if we loop over struct irq_desc objects.

Quoted 6 lines

Still, we can modify pci_pm_suspend() (and the other PCI callbacks analogously) so that it masks the interrupt of the device right before returning to the caller if the device has not been put into a low power state before. After that all devices will either be in low power states, so they won't be able to generate interrupts, or have their interrupts masked. In the latter case the core can then put them into low power states in suspend_late().

That's going to be hard to get right vs. shared interrupts no ?

I think the "other" solution overall is much more simple.

Cheers,
Ben.

#51 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Ingo Molnar

* Ingo Molnar <[email protected]> wrote:

[...]

Quoted 3 lines

That principle works both for networking and for other IO transports - but we have little support for it yet. It would work really well for workloads where one physical device is shared by many CPUs.

Which is really what we have in _practice_ - only benchmarketing sets up one physical device per CPU and avoids all the ugly consequences of gathering/spreadig a channel of information from/to a single device to multiple CPUs.

Quoted 4 lines

(A lesser method that approximates this is the use of lots of submission/completion rings per device and their binding to cpus - but that can never really approach the number of CPUs really possible in a system.)

btw., more advanced device IRQ models was one of the thinking behind sparse IRQ support: defining a really large NR_IRQS limit on x86 by default, on all form factors, and making it really easy and cheap to have a _ton_ of IRQs in a Linux system might give hw designers ideas to create such hardware.

/me dreams on ;-)

Quoted 4 lines

And in this most advanced mode of MSI IRQs, and if MSI devices had the ability to direct IRQs to a specific CPU (they dont have that right now AFAICT), we'd run into the overhead scenarios you describe above, and your edge-triggered flow is the most performant one.

So i'd still like your tentative Signed-off-by for your patch - it's i think not v2.6.29 material but if it stays problem free in testing we can try it in v2.6.30. If it causes problem it will be clearly bisectable and clearly revertable.

Maybe we could split it in two: and for MSI we could introduce a 'simpler and faster' edge flow as well - and keep the legacy handler untouched. That way it's low-risk in its entirety. (and avoids the MSI ->mask complication as well.)

Ingo

#56 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 5 lines · Linus Torvalds

Hmm. The _normal_ simple irq handler does this the way I described, but for some reason the "handle_edge_irq()" does not. And the reason is actually a buglet: it needs to mask things for the "recursive interrupt" case.

Quoted 8 lines

Btw, just to clarify: none of this happens at the actual "irq_disable()"
time: it only happens if you get an interrupt _while_ it's disabled. Which obviously shouldn't happen in the shutdown/wakeup path anyway for MSI, since the interrupts aren't shared, but it would be good to just be extra safe.

I do suspect we could/should just get rid of the msi masking entirely, but that may be too scary a step.

I agree :-) It's been a source of problem anyway, I remember hearing some device reacting strangely (aka, losing MSIs) when masked, and it's not even implemented by all devices (somebody had the "smart" idea of making an optional feature). So it ends up being a lot of not very useful code...

Quoted 4 lines

For the current suspend/resume situation, maybe it's enough to know that it shouldn't be happening anyway, and even if it _does_ happen on a device that has been shut down, it's just not going to do anything. Sure, it's doing that "writel/readl", but if it gets lost, who really cares? Nobody.

Note that I still believe that life would be simpler to just keep the current local_irq_save() and just tweak might_sleep() etc... so that ACPI is safe to call. At least it can be done orthogonally to this interrupt change.

IE. As I said earlier, that interrupt masking -will- change the exposed semantics of suspend_late() to drivers. In fact, iirc, it's you who advertised initially suspend_late() as being the 'easy' way to write a PCI driver suspend routine specifically because you don't have to bother about being re-entered from anywhere for things like request processing etc... With the change, the kernel is essentially still operating, timers are ticking, we may even be scheduling, so this important assumption is gone. That means a lot more chances for driver to screw up.

Cheers,
Ben.

#52 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Ingo Molnar
Quoted 9 lines · Ingo Molnar

So i'd still like your tentative Signed-off-by for your patch - it's i think not v2.6.29 material but if it stays problem free in testing we can try it in v2.6.30. If it causes problem it will be clearly bisectable and clearly revertable.

Maybe we could split it in two: and for MSI we could introduce a 'simpler and faster' edge flow as well - and keep the legacy handler untouched. That way it's low-risk in its entirety. (and avoids the MSI ->mask complication as well.)

Yes, seperating out the MSI handler into it's own flow control is the right way to go. We had trouble with real edge hardware and IIRC most of the problems originated from ARM. rmk ??

Thanks,

tglx

#54 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Ingo Molnar
Quoted 5 lines · Ingo Molnar

So i'd still like your tentative Signed-off-by for your patch - it's i think not v2.6.29 material but if it stays problem free in testing we can try it in v2.6.30. If it causes problem it will be clearly bisectable and clearly revertable.

Hmm. You can have my sign-off, because I certainly don't think the patch is _wrong_, but I also don't think it's necessarily really worth it unless somebody can show a real upside. As mentioned, I don't think this matters in practice for the whole MSI suspend issue - doing the "mask()" thing feels stupid and certainl;y wasn't what I expected us to do, but it really shouldn't matter from an actual behavioural standpoint at all.

And if, as you say, we don't expect to have interrupts spread out across CPUs (and with the current io-apic they certainly don't happen - I don't know about x2apic) my performance arguments are bogus too.

So the patch probably is the right thing to do, but the upsides are slim to nonexistent, and the downside you pointed out of somebody using that "handle_edge_irq()" thing with an interrupt that turns out to be level after all makes me worry a bit.

But feel free to put it in some experimental branch with my sign-off.

Linus

#57 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Benjamin Herrenschmidt
Quoted 29 lines · Benjamin Herrenschmidt
Quoted 8 lines · Linus Torvalds
Quoted 5 lines · Linus Torvalds

Hmm. The _normal_ simple irq handler does this the way I described, but for some reason the "handle_edge_irq()" does not. And the reason is actually a buglet: it needs to mask things for the "recursive interrupt" case.

Quoted 8 lines

Btw, just to clarify: none of this happens at the actual "irq_disable()"
time: it only happens if you get an interrupt _while_ it's disabled. Which obviously shouldn't happen in the shutdown/wakeup path anyway for MSI, since the interrupts aren't shared, but it would be good to just be extra safe.

I do suspect we could/should just get rid of the msi masking entirely, but that may be too scary a step.

I agree :-) It's been a source of problem anyway, I remember hearing some device reacting strangely (aka, losing MSIs) when masked, and it's not even implemented by all devices (somebody had the "smart" idea of making an optional feature). So it ends up being a lot of not very useful code...

Quoted 4 lines

For the current suspend/resume situation, maybe it's enough to know that it shouldn't be happening anyway, and even if it _does_ happen on a device that has been shut down, it's just not going to do anything. Sure, it's doing that "writel/readl", but if it gets lost, who really cares? Nobody.

Quoted 4 lines

Note that I still believe that life would be simpler to just keep the current local_irq_save() and just tweak might_sleep() etc... so that ACPI is safe to call. At least it can be done orthogonally to this interrupt change.

That would change the ordering of ACPI method calls, which also is important and prone to breaking, as I wrote in the original message.

Quoted 9 lines

IE. As I said earlier, that interrupt masking -will- change the exposed semantics of suspend_late() to drivers. In fact, iirc, it's you who advertised initially suspend_late() as being the 'easy' way to write a PCI driver suspend routine specifically because you don't have to bother about being re-entered from anywhere for things like request processing etc... With the change, the kernel is essentially still operating, timers are ticking, we may even be scheduling, so this important assumption is gone. That means a lot more chances for driver to screw up.

Whatever we do, it will have drawbacks and the approach with moving local_irq_disable() seems to be more straightforward to me.

Thanks,
Rafael

#70 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Benjamin Herrenschmidt
Quoted 41 lines · Benjamin Herrenschmidt
Quoted 9 lines · Rafael J. Wysocki
Quoted 9 lines

Now, there's one subtle problem with resume in this picture. Namely, before running the "early resume of devices" we have to make sure that the interrupts will be masked. However, masking MSI-X, for example, means writing into the memory space of the device, so we can't do it at this point. Of course, we can assume that MSI/MSI-X will be masked when we get control from the BIOS (moreover, they are not shareable, so we can just ignore them at this point), but still we'll have to mask the other interrupts before doing the local_irq_enable() on resume - marked by the (*) above. This appears to be doable, though.

Which is why I prefer making mutex/semaphores/allocations "safe" to use in that late suspend phase with IRQs off.

It sounds like a less invasive thing, simpler, change, allowing to move the ACPI stuff back to where it belongs, and it would help solving other problems such as the problems I exposed with video resume, which I'm trying to do -very- early (ie, before sysdev's even).

In fact, as I may have said elsewhere, I'm also being bitten by the PCI layer doing kmalloc(...GFP_KERNEL) all over the place nowadays including in things like pci_get_device() which are hurting some memory controller code I have that runs in late suspend (I could refactor that code to do the pci_get_* earlier, it's just one more thing..).

Quoted 8 lines

Having reconsidered it, I think that the "loop of disable_irq()" may be problematic due to MSI/MSI-X and devices that are put into D3 during the "normal" suspend. That is, we shouldn't try to mask MSI/MSI-X for devices in D3 (especially MSI-X, since that involves writing to the device's memory space). This implies that devices in D3 should be avoided in the "loop of disable_irq()", but that could be tricky if we loop over struct irq_desc objects.

Quoted 6 lines

Still, we can modify pci_pm_suspend() (and the other PCI callbacks analogously) so that it masks the interrupt of the device right before returning to the caller if the device has not been put into a low power state before. After that all devices will either be in low power states, so they won't be able to generate interrupts, or have their interrupts masked. In the latter case the core can then put them into low power states in suspend_late().

Quoted 3 lines

That's going to be hard to get right vs. shared interrupts no ?

I think the "other" solution overall is much more simple.

No, it is not and the reason is the ACPI ordering (sorry for repeating myself).

Thanks,
Rafael

#72 lore raw
In reply to Linus Torvalds
Quoted 30 lines · Linus Torvalds
Quoted 23 lines · Rafael J. Wysocki

As I said, I tend to prefer the "loop of disable_irq()" approach, because it would allow us to preserve the current ordering of ACPI operations. Namely, if we do:

suspend devices (normal suspend)
loop of disable_irq()
late suspend of devices
_PTS
disable nonboot CPUs
local_irq_disable()
sysdev suspend
enter sleep state
get control from the BIOS
sysdev resume
(*)
local_irq_enable()
enable nonboot CPUs
_WAK
early resume of devices
loop of enable_irq()
resume devices (normal resume)

Quoted 4 lines

the ordering of _PTS with respect to putting devices into low power states and disabling the nonboot CPUs will be the same as it is now and the same applies to _WAK and putting devices into D0 etc. (I really _really_ wouldn't like to change this ordering, since this alone is likely to break things badly).

Quoted 6 lines

Yes.

Also, make the "loop of disable/enable_irq()" phase be a helper function that also sets system_state to SYSTEM_SUSPENDING/SYSTEM_RUNNING respectively, and it should all be pretty clean, and the changes really should be pretty minimal.

OK

So, I'm going to implement something along these lines. We'll see how it works out.

Thanks,
Rafael

#83 lore raw
In reply to Ingo Molnar
Quoted 8 lines · Ingo Molnar
Quoted 2 lines · Linus Torvalds

Ingo: how do we walk over all the interrupt descriptors in todays world? The whole sparse-vs-nonsparse makes things a bit more complex.

this:

Quoted 8 lines
        struct irq_desc *desc;
        int irq;

        for_each_irq_desc(i, desc) {
	...
	}

should do the trick.

So, what do I do in the loop to disable irqs for all devices on the IOAPIC level. Would disable_irq(irq) be sufficient?

Also, how can I skip the timer interrupt (we don't want to disable that)?

Rafael

#58 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Rafael J. Wysocki
Quoted 2 lines · Rafael J. Wysocki

That would change the ordering of ACPI method calls, which also is important and prone to breaking, as I wrote in the original message.

Ok, I'm not -that- familiar with ACPI, but I don't see where this ordering change you seem to fear is ... Ie, what gets re-ordered vs. what ?

Ben.

#59 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

Quoted 8 lines · Benjamin Herrenschmidt
Quoted 2 lines · Rafael J. Wysocki

That would change the ordering of ACPI method calls, which also is important and prone to breaking, as I wrote in the original message.

Ok, I'm not -that- familiar with ACPI, but I don't see where this ordering change you seem to fear is ... Ie, what gets re-ordered vs. what ?

(Newer) ACPI says that devices should be put into low power states (presumably with the help of appropriate ACPI AML routines) before the _PTS method is called. In turn, we're supposed to disable nonboot CPUs after calling _PTS. There is analogous requirement for the _WAK method during resume.

Currently, the suspend code ordering follows these rules, but if we move the putting of devices into low power states into the suspend_late part, they will have to be done after _PTS and that is likely to break things (we've already had this problem once and I have really bad memories related to it).

Thanks,
Rafael

#29 lore raw
In reply to Linus Torvalds
Quoted 2 lines

You've found a bug somewhere.

Yup :-)

Quoted 15 lines

We _should_ be saving things, the legacy code does something like this:

        if (drv && drv->suspend) {
                pci_dev->state_saved = false;

                i = drv->suspend(pci_dev, state);
                suspend_report_result(drv->suspend, i);
                if (i)
                        return i;

                if (pci_dev->state_saved)
                        goto Fixup;

                if (WARN_ON_ONCE(pci_dev->current_state != PCI_D0))
                        goto Fixup;

It looks like the above is what breaks. Looks like current_state is UNKNOWN. The device is an old mach64 that has no PCI PM capability, thus the driver doesn't call any PCI PM stuff, the state basically stays set to what the core set it to at probe time which appears to be PCI_UNKNOWN.

Thus we don't call pci_save_state().

Then ...

Quoted 8 lines
        }

        pci_save_state(pci_dev);

ie if your ->suspend function doesn't use pci_save_state() itself (which sets that "state_saved" flag to true), then the generic code will do it for you.

Quoted 7 lines

Also, on the resume path, we actually have

        if (pci_dev->state_saved)
                pci_restore_standard_config(pci_dev);

so I wonder how the heck you got that blast of all zeroes - because we clearly shouldn't be trying to restore any unsaved state!

Well, that's it ... we don't actually test pci_dev->state_saved in whatever is currently upstream. The code is:

static void pci_pm_default_resume_noirq(struct pci_dev *pci_dev)

{
	pci_restore_standard_config(pci_dev);
	pci_fixup_device(pci_fixup_resume_early, pci_dev);
}

Oops...

Rafael, the second one is trivial to fix, but what about the first one ? Should we not count UNKNOWN in that goto or should we set legacy stuff that don't do PCI PM to PCI_D0 somewhere ? Or both ? :-)

Cheers,
Ben.

#60 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Rafael J. Wysocki
Quoted 10 lines

(Newer) ACPI says that devices should be put into low power states (presumably with the help of appropriate ACPI AML routines) before the _PTS method is called. In turn, we're supposed to disable nonboot CPUs after calling _PTS. There is analogous requirement for the _WAK method during resume.

Currently, the suspend code ordering follows these rules, but if we move the putting of devices into low power states into the suspend_late part, they will have to be done after _PTS and that is likely to break things (we've already had this problem once and I have really bad memories related to it).

Wait wait wait ... the -whole- point of the exercise, wether using local_irq_save or disable_irq, -is- to put the ACPI bit -after- setting the device in low power state and before the restore on wakeup...

So basically, that isn't changing.

The -one- thing that indeed conflicts here is that we disable nonboot CPUs earlier. Right ?

Now, I doubt that would be a big issue, ie, we are supposedly capable of dynamically disabling/enabling CPUs anyway, but if it is, then indeed I see how the using of higher level PIC irq disabling would allow to move the whole suspend_late() over to before disabling non-boot CPUs.

That does introduce a significant change in semantics for drivers in the sense that now, suspend_late will be called with timers running, things scheduling, requests coming in, etc... One of the big reasons for doing suspend_late with IRQs off was precisely that drivers wouldn't have to synchronize with all these things. They now do.

Thus I tend to think that keeping the disabling of nonboot CPUs earlier than the suspending of devices is the least of two evils. _BUT_ As I said, I'm no ACPI expert and not -that- familiar with x86 land, it might indeed be a can of worms.

Cheers,
Ben.

#84 lore raw
In reply to Rafael J. Wysocki
Quoted 4 lines · Rafael J. Wysocki
Quoted 4 lines
        struct irq_desc *desc;
        int irq;
Quoted 8 lines
Quoted 5 lines
        for_each_irq_desc(i, desc) {
	...
	}

should do the trick.

So, what do I do in the loop to disable irqs for all devices on the IOAPIC level. Would disable_irq(irq) be sufficient?

Yes. It literally should be something as simple as

	void disable_device_irqs(void)
	{
		struct irq_desc *desc;
		int i;

		for_each_irq_desc(i, desc)
			disable_irq(i);
	}

	void enable_device_irqs(void)
	{
		struct irq_desc *desc;
		int i;

		for_each_irq_desc(i, desc)
			enable_irq(i);
	}

although we might do something smarter eventually (ie start adding tests for specific flags etc - at that point we'll also need to mark which interrupts we've disabled so that we get the disable/enable nesting right).

And it probably makes sense to do this inside kernel/irq/manage.c or similar, since any extensions are going to be very aware of irq internals. For example, the thing might well end up taking the desc->lock and peek inside the descriptor to see if it's really a "device" interrupt or a "system" interrupt etc.

Linus

#61 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Benjamin Herrenschmidt
Quoted 4 lines · Benjamin Herrenschmidt

Wait wait wait ... the -whole- point of the exercise, wether using local_irq_save or disable_irq, -is- to put the ACPI bit -after- setting the device in low power state and before the restore on wakeup...

You're missing context.

ACPI isn't just for the device power settings. It's for the CPU's too, and it's for things like "prepare to sleep". And they all have _different_ requirements.

What Rafael is trying to tell you is that we have ACPI-initiated ordering requirements between turning CPU's off, and turning devices off, and he's not willing to change that.

Quoted 2 lines

The -one- thing that indeed conflicts here is that we disable nonboot CPUs earlier. Right ?

This.

Quoted 1 line

Now, I doubt that would be a big issue

With ACPI, there is no such thing as a "big issue". There are only tons of small horrid details that. And the "big issue" is that all the small details are insane.

Linus

#30 lore raw
In reply to Benjamin Herrenschmidt
Quoted 18 lines · Benjamin Herrenschmidt
Quoted 2 lines

You've found a bug somewhere.

Yup :-)

Quoted 13 lines

We _should_ be saving things, the legacy code does something like this:

        if (drv && drv->suspend) {
                pci_dev->state_saved = false;

                i = drv->suspend(pci_dev, state);
                suspend_report_result(drv->suspend, i);
                if (i)
                        return i;

                if (pci_dev->state_saved)
                        goto Fixup;
Quoted 8 lines
Quoted 2 lines
                if (WARN_ON_ONCE(pci_dev->current_state != PCI_D0))
                        goto Fixup;

It looks like the above is what breaks. Looks like current_state is UNKNOWN. The device is an old mach64 that has no PCI PM capability, thus the driver doesn't call any PCI PM stuff, the state basically stays set to what the core set it to at probe time which appears to be PCI_UNKNOWN.

Overlooked, sorry.

Quoted 23 lines

Thus we don't call pci_save_state().

Then ...

Quoted 8 lines
        }

        pci_save_state(pci_dev);

ie if your ->suspend function doesn't use pci_save_state() itself (which sets that "state_saved" flag to true), then the generic code will do it for you.

Quoted 7 lines

Also, on the resume path, we actually have

        if (pci_dev->state_saved)
                pci_restore_standard_config(pci_dev);

so I wonder how the heck you got that blast of all zeroes - because we clearly shouldn't be trying to restore any unsaved state!

Well, that's it ... we don't actually test pci_dev->state_saved in whatever is currently upstream. The code is:

Quoted 7 lines

static void pci_pm_default_resume_noirq(struct pci_dev *pci_dev)

{
	pci_restore_standard_config(pci_dev);
	pci_fixup_device(pci_fixup_resume_early, pci_dev);
}

Oops...

The assumption here is that the state will be saved either by the driver or by the core, so the bug here is a consequence of the previous one.

I'll add a check.

Quoted 3 lines

Rafael, the second one is trivial to fix, but what about the first one ? Should we not count UNKNOWN in that goto or should we set legacy stuff that don't do PCI PM to PCI_D0 somewhere ? Or both ? :-)

Well, if the state is UNKNOWN, I think it's safe to save the config, so the WARN_ON should really catch the low power states only. I'll fix that.

Thanks for debugging it!

Rafael

#67 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Benjamin Herrenschmidt
Quoted 25 lines · Benjamin Herrenschmidt
Quoted 10 lines

(Newer) ACPI says that devices should be put into low power states (presumably with the help of appropriate ACPI AML routines) before the _PTS method is called. In turn, we're supposed to disable nonboot CPUs after calling _PTS. There is analogous requirement for the _WAK method during resume.

Currently, the suspend code ordering follows these rules, but if we move the putting of devices into low power states into the suspend_late part, they will have to be done after _PTS and that is likely to break things (we've already had this problem once and I have really bad memories related to it).

Wait wait wait ... the -whole- point of the exercise, wether using local_irq_save or disable_irq, -is- to put the ACPI bit -after- setting the device in low power state and before the restore on wakeup...

So basically, that isn't changing.

The -one- thing that indeed conflicts here is that we disable nonboot CPUs earlier. Right ?

Now, I doubt that would be a big issue, ie, we are supposedly capable of dynamically disabling/enabling CPUs anyway, but if it is, then indeed I see how the using of higher level PIC irq disabling would allow to move the whole suspend_late() over to before disabling non-boot CPUs.

Quoted 5 lines

That does introduce a significant change in semantics for drivers in the sense that now, suspend_late will be called with timers running, things scheduling, requests coming in, etc... One of the big reasons for doing suspend_late with IRQs off was precisely that drivers wouldn't have to synchronize with all these things. They now do.

I realize the problem, but IMO only a few drivers will be affected, since only a few of them implement suspend_late/resume_early.

Quoted 4 lines

Thus I tend to think that keeping the disabling of nonboot CPUs earlier than the suspending of devices is the least of two evils. _BUT_ As I said, I'm no ACPI expert and not -that- familiar with x86 land, it might indeed be a can of worms.

It is. :-)

Thanks,
Rafael

#62 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

With ACPI, there is no such thing as a "big issue". There are only tons of small horrid details that. And the "big issue" is that all the small details are insane.

Allright, fair enough :-) Here's my grand plan defeated by the uglyness of ACPI ... argh ! :-)

In any case, both approach will solve the problem at hand and I'll be able to move some of that pmac uglyness into the platform hooks.

BTW. Do we want one or two system states ? IE. regarding silently "upgrading" GFP_KERNEL, I believe it needs to be done in two different phases. The main suspend loop (after all the prepare() calls complete) should be done with GFP_KERNEL -> GFP_NOIO, and the last phase with IRQs really disabled (local_irq_save, aka. sysdev's) could use being done with GFP_KERNEL -> GFP_ATOMIC.

The later will not be used much, but with the PCI layer now doing kmalloc GFP_KERNEL all over the place in trivial things like pci_get_*, I think it's worth it as sysdev's like cpufreq, or other low level arch bits like memory controller tweaking does involve mucking around with PCI devices.

Cheers,
Ben.

#68 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki

I realize the problem, but IMO only a few drivers will be affected, since only a few of them implement suspend_late/resume_early.

Also, since we are fixing the irq issues and the basic PCI suspend/resume logic to be handled by the generic code, I suspect a lot of the drivers that _do_ the late/early stuff will be happy to simply get rid of them.

Linus

#53 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Thomas Gleixner
Quoted 6 lines · Thomas Gleixner
Quoted 5 lines · Ingo Molnar

So i'd still like your tentative Signed-off-by for your patch - it's i think not v2.6.29 material but if it stays problem free in testing we can try it in v2.6.30. If it causes problem it will be clearly bisectable and clearly revertable.

Quoted 8 lines
Quoted 4 lines

Maybe we could split it in two: and for MSI we could introduce a 'simpler and faster' edge flow as well - and keep the legacy handler untouched. That way it's low-risk in its entirety. (and avoids the MSI ->mask complication as well.)

Yes, seperating out the MSI handler into it's own flow control is the right way to go. We had trouble with real edge hardware and IIRC most of the problems originated from ARM. rmk ??

No idea, too long ago.

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:
#93 lore raw
In reply to Rafael J. Wysocki

Hi!

Quoted 9 lines
Quoted 1 line

I'll have a look at the ACPI thing.

Generally speaking, we'd need to run acpi_evaluate_object() with interrupts off.

There are two apparent problems with that, from a quick look:

* The ACPI_MTX_INTERPRETER mutex needs to be acquired, but we know we won't need that mutex with interrupts off, so presumably we can work around this.

In such case we either need refrigerator, or we should grab MTX_INTERPRETER before we start suspending, hmm.

Quoted 6 lines

* Memory allocations with GFP_KERNEL are made, which is even worse, because we really shouldn't do that during suspend _at_ _all_, even during the regular ->suspend() with interrupts on, because there's not guarantee that swap will will be available at that time. So, for the sake of correctness, we should get rid of the GFP_KERNEL from the ACPI code paths executed during suspend-resume anyway.

Well, we have some small reserves for stuff like that...
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#97 lore raw

kmalloc during suspend, was Re: PCI PM: Restore standard config registers of all devices early

In reply to Benjamin Herrenschmidt
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 7 lines
Quoted 6 lines

(*) There are reasons to think that kmalloc/gfp should both silently turn into GFP_NOIO always while the suspend process is started, but that's somewhat a different subject. Rafael, did we ever act on that ? It's an old discussion we had but I don't know if we actually implemented anything.

Quoted 5 lines
Quoted 2 lines

We have the ->prepare(), ->complete() callbacks that, among other things, can be used for allocating and freeing memory with GFP_KERNEL safely.

First, that's assuming drivers will be smart enough to figure that out. They won't, believe me.

For big alocations, I see no other way. If someone wants 100MB for firmmware image...

Quoted 4 lines

Then, as I said, this doesn't work in practice because there is no way drivers and/or underlying subsystems will start "remembering" when they are within a prepare/complete pair, and suddenly do allocations differently. That's just going to break.

They could set a flag in ->prepare()... but yes, drivers will get that wrong.

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#63 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Benjamin Herrenschmidt

Wait wait wait ... the -whole- point of the exercise, wether using local_irq_save or disable_irq, -is- to put the ACPI bit -after- setting the device in low power state and before the restore on wakeup...

Quoted 5 lines

You're missing context.

ACPI isn't just for the device power settings. It's for the CPU's too, and it's for things like "prepare to sleep". And they all have _different_ requirements.

As long as we claim to support cpu hotplug, suspending the cpus early should work. User might have unplugged the cpu manually...

If something is broken in acpi, we may need to disable cpu*/onlone on
affected systems.
Pavel

Quoted 22 lines

What Rafael is trying to tell you is that we have ACPI-initiated ordering requirements between turning CPU's off, and turning devices off, and he's not willing to change that.

Quoted 2 lines

The -one- thing that indeed conflicts here is that we disable nonboot CPUs earlier. Right ?

This.

Quoted 1 line

Now, I doubt that would be a big issue

With ACPI, there is no such thing as a "big issue". There are only tons of small horrid details that. And the "big issue" is that all the small details are insane.

Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#64 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

Quoted 6 lines

As long as we claim to support cpu hotplug, suspending the cpus early should work. User might have unplugged the cpu manually...

If something is broken in acpi, we may need to disable cpu*/onlone on
affected systems.

You seem to miss some context here... it seems that ACPI mandates that CPUs are suspended after devices and you know that we can't just start blacklisting half of the machines out there just because we happen not to do the same as what Windows does...

So let's be on the safe side.

Cheers,
Ben

#65 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

Quoted 11 lines · Benjamin Herrenschmidt
Quoted 6 lines

As long as we claim to support cpu hotplug, suspending the cpus early should work. User might have unplugged the cpu manually...

If something is broken in acpi, we may need to disable cpu*/onlone on
affected systems.

You seem to miss some context here... it seems that ACPI mandates that CPUs are suspended after devices and you know that we can't just start blacklisting half of the machines out there just because we happen not to do the same as what Windows does...

I did not see that ACPI mandate... where is it?

That would interfere with user's ability to online/offline cpus,
badly.

Are you saying that we should online all the CPUs before starting suspend to stay on the "safe side"?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#66 lore raw

Re: Reworking suspend-resume sequence (was: Re: PCI PM: Restore standard config registers of all devices early)

Quoted 8 lines · Pavel Machek
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 6 lines

As long as we claim to support cpu hotplug, suspending the cpus early should work. User might have unplugged the cpu manually...

If something is broken in acpi, we may need to disable cpu*/onlone on
affected systems.

Quoted 6 lines
Quoted 4 lines

You seem to miss some context here... it seems that ACPI mandates that CPUs are suspended after devices and you know that we can't just start blacklisting half of the machines out there just because we happen not to do the same as what Windows does...

I did not see that ACPI mandate... where is it?

15.1.6 in ACPI 3.0b (for example).

This is more complicated, though. Turning the CPUs on/off involves things that need to be ordered with respect to _PTS, _WAK in a specific way (ie. just like we do it right now, which is _PTS before disable_nonboot_cpus() and enable_nonboot_cpus() before _WAK). Otherwise at least some machines will break (verified experimentally some time ago).

Thanks,
Rafael