Skip to contentSkip to conversation

PHY interruptus horribilis

PATCH Superseded Lukas Wunner 2 participants 5 messages Jun 8, 2022

Subsystems (4)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#1 lore raw

PATCH v2 1/1 net: phy: Don't trigger state machine while in suspend

In reply to Lukas Wunner

Upon system sleep, mdio_bus_phy_suspend() stops the phy_state_machine(), but subsequent interrupts may retrigger it:

They may have been left enabled to facilitate wakeup and are not quiesced until the ->suspend_noirq() phase. Unwanted interrupts may hence occur between mdio_bus_phy_suspend() and dpm_suspend_noirq(), as well as between dpm_resume_noirq() and mdio_bus_phy_resume().

Retriggering the phy_state_machine() through an interrupt is not only undesirable for the reason given in mdio_bus_phy_suspend() (freezing it midway with phydev->lock held), but also because the PHY may be inaccessible after it's suspended: Accesses to USB-attached PHYs are blocked once usb_suspend_both() clears the can_submit flag and PHYs on PCI network cards may become inaccessible upon suspend as well.

Amend phy_interrupt() to avoid triggering the state machine if the PHY is suspended. Signal wakeup instead if the attached net_device or its parent has been configured as a wakeup source. (Those conditions are identical to mdio_bus_phy_may_suspend().) Postpone handling of the interrupt until the PHY has resumed.

Before stopping the phy_state_machine() in mdio_bus_phy_suspend(), wait for a concurrent phy_interrupt() to run to completion. That is necessary because phy_interrupt() may have checked the PHY's suspend status before the system sleep transition commenced and it may thus retrigger the state machine after it was stopped.

Likewise, after re-enabling interrupt handling in mdio_bus_phy_resume(), wait for a concurrent phy_interrupt() to complete to ensure that interrupts which it postponed are properly rerun.

---
 drivers/net/phy/phy.c        | 23 +++++++++++++++++++++++
 drivers/net/phy/phy_device.c | 23 +++++++++++++++++++++++
 include/linux/phy.h          |  6 ++++++
 3 files changed, 52 insertions(+)
drivers/net/phy/phy.c +23 -0
index ef62f357b76d..8d3ee3a6495b 100644
@@ -31,6 +31,7 @@
 #include <linux/io.h>
 #include <linux/uaccess.h>
 #include <linux/atomic.h>
+#include <linux/suspend.h>
 #include <net/netlink.h>
 #include <net/genetlink.h>
 #include <net/sock.h>
@@ -976,6 +977,28 @@ static irqreturn_t phy_interrupt(int irq, void *phy_dat)
 	struct phy_driver *drv = phydev->drv;
 	irqreturn_t ret;
 
+	/* Wakeup interrupts may occur during a system sleep transition.
+	 * Postpone handling until the PHY has resumed.
+	 */
+	if (IS_ENABLED(CONFIG_PM_SLEEP) && phydev->irq_suspended) {
+		struct net_device *netdev = phydev->attached_dev;
+
+		if (netdev) {
+			struct device *parent = netdev->dev.parent;
+
+			if (netdev->wol_enabled)
+				pm_system_wakeup();
+			else if (device_may_wakeup(&netdev->dev))
+				pm_wakeup_dev_event(&netdev->dev, 0, true);
+			else if (parent && device_may_wakeup(parent))
+				pm_wakeup_dev_event(parent, 0, true);
+		}
+
+		phydev->irq_rerun = 1;
+		disable_irq_nosync(irq);
+		return IRQ_HANDLED;
+	}
+
 	mutex_lock(&phydev->lock);
 	ret = drv->handle_interrupt(phydev);
 	mutex_unlock(&phydev->lock);
drivers/net/phy/phy_device.c +23 -0
index 431a8719c635..46acddd865a7 100644
@@ -278,6 +278,15 @@ static __maybe_unused int mdio_bus_phy_suspend(struct device *dev)
 	if (phydev->mac_managed_pm)
 		return 0;
 
+	/* Wakeup interrupts may occur during the system sleep transition when
+	 * the PHY is inaccessible. Set flag to postpone handling until the PHY
+	 * has resumed. Wait for concurrent interrupt handler to complete.
+	 */
+	if (phy_interrupt_is_valid(phydev)) {
+		phydev->irq_suspended = 1;
+		synchronize_irq(phydev->irq);
+	}
+
 	/* We must stop the state machine manually, otherwise it stops out of
 	 * control, possibly with the phydev->lock held. Upon resume, netdev
 	 * may call phy routines that try to grab the same lock, and that may
@@ -315,6 +324,20 @@ static __maybe_unused int mdio_bus_phy_resume(struct device *dev)
 	if (ret < 0)
 		return ret;
 no_resume:
+	if (phy_interrupt_is_valid(phydev)) {
+		phydev->irq_suspended = 0;
+		synchronize_irq(phydev->irq);
+
+		/* Rerun interrupts which were postponed by phy_interrupt()
+		 * because they occurred during the system sleep transition.
+		 */
+		if (phydev->irq_rerun) {
+			phydev->irq_rerun = 0;
+			enable_irq(phydev->irq);
+			irq_wake_thread(phydev->irq, phydev);
+		}
+	}
+
 	if (phydev->attached_dev && phydev->adjust_link)
 		phy_start_machine(phydev);
 
include/linux/phy.h +6 -0
index 508f1149665b..b09f7d36cff2 100644
@@ -572,6 +572,10 @@ struct macsec_ops;
  * @mdix_ctrl: User setting of crossover
  * @pma_extable: Cached value of PMA/PMD Extended Abilities Register
  * @interrupts: Flag interrupts have been enabled
+ * @irq_suspended: Flag indicating PHY is suspended and therefore interrupt
+ *                 handling shall be postponed until PHY has resumed
+ * @irq_rerun: Flag indicating interrupts occurred while PHY was suspended,
+ *             requiring a rerun of the interrupt handler after resume
  * @interface: enum phy_interface_t value
  * @skb: Netlink message for cable diagnostics
  * @nest: Netlink nest used for cable diagnostics
@@ -626,6 +630,8 @@ struct phy_device {
 
 	/* Interrupts are enabled */
 	unsigned interrupts:1;
+	unsigned irq_suspended:1;
+	unsigned irq_rerun:1;
 
 	enum phy_state state;
 
Signature · 1 line
2.35.2
Acked-by on v2 1/1
#2 lore raw

Re: [PATCH net v2 1/1] net: phy: Don't trigger state machine while in suspend

Quoted 28 lines · Lukas Wunner

Upon system sleep, mdio_bus_phy_suspend() stops the phy_state_machine(), but subsequent interrupts may retrigger it:

They may have been left enabled to facilitate wakeup and are not quiesced until the ->suspend_noirq() phase. Unwanted interrupts may hence occur between mdio_bus_phy_suspend() and dpm_suspend_noirq(), as well as between dpm_resume_noirq() and mdio_bus_phy_resume().

Retriggering the phy_state_machine() through an interrupt is not only undesirable for the reason given in mdio_bus_phy_suspend() (freezing it midway with phydev->lock held), but also because the PHY may be inaccessible after it's suspended: Accesses to USB-attached PHYs are blocked once usb_suspend_both() clears the can_submit flag and PHYs on PCI network cards may become inaccessible upon suspend as well.

Amend phy_interrupt() to avoid triggering the state machine if the PHY is suspended. Signal wakeup instead if the attached net_device or its parent has been configured as a wakeup source. (Those conditions are identical to mdio_bus_phy_may_suspend().) Postpone handling of the interrupt until the PHY has resumed.

Before stopping the phy_state_machine() in mdio_bus_phy_suspend(), wait for a concurrent phy_interrupt() to run to completion. That is necessary because phy_interrupt() may have checked the PHY's suspend status before the system sleep transition commenced and it may thus retrigger the state machine after it was stopped.

Quoted 8 lines

Likewise, after re-enabling interrupt handling in mdio_bus_phy_resume(), wait for a concurrent phy_interrupt() to complete to ensure that interrupts which it postponed are properly rerun.

Looks reasonable to me.

Quoted 115 lines
---
 drivers/net/phy/phy.c        | 23 +++++++++++++++++++++++
 drivers/net/phy/phy_device.c | 23 +++++++++++++++++++++++
 include/linux/phy.h          |  6 ++++++
 3 files changed, 52 insertions(+)
drivers/net/phy/phy.c +23 -0
index ef62f357b76d..8d3ee3a6495b 100644
@@ -31,6 +31,7 @@
 #include <linux/io.h>
 #include <linux/uaccess.h>
 #include <linux/atomic.h>
+#include <linux/suspend.h>
 #include <net/netlink.h>
 #include <net/genetlink.h>
 #include <net/sock.h>
@@ -976,6 +977,28 @@ static irqreturn_t phy_interrupt(int irq, void *phy_dat)
        struct phy_driver *drv = phydev->drv;
        irqreturn_t ret;
 
+       /* Wakeup interrupts may occur during a system sleep transition.
+        * Postpone handling until the PHY has resumed.
+        */
+       if (IS_ENABLED(CONFIG_PM_SLEEP) && phydev->irq_suspended) {
+               struct net_device *netdev = phydev->attached_dev;
+
+               if (netdev) {
+                       struct device *parent = netdev->dev.parent;
+
+                       if (netdev->wol_enabled)
+                               pm_system_wakeup();
+                       else if (device_may_wakeup(&netdev->dev))
+                               pm_wakeup_dev_event(&netdev->dev, 0, true);
+                       else if (parent && device_may_wakeup(parent))
+                               pm_wakeup_dev_event(parent, 0, true);
+               }
+
+               phydev->irq_rerun = 1;
+               disable_irq_nosync(irq);
+               return IRQ_HANDLED;
+       }
+
        mutex_lock(&phydev->lock);
        ret = drv->handle_interrupt(phydev);
        mutex_unlock(&phydev->lock);
drivers/net/phy/phy_device.c +23 -0
index 431a8719c635..46acddd865a7 100644
@@ -278,6 +278,15 @@ static __maybe_unused int mdio_bus_phy_suspend(struct device *dev)
        if (phydev->mac_managed_pm)
                return 0;
 
+       /* Wakeup interrupts may occur during the system sleep transition when
+        * the PHY is inaccessible. Set flag to postpone handling until the PHY
+        * has resumed. Wait for concurrent interrupt handler to complete.
+        */
+       if (phy_interrupt_is_valid(phydev)) {
+               phydev->irq_suspended = 1;
+               synchronize_irq(phydev->irq);
+       }
+
        /* We must stop the state machine manually, otherwise it stops out of
         * control, possibly with the phydev->lock held. Upon resume, netdev
         * may call phy routines that try to grab the same lock, and that may
@@ -315,6 +324,20 @@ static __maybe_unused int mdio_bus_phy_resume(struct device *dev)
        if (ret < 0)
                return ret;
 no_resume:
+       if (phy_interrupt_is_valid(phydev)) {
+               phydev->irq_suspended = 0;
+               synchronize_irq(phydev->irq);
+
+               /* Rerun interrupts which were postponed by phy_interrupt()
+                * because they occurred during the system sleep transition.
+                */
+               if (phydev->irq_rerun) {
+                       phydev->irq_rerun = 0;
+                       enable_irq(phydev->irq);
+                       irq_wake_thread(phydev->irq, phydev);
+               }
+       }
+
        if (phydev->attached_dev && phydev->adjust_link)
                phy_start_machine(phydev);
 
include/linux/phy.h +6 -1
index 508f1149665b..b09f7d36cff2 100644
@@ -572,6 +572,10 @@ struct macsec_ops;
  * @mdix_ctrl: User setting of crossover
  * @pma_extable: Cached value of PMA/PMD Extended Abilities Register
  * @interrupts: Flag interrupts have been enabled
+ * @irq_suspended: Flag indicating PHY is suspended and therefore interrupt
+ *                 handling shall be postponed until PHY has resumed
+ * @irq_rerun: Flag indicating interrupts occurred while PHY was suspended,
+ *             requiring a rerun of the interrupt handler after resume
  * @interface: enum phy_interface_t value
  * @skb: Netlink message for cable diagnostics
  * @nest: Netlink nest used for cable diagnostics
@@ -626,6 +630,8 @@ struct phy_device {
 
        /* Interrupts are enabled */
        unsigned interrupts:1;
+       unsigned irq_suspended:1;
+       unsigned irq_rerun:1;
 
        enum phy_state state;
 
--

2.35.2

#0 lore raw

v2 cover PHY interruptus horribilis

Andrew Lunn (PHY maintainer) asked me to resend this patch and cc the IRQ maintainer. I'm also cc'ing PM maintainers for good measure.

The patch addresses an issue with PHY interrupts occurring during a system sleep transition after the PHY has already been suspended.

The IRQ subsystem uses an internal flag IRQD_WAKEUP_ARMED to avoid handling such interrupts, but it's not set until suspend_device_irqs() is called during the ->suspend_noirq() phase. That's too late in this case as PHYs are suspended in the ->suspend() phase. And there's no external interface to set the flag earlier.

As I'm lacking access to the flag, I'm open coding its functionality in this patch. Is this the correct approach or should I instead look into providing an external interface to the flag?

Side note: suspend_device_irqs() and resume_device_irqs() have been exported since forever even though there's no module user...

Thanks!

Changes since v1:
* Extend rationale in the commit message.
* Drop Fixes tag, add Tested-by tag (Marek).

Link to v1:
https://lore.kernel.org/netdev/688f559346ea747d3b47a4d16ef8277e093f9ebe.1653556322.git.lukas@wunner.de/

Lukas Wunner (1):
net: phy: Don't trigger state machine while in suspend

 drivers/net/phy/phy.c        | 23 +++++++++++++++++++++++
 drivers/net/phy/phy_device.c | 23 +++++++++++++++++++++++
 include/linux/phy.h          |  6 ++++++
 3 files changed, 52 insertions(+)
Signature · 1 line
2.35.2
#3 lore raw
Quoted 4 lines · Lukas Wunner

Andrew Lunn (PHY maintainer) asked me to resend this patch and cc the IRQ maintainer. I'm also cc'ing PM maintainers for good measure.

Quoted 8 lines

The patch addresses an issue with PHY interrupts occurring during a system sleep transition after the PHY has already been suspended.

The IRQ subsystem uses an internal flag IRQD_WAKEUP_ARMED to avoid handling such interrupts, but it's not set until suspend_device_irqs() is called during the ->suspend_noirq() phase. That's too late in this case as PHYs are suspended in the ->suspend() phase. And there's no external interface to set the flag earlier.

Yes, it is not there intentionally.

Strictly speaking, IRQD_WAKEUP_ARMED is there to indicate to the IRQ subsystem that the given IRQ is a system wakeup one and has been left enabled specifically in order to signal system wakeup. It allows the IRQ to trigger between suspend_device_irqs() and resume_device_irqs() exactly once, which causes the system to wake up from suspend-to-idle (that's the primary use case for it) or aborts system suspends in progress.

As you have noticed, it is set automatically by suspend_device_irqs() if the given IRQ has IRQD_WAKEUP_STATE which is the case when it has been enabled for system wakeup.

Quoted 3 lines

As I'm lacking access to the flag, I'm open coding its functionality in this patch. Is this the correct approach or should I instead look into providing an external interface to the flag?

The idea is that the regular IRQ "action" handler will run before suspend_device_irqs(), so it should take care of signaling wakeup if need be. IRQD_WAKEUP_ARMED to trigger wakeup when IRQ "action" handlers don't run.

Quoted 2 lines

Side note: suspend_device_irqs() and resume_device_irqs() have been exported since forever even though there's no module user...

Well, that's a mistake.

#4 lore raw
Quoted 29 lines · Rafael J. Wysocki
Quoted 4 lines · Lukas Wunner

Andrew Lunn (PHY maintainer) asked me to resend this patch and cc the IRQ maintainer. I'm also cc'ing PM maintainers for good measure.

Quoted 8 lines

The patch addresses an issue with PHY interrupts occurring during a system sleep transition after the PHY has already been suspended.

The IRQ subsystem uses an internal flag IRQD_WAKEUP_ARMED to avoid handling such interrupts, but it's not set until suspend_device_irqs() is called during the ->suspend_noirq() phase. That's too late in this case as PHYs are suspended in the ->suspend() phase. And there's no external interface to set the flag earlier.

Yes, it is not there intentionally.

Strictly speaking, IRQD_WAKEUP_ARMED is there to indicate to the IRQ subsystem that the given IRQ is a system wakeup one and has been left enabled specifically in order to signal system wakeup. It allows the IRQ to trigger between suspend_device_irqs() and resume_device_irqs() exactly once, which causes the system to wake up from suspend-to-idle (that's the primary use case for it) or aborts system suspends in progress.

As you have noticed, it is set automatically by suspend_device_irqs() if the given IRQ has IRQD_WAKEUP_STATE which is the case when it has been enabled for system wakeup.

Quoted 8 lines
Quoted 3 lines

As I'm lacking access to the flag, I'm open coding its functionality in this patch. Is this the correct approach or should I instead look into providing an external interface to the flag?

The idea is that the regular IRQ "action" handler will run before suspend_device_irqs(), so it should take care of signaling wakeup if need be. IRQD_WAKEUP_ARMED to trigger wakeup when IRQ "action" handlers don't run.

That said IMV there could be a wrapper around suspend_device_irq() to mark a specific IRQ as "suspended" before running suspend_device_irqs(), but that would require adding "suspend depth" to struct irq_desc, so the IRQ is not "resumed" prematurely by resume_device_irqs(). And there needs to be an analogous wrapper around resume_irq() for the resume path.

Does the single prospective user justify this?