Skip to contentSkip to conversation

net: phy: add device link between MAC device and MDIO device

PATCH In review Wei Fang 5 participants 15 messages Jan 26 – Jan 30

Subsystems (4)
lore
Share & tools
lore

Wei Fang

author
#0 lore raw

PATCH 1/1 net: phy: add device link between MAC device and MDIO device

For the shared MDIO bus use case, multiple MACs will share the same MDIO bus. Therefore, these MACs all depend on this MDIO bus, however, current PHY driver does not create this dependency relationship between the MACs and the shared MDIO devices. If this shared MDIO bus is removed, it could potentially cause a crash. For example, the i.MX6ULL platform has two FEC ports (fec1 and fec2) and uses phylib interfaces to manage the external PHY. And fec1 uses the MDIO bus shared by fec2. When the fec2 driver is removed, users can see the following warning log.

$ echo 20b4000.ethernet > /sys/bus/platform/drivers/fec/unbind
WARNING: drivers/net/phy/phy.c:1352 at _phy_state_machine+0xe8/0x38c, CPU#0: kworker/0:6/35
phy_check_link_status+0x0/0xf8: returned: -13
Call trace:
 unwind_backtrace from show_stack+0x10/0x14
 show_stack from dump_stack_lvl+0x54/0x68
 dump_stack_lvl from __warn+0x84/0xf4
 __warn from warn_slowpath_fmt+0x1a8/0x1bc
 warn_slowpath_fmt from _phy_state_machine+0xe8/0x38c
 _phy_state_machine from phy_state_machine+0x20/0x50
 phy_state_machine from process_one_work+0x168/0x2f4

Another example is the i.MX95-15x15 platform which has two ENETC ports and uses the phylink interfaces to manage the exteranl PHY. All the external PHYs can be accessed the EMDIO, it is a different device from ENETC. When the EMDIO driver is removed, users can see the below crash log and the console is hanged.

$ echo 0002:01:00.0 > /sys/bus/pci/drivers/fsl_enetc_mdio/unbind
$ ifconfig eth0 down
Unable to handle kernel NULL pointer dereference at virtual address 00000000000001b8
pc : _phy_state_machine+0x230/0x36c
lr : _phy_state_machine+0x2b4/0x36c
Call trace:
 _phy_state_machine+0x230/0x36c (P)
 phy_stop+0x74/0x190
 phylink_stop+0x28/0xb8
 enetc_close+0x28/0x8c
 netif_change_flags+0x24/0x6c
 dev_change_flags+0x48/0x7c
 devinet_ioctl+0x328/0x604
 inet_ioctl+0x204/0x220

The commit bc66fa87d4fd ("net: phy: Add link between phy dev and mac dev") has created a device link between the MAC and the PHY if the MAC uses a shared MDIO bus (The MDIO and the MAC are two separate devices). Sarosh Hasan tried to change the DL_FLAG_STATELESS flag to DL_FLAG_AUTOREMOVE_SUPPLIER to fix the issue [1]. However, the solution does not take into account the hot-swappable PHY devices (such as SFP). so when the PHY device is unplugged, the MAC driver will automatically be removed, which is not the expected behavior.

Therefore, to solve this issue of the shared MDIO bus, we create the device link between the MAC device and the MDIO device, rather than between the MAC device and the PHY device. And when the shared MDIO bus is removed, all MAC drivers that depend on it will also be removed.

---
 drivers/net/phy/phy_device.c | 24 ++++++++++++++----------
 include/linux/phy.h          |  4 ----
 2 files changed, 14 insertions(+), 14 deletions(-)
drivers/net/phy/phy_device.c +14 -10
index 81984d4ebb7c..d5ac7506fe39 100644
@@ -1768,12 +1768,21 @@ int phy_attach_direct(struct net_device *dev, struct phy_device *phydev,
 
 	/**
 	 * If the external phy used by current mac interface is managed by
-	 * another mac interface, so we should create a device link between
-	 * phy dev and mac dev.
+	 * another MDIO controller, which means that the MAC and MDIO are
+	 * separated devices, then we should create a device link between
+	 * the MAC device and the MDIO device.
 	 */
-	if (dev && phydev->mdio.bus->parent && dev->dev.parent != phydev->mdio.bus->parent)
-		phydev->devlink = device_link_add(dev->dev.parent, &phydev->mdio.dev,
-						  DL_FLAG_PM_RUNTIME | DL_FLAG_STATELESS);
+	if (dev && phydev->mdio.bus->parent &&
+	    dev->dev.parent != phydev->mdio.bus->parent) {
+		if (!device_link_add(dev->dev.parent, phydev->mdio.bus->parent,
+				     DL_FLAG_PM_RUNTIME |
+				     DL_FLAG_AUTOREMOVE_SUPPLIER)) {
+			phydev_err(phydev,
+				   "Failed to add devlink between MAC and MDIO\n");
+			err = -EINVAL;
+			goto error;
+		}
+	}
 
 	return err;
 
@@ -1845,11 +1854,6 @@ void phy_detach(struct phy_device *phydev)
 	struct module *ndev_owner = NULL;
 	struct mii_bus *bus;
 
-	if (phydev->devlink) {
-		device_link_del(phydev->devlink);
-		phydev->devlink = NULL;
-	}
-
 	if (phydev->sysfs_links) {
 		if (dev)
 			sysfs_remove_link(&dev->dev.kobj, "phydev");
include/linux/phy.h +0 -4
index fbbe028cc4b7..d6f2039a63d3 100644
@@ -559,8 +559,6 @@ struct phy_oatc14_sqi_capability {
  *
  * @mdio: MDIO bus this PHY is on
  * @drv: Pointer to the driver for this PHY instance
- * @devlink: Create a link between phy dev and mac dev, if the external phy
- *           used by current mac interface is managed by another mac interface.
  * @phyindex: Unique id across the phy's parent tree of phys to address the PHY
  *	      from userspace, similar to ifindex. A zero index means the PHY
  *	      wasn't assigned an id yet.
@@ -666,8 +664,6 @@ struct phy_device {
 	/* And management functions */
 	const struct phy_driver *drv;
 
-	struct device_link *devlink;
-
 	u32 phyindex;
 	u32 phy_id;
 
Signature · 1 line
2.34.1
maintainer
#1 lore raw
Quoted 10 lines

The commit bc66fa87d4fd ("net: phy: Add link between phy dev and mac dev") has created a device link between the MAC and the PHY if the MAC uses a shared MDIO bus (The MDIO and the MAC are two separate devices).

Sarosh Hasan tried to change the DL_FLAG_STATELESS flag to DL_FLAG_AUTOREMOVE_SUPPLIER to fix the issue [1]. However, the solution does not take into account the hot-swappable PHY devices (such as SFP). so when the PHY device is unplugged, the MAC driver will automatically be removed, which is not the expected behavior.

Quoted 4 lines

Therefore, to solve this issue of the shared MDIO bus, we create the device link between the MAC device and the MDIO device, rather than between the MAC device and the PHY device. And when the shared MDIO bus is removed, all MAC drivers that depend on it will also be removed.

Shouldn't there be a full chain here? MAC->MDIO->PHY?

For the case of the SFP, maybe it should be

MAC->SFP Cage and I2C MDIO->PHY

So if the PHY inside an SFP disappears, the I2C MDIO bus is removed. But there is no link upwards from that, so the SFP cage and the MAC are left alone?

Andrew

#2 lore raw
Quoted 15 lines · Andrew Lunn
Quoted 10 lines

The commit bc66fa87d4fd ("net: phy: Add link between phy dev and mac dev") has created a device link between the MAC and the PHY if the MAC uses a shared MDIO bus (The MDIO and the MAC are two separate devices).

Sarosh Hasan tried to change the DL_FLAG_STATELESS flag to DL_FLAG_AUTOREMOVE_SUPPLIER to fix the issue [1]. However, the solution does not take into account the hot-swappable PHY devices (such as SFP). so when the PHY device is unplugged, the MAC driver will automatically be removed, which is not the expected behavior.

Quoted 4 lines

Therefore, to solve this issue of the shared MDIO bus, we create the device link between the MAC device and the MDIO device, rather than between the MAC device and the PHY device. And when the shared MDIO bus is removed, all MAC drivers that depend on it will also be removed.

Quoted 8 lines

Shouldn't there be a full chain here? MAC->MDIO->PHY?

For the case of the SFP, maybe it should be

MAC->SFP Cage and I2C MDIO->PHY

So if the PHY inside an SFP disappears, the I2C MDIO bus is removed.

That's a crazy idea. The SFP cage driver handles the presence of the I2C-based MDIO bus, and has to because the type of bus created depends on the module, and the lifetime of the bus is determined by the time that the module is plugged in, not whether there's a PHY present.

If removing a PHY on a MDIO bus results in the bus to which it was connected to being removed, that's utterly stupid, because that means all other PHYs will end up being removed when the bus goes away.

Consider a MDIO bus that has some independent PHYs and a DSA switch connected. Do we really want removal of the PHY to destroy the MDIO bus, and thus also take out the DSA switch?

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!
author
#3 lore raw
In reply to Andrew Lunn
Quoted 10 lines
Quoted 10 lines

The commit bc66fa87d4fd ("net: phy: Add link between phy dev and mac dev") has created a device link between the MAC and the PHY if the MAC uses a shared MDIO bus (The MDIO and the MAC are two separate devices).

Sarosh Hasan tried to change the DL_FLAG_STATELESS flag to DL_FLAG_AUTOREMOVE_SUPPLIER to fix the issue [1]. However, the solution does not take into account the hot-swappable PHY devices (such as SFP). so when the PHY device is unplugged, the MAC driver will automatically be removed, which is not the expected behavior.

Quoted 6 lines
Quoted 4 lines

Therefore, to solve this issue of the shared MDIO bus, we create the device link between the MAC device and the MDIO device, rather than between the MAC device and the PHY device. And when the shared MDIO bus is removed, all MAC drivers that depend on it will also be removed.

Shouldn't there be a full chain here? MAC->MDIO->PHY?

The PHY is already a child of the MDIO bus device, so the MDIO->PHY relationship is already represented.

Quoted 9 lines

For the case of the SFP, maybe it should be

MAC->SFP Cage and I2C MDIO->PHY

So if the PHY inside an SFP disappears, the I2C MDIO bus is removed. But there is no link upwards from that, so the SFP cage and the MAC are left alone?

Sorry, I have no experience with SPF module, but from Russell's reply, it seems unlikely that the MDIO bus would be removed when removing the PHY. I also modified Sarosh's patch based on this premise (the MDIO bus will not be removed due to the removal of the PHY).

maintainer
#4 lore raw
In reply to Wei Fang
Quoted 9 lines · On Mon, 26 Jan 2026 18:44:09 +0800 Wei Fang wrote:

The commit bc66fa87d4fd ("net: phy: Add link between phy dev and mac dev") has created a device link between the MAC and the PHY if the MAC uses a shared MDIO bus (The MDIO and the MAC are two separate devices). Sarosh Hasan tried to change the DL_FLAG_STATELESS flag to DL_FLAG_AUTOREMOVE_SUPPLIER to fix the issue [1]. However, the solution does not take into account the hot-swappable PHY devices (such as SFP). so when the PHY device is unplugged, the MAC driver will automatically be removed, which is not the expected behavior.

Quoted 4 lines

Therefore, to solve this issue of the shared MDIO bus, we create the device link between the MAC device and the MDIO device, rather than between the MAC device and the PHY device. And when the shared MDIO bus is removed, all MAC drivers that depend on it will also be removed.

Anyone willing to venture a review tag?

#5 lore raw
Quoted 10 lines · Jakub Kicinski
Quoted 9 lines · On Mon, 26 Jan 2026 18:44:09 +0800 Wei Fang wrote:

The commit bc66fa87d4fd ("net: phy: Add link between phy dev and mac dev") has created a device link between the MAC and the PHY if the MAC uses a shared MDIO bus (The MDIO and the MAC are two separate devices). Sarosh Hasan tried to change the DL_FLAG_STATELESS flag to DL_FLAG_AUTOREMOVE_SUPPLIER to fix the issue [1]. However, the solution does not take into account the hot-swappable PHY devices (such as SFP). so when the PHY device is unplugged, the MAC driver will automatically be removed, which is not the expected behavior.

Quoted 6 lines
Quoted 4 lines

Therefore, to solve this issue of the shared MDIO bus, we create the device link between the MAC device and the MDIO device, rather than between the MAC device and the PHY device. And when the shared MDIO bus is removed, all MAC drivers that depend on it will also be removed.

Anyone willing to venture a review tag?

No, I don't agrew with the patch.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!
author
#6 lore raw
Quoted 13 lines · Jakub Kicinski
Quoted 4 lines · On Mon, 26 Jan 2026 18:44:09 +0800 Wei Fang wrote:
Quoted 3 lines

The commit bc66fa87d4fd ("net: phy: Add link between phy dev and mac dev") has created a device link between the MAC and the PHY if the MAC uses a shared MDIO bus (The MDIO and the MAC are two separate

devices).

Quoted 3 lines
Quoted 3 lines

Sarosh Hasan tried to change the DL_FLAG_STATELESS flag to DL_FLAG_AUTOREMOVE_SUPPLIER to fix the issue [1]. However, the solution does not take into account the hot-swappable PHY devices (such as

SFP).

Quoted 3 lines
Quoted 3 lines

so when the PHY device is unplugged, the MAC driver will automatically be removed, which is not the expected behavior.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

Therefore, to solve this issue of the shared MDIO bus, we create the device link between the MAC device and the MDIO device, rather than between the MAC device and the PHY device. And when the shared MDIO bus is removed, all MAC drivers that depend on it will also be removed.

Anyone willing to venture a review tag?

No, I don't agrew with the patch.

Is it because of SFP? If so, would the following modification be more reasonable, while SFP still follows the previous logic? I'm not sure if phydev->sfp_bus_attached can be used to distinguish between SFP PHYs and non-SFP PHYs.

Anyway, this problem has existed for a long time, and I think we should find a way to solve it in upstream. If there's a better approach, I can help verify it.

drivers/net/phy/phy_device.c +11 -3
index 81984d4ebb7c..c48335bfc7b6 100644
@@ -1771,9 +1771,17 @@ int phy_attach_direct(struct net_device *dev, struct phy_device *phydev,
         * another mac interface, so we should create a device link between
         * phy dev and mac dev.
         */
-       if (dev && phydev->mdio.bus->parent && dev->dev.parent != phydev->mdio.bus->parent)
-               phydev->devlink = device_link_add(dev->dev.parent, &phydev->mdio.dev,
-                                                 DL_FLAG_PM_RUNTIME | DL_FLAG_STATELESS);
+       if (dev && bus->parent && dev->dev.parent != bus->parent) {
+               if (phydev->sfp_bus_attached)
+                       phydev->devlink = device_link_add(dev->dev.parent,
+                                                         bus->parent,
+                                                         DL_FLAG_PM_RUNTIME |
+                                                         DL_FLAG_STATELESS);
+               else
+                       device_link_add(dev->dev.parent, bus->parent,
+                                       DL_FLAG_PM_RUNTIME |
+                                       DL_FLAG_AUTOREMOVE_SUPPLIER);
+       }
#7 lore raw

Hi,

Quoted 20 lines · Wei Fang
Quoted 20 lines · Jakub Kicinski
Quoted 4 lines · On Mon, 26 Jan 2026 18:44:09 +0800 Wei Fang wrote:
Quoted 3 lines

The commit bc66fa87d4fd ("net: phy: Add link between phy dev and mac dev") has created a device link between the MAC and the PHY if the MAC uses a shared MDIO bus (The MDIO and the MAC are two separate

devices).

Quoted 3 lines
Quoted 3 lines

Sarosh Hasan tried to change the DL_FLAG_STATELESS flag to DL_FLAG_AUTOREMOVE_SUPPLIER to fix the issue [1]. However, the solution does not take into account the hot-swappable PHY devices (such as

SFP).

Quoted 9 lines
Quoted 7 lines

so when the PHY device is unplugged, the MAC driver will automatically be removed, which is not the expected behavior.

Therefore, to solve this issue of the shared MDIO bus, we create the device link between the MAC device and the MDIO device, rather than between the MAC device and the PHY device. And when the shared MDIO bus is removed, all MAC drivers that depend on it will also be removed.

Anyone willing to venture a review tag?

Quoted 7 lines
Quoted 2 lines

No, I don't agrew with the patch.

Is it because of SFP? If so, would the following modification be more reasonable, while SFP still follows the previous logic? I'm not sure if phydev->sfp_bus_attached can be used to distinguish between SFP PHYs and non-SFP PHYs.

If you want to know if the PHY is in an SFP module, you should use phy_on_sfp() instead :

https://elixir.bootlin.com/linux/v6.18.6/source/include/linux/phy.h#L1766

It's set for this kind of setups :

SFP module

+-----+      +-----------+
| MAC | ---- | PHY       |
+-----+      +--\--------+
                 \
                  \_ phy_on_sfp(phydev) == true

phydev->sfp_bus_attached is for the media converter user-case :

SFP Module

+-----+      +-----+      +------------+
| MAC | ---- | PHY | ---- |   ???      |
+-----+      +--\--+      +------------+
                 \
                  \_ phydev->sfp_bus_attached == true
Quoted 4 lines

Anyway, this problem has existed for a long time, and I think we should find a way to solve it in upstream. If there's a better approach, I can help verify it.

Sorry I wasn't available for most part of yesterday for that discussion.

I'll give a try to this patch today, with the above change and let you know how this goes.

Maxime

author
#8 lore raw
Quoted 34 lines
Quoted 10 lines
Quoted 4 lines
Quoted 1 line

Anyone willing to venture a review tag?

No, I don't agrew with the patch.

Is it because of SFP? If so, would the following modification be more reasonable, while SFP still follows the previous logic? I'm not sure if
phydev->sfp_bus_attached can be used to distinguish between SFP PHYs and non-SFP PHYs.

If you want to know if the PHY is in an SFP module, you should use phy_on_sfp() instead :

https://elixir.boo/
tlin.com%2Flinux%2Fv6.18.6%2Fsource%2Finclude%2Flinux%2Fphy.h%23L1766 &data=05%7C02%7Cwei.fang%40nxp.com%7C94dbe2fb454c4a8fd96708de5fd9 b94a%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6390535858824 03531%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLj AuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C% 7C%7C&sdata=FqnLEaC8Jt0zIgVQpIkbp0fg7WCrBuJjJFkCExRQKoc%3D&reserved =0

It's set for this kind of setups :

SFP module

+-----+      +-----------+
| MAC | ---- | PHY       |
+-----+      +--\--------+
                 \
                  \_ phy_on_sfp(phydev) == true
Quoted 8 lines

phydev->sfp_bus_attached is for the media converter user-case :

SFP Module

+-----+      +-----+      +------------+
| MAC | ---- | PHY | ---- |   ???      |
+-----+      +--\--+      +------------+
                 \
                  \_ phydev->sfp_bus_attached == true

Thank you very much for your explanation and the diagram. I also noticed phydev->is_on_sfp_module, which is also used in phy_attach_direct(), but I didn't know the difference between them.

Quoted 10 lines
Quoted 4 lines

Anyway, this problem has existed for a long time, and I think we should find a way to solve it in upstream. If there's a better approach, I can help verify it.

Sorry I wasn't available for most part of yesterday for that discussion.

I'll give a try to this patch today, with the above change and let you know how this goes.

Thanks again. This problem has been bothering us for a long time, it would be great if this problem could be solved.

#9 lore raw
In reply to Wei Fang

Hi again,

On 30/01/2026 04:41, Wei Fang wrote:
[...]

Quoted 6 lines
drivers/net/phy/phy_device.c +0 -0
index 81984d4ebb7c..c48335bfc7b6 100644

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, struct phy_device *phydev, · line 1775

Quoted 8 lines
drivers/net/phy/phy_device.c +2 -3
@@ -1771,9 +1771,17 @@ int phy_attach_direct(struct net_device *dev, struct phy_device *phydev,
         * another mac interface, so we should create a device link between
         * phy dev and mac dev.
         */
-       if (dev && phydev->mdio.bus->parent && dev->dev.parent != phydev->mdio.bus->parent)
-               phydev->devlink = device_link_add(dev->dev.parent, &phydev->mdio.dev,
-                                                 DL_FLAG_PM_RUNTIME | DL_FLAG_STATELESS);
+       if (dev && bus->parent && dev->dev.parent != bus->parent) {
+               if (phydev->sfp_bus_attached)

^ assuming we replace with phy_on_sfp()

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, struct phy_device *phydev, · line 1784

Quoted 9 lines
drivers/net/phy/phy_device.c +9 -0
+                       phydev->devlink = device_link_add(dev->dev.parent,
+                                                         bus->parent,
+                                                         DL_FLAG_PM_RUNTIME |
+                                                         DL_FLAG_STATELESS);
+               else
+                       device_link_add(dev->dev.parent, bus->parent,
+                                       DL_FLAG_PM_RUNTIME |
+                                       DL_FLAG_AUTOREMOVE_SUPPLIER);
+       }

I think we should let the phy_on_sfp() case alone for now. The lifetime of the SFP phydev and the MAC are completely independent here, the PHY can be probed/removed without the MAC being there, and vice versa.

The MDIO bus that controls that SFP PHY's lifetime is the same as the PHY itself, the mdio bus in that case exists solely to drive that single PHY, and in can't be shared between multiple PHYs even if we have multiple SFP ports in the entire system.

The logical thing to do IMO is not to tie the MAC and the MDIO Bus itself, but rather the MAC and the PHY as was done before. As you state, we couldn't set DL_FLAG_AUTOREMOVE_SUPPLIER because of SFP PHYs, but if we add the check on "phy_on_sfp()", it should now be fine.

Maxime

author
#10 lore raw
Quoted 41 lines · Wei Fang

[...]

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev,

Quoted 6 lines
drivers/net/phy/phy_device.c +0 -0
index 81984d4ebb7c..c48335bfc7b6 100644
@@ -1771,9 +1771,17 @@ int phy_attach_direct(struct net_device *dev,

struct phy_device *phydev,

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, · line 1771

Quoted 1 line
drivers/net/phy/phy_device.c +0 -0
         * another mac interface, so we should create a device link

between

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, struct phy_device *phydev,

Quoted 3 lines
drivers/net/phy/phy_device.c +0 -1
         * phy dev and mac dev.
         */
-       if (dev && phydev->mdio.bus->parent && dev->dev.parent !=

phydev->mdio.bus->parent)

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, · line 1775

Quoted 1 line
drivers/net/phy/phy_device.c +0 -1
-               phydev->devlink = device_link_add(dev->dev.parent,

&phydev->mdio.dev,

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, · line 1776

Quoted 1 line
drivers/net/phy/phy_device.c +0 -1
-

DL_FLAG_PM_RUNTIME | DL_FLAG_STATELESS);

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, struct phy_device *phydev, · line 1775

Quoted 2 lines
drivers/net/phy/phy_device.c +2 -0
+       if (dev && bus->parent && dev->dev.parent != bus->parent) {
+               if (phydev->sfp_bus_attached)

^ assuming we replace with phy_on_sfp()

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, · line 1776

Quoted 1 line
drivers/net/phy/phy_device.c +1 -0
+                       phydev->devlink =

device_link_add(dev->dev.parent,

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, · line 1777

Quoted 1 line
drivers/net/phy/phy_device.c +1 -0
+

bus->parent,

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, · line 1778

Quoted 1 line
drivers/net/phy/phy_device.c +1 -0
+

DL_FLAG_PM_RUNTIME |

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, · line 1779

Quoted 1 line
drivers/net/phy/phy_device.c +1 -0
+

DL_FLAG_STATELESS);

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, struct phy_device *phydev,

Quoted 4 lines
drivers/net/phy/phy_device.c +4 -0
+               else
+                       device_link_add(dev->dev.parent, bus->parent,
+                                       DL_FLAG_PM_RUNTIME |
+

DL_FLAG_AUTOREMOVE_SUPPLIER);

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, struct phy_device *phydev, · line 1784

Quoted 1 line
drivers/net/phy/phy_device.c +1 -0
+       }

I think we should let the phy_on_sfp() case alone for now. The lifetime of the SFP phydev and the MAC are completely independent here, the PHY can be probed/removed without the MAC being there, and vice versa.

Quoted 7 lines

The MDIO bus that controls that SFP PHY's lifetime is the same as the PHY itself, the mdio bus in that case exists solely to drive that single PHY, and in can't be shared between multiple PHYs even if we have multiple SFP ports in the entire system.

The logical thing to do IMO is not to tie the MAC and the MDIO Bus itself, but rather the MAC and the PHY as was done before. As you state,

Okay, I agree. I will only change the devlink flag to DL_FLAG_AUTOREMOVE_SUPPLIER. And thanks for you prompt feedback. :)

Quoted 3 lines

we couldn't set DL_FLAG_AUTOREMOVE_SUPPLIER because of SFP PHYs, but if we add the check on "phy_on_sfp()", it should now be fine.

#11 lore raw
In reply to Wei Fang

Hi

[...]

Quoted 6 lines
drivers/net/phy/phy_device.c +0 -0
index 81984d4ebb7c..d5ac7506fe39 100644

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, struct phy_device *phydev, · line 1774

Quoted 9 lines
drivers/net/phy/phy_device.c +3 -2
@@ -1768,12 +1768,21 @@ int phy_attach_direct(struct net_device *dev, struct phy_device *phydev,
 
 	/**
 	 * If the external phy used by current mac interface is managed by
-	 * another mac interface, so we should create a device link between
-	 * phy dev and mac dev.
+	 * another MDIO controller, which means that the MAC and MDIO are
+	 * separated devices, then we should create a device link between
+	 * the MAC device and the MDIO device.
 	 */

I was confused by the use of the "MDIO device" terminology here, which refers to a device sitting on an mdio bus, such as a PHY. I think though that what you actually refer to is the MDIO controller itself, right ?

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, struct phy_device *phydev, · line 1779

Quoted 8 lines
drivers/net/phy/phy_device.c +5 -3
-	if (dev && phydev->mdio.bus->parent && dev->dev.parent != phydev->mdio.bus->parent)
-		phydev->devlink = device_link_add(dev->dev.parent, &phydev->mdio.dev,
-						  DL_FLAG_PM_RUNTIME | DL_FLAG_STATELESS);
+	if (dev && phydev->mdio.bus->parent &&
+	    dev->dev.parent != phydev->mdio.bus->parent) {
+		if (!device_link_add(dev->dev.parent, phydev->mdio.bus->parent,
+				     DL_FLAG_PM_RUNTIME |
+				     DL_FLAG_AUTOREMOVE_SUPPLIER)) {

Don't we have the same problem with SFP ? The struct mii_bus for SFP PHYs also completely disappears when you remove the module.

That being said, I ran some tests with SFP and this patch, and the netdev actually didn't dissapear under my feet.

I'm a total noob with fw_devlink, but I can see there seems to already be a link between MAC devices and the associated SFP bus :

on cyclone V :

# ls /sys/class/devlink
 [...]
platform:sfp--platform:ff702000.ethernet

on macchiatobin :

# ls /sys/class/devlink
 [...]
platform:sfp-eth3--platform:f4000000.ethernet

So I guess this is what's preventing the netdev from going away.

Now, these seems to be automagically populated based on DT, but I don't know if it also works with non-DT platforms. If this is DT-specific, then I guess we still can't use DL_FLAG_AUTOREMOVE_SUPPLIER.

Maxime

author
#12 lore raw
Quoted 15 lines

Review int phy_attach_direct(struct net_device *dev,

Quoted 1 line
(quoted diff) +0 -0
@@ -1768,12 +1768,21 @@ int phy_attach_direct(struct net_device *dev,

struct phy_device *phydev,

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, struct phy_device *phydev, · line 1774

Quoted 9 lines
drivers/net/phy/phy_device.c +3 -2
 	/**
 	 * If the external phy used by current mac interface is managed by
-	 * another mac interface, so we should create a device link between
-	 * phy dev and mac dev.
+	 * another MDIO controller, which means that the MAC and MDIO are
+	 * separated devices, then we should create a device link between
+	 * the MAC device and the MDIO device.
 	 */

I was confused by the use of the "MDIO device" terminology here, which refers to a device sitting on an mdio bus, such as a PHY. I think though that what you actually refer to is the MDIO controller itself, right ?

Yes, you are right, I was referring to the MDIO controller.

Quoted 7 lines

Review int phy_attach_direct(struct net_device *dev, · line 1773

Quoted 2 lines
(quoted diff) +0 -1
-	if (dev && phydev->mdio.bus->parent && dev->dev.parent !=

phydev->mdio.bus->parent)

Review int phy_attach_direct(struct net_device *dev, · line 1774

Quoted 1 line
(quoted diff) +0 -1
-		phydev->devlink = device_link_add(dev->dev.parent,

&phydev->mdio.dev,

Review int phy_attach_direct(struct net_device *dev, · line 1775

Quoted 1 line
(quoted diff) +0 -1
-						  DL_FLAG_PM_RUNTIME |

DL_FLAG_STATELESS);

Quoted 8 lines

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, struct phy_device *phydev, · line 1779

Quoted 5 lines
drivers/net/phy/phy_device.c +5 -0
+	if (dev && phydev->mdio.bus->parent &&
+	    dev->dev.parent != phydev->mdio.bus->parent) {
+		if (!device_link_add(dev->dev.parent, phydev->mdio.bus->parent,
+				     DL_FLAG_PM_RUNTIME |
+				     DL_FLAG_AUTOREMOVE_SUPPLIER)) {

Don't we have the same problem with SFP ? The struct mii_bus for SFP PHYs also completely disappears when you remove the module.

Sorry, I'm not familiar with SFP and I do not have a board with SPF module. So I'm not sure whether the MDIO bus will be removed if the SFP module is unplugged. But if the multiple SPF modules share the same MDIO controller, unplugging one SFP module will cause the MDIO bus to be removed. Won't this cause the other SFP modules to stop working?

Quoted 19 lines

That being said, I ran some tests with SFP and this patch, and the netdev actually didn't dissapear under my feet.

I'm a total noob with fw_devlink, but I can see there seems to already be a link between MAC devices and the associated SFP bus :

on cyclone V :

# ls /sys/class/devlink
 [...]
platform:sfp--platform:ff702000.ethernet

on macchiatobin :

# ls /sys/class/devlink
 [...]
platform:sfp-eth3--platform:f4000000.ethernet
Quoted 6 lines

So I guess this is what's preventing the netdev from going away.

Now, these seems to be automagically populated based on DT, but I don't know if it also works with non-DT platforms. If this is DT-specific, then I guess we still can't use DL_FLAG_AUTOREMOVE_SUPPLIER.

Previously, when Sorash attempted to add DL_FLAG_AUTOREMOVE_SUPPLIER to the devlink between the MAC controller and the PHY, you reported that unplugging the PHY would cause the MAC to be removed as well, resulting in system hangs.

https://lore.kernel.org/imx/[email protected]/

If your guess is correct, removing the SPF module will cause the MDIO bus to be removed, then I think the MAC driver should also be removed. Therefore, based on this patch, you should be able to reproduce the same problem caused by the Sorash's patch. However, your current results seem different from before. Perhaps the MDIO bus was not removed along with the SFP module.

#13 lore raw
Quoted 34 lines · Wei Fang
Quoted 15 lines

Review int phy_attach_direct(struct net_device *dev,

Quoted 1 line
(quoted diff) +0 -0
@@ -1768,12 +1768,21 @@ int phy_attach_direct(struct net_device *dev,

struct phy_device *phydev,

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, struct phy_device *phydev, · line 1774

Quoted 9 lines
drivers/net/phy/phy_device.c +3 -2
 	/**
 	 * If the external phy used by current mac interface is managed by
-	 * another mac interface, so we should create a device link between
-	 * phy dev and mac dev.
+	 * another MDIO controller, which means that the MAC and MDIO are
+	 * separated devices, then we should create a device link between
+	 * the MAC device and the MDIO device.
 	 */

I was confused by the use of the "MDIO device" terminology here, which refers to a device sitting on an mdio bus, such as a PHY. I think though that what you actually refer to is the MDIO controller itself, right ?

Yes, you are right, I was referring to the MDIO controller.

Quoted 7 lines

Review int phy_attach_direct(struct net_device *dev, · line 1773

Quoted 2 lines
(quoted diff) +0 -1
-	if (dev && phydev->mdio.bus->parent && dev->dev.parent !=

phydev->mdio.bus->parent)

Review int phy_attach_direct(struct net_device *dev, · line 1774

Quoted 1 line
(quoted diff) +0 -1
-		phydev->devlink = device_link_add(dev->dev.parent,

&phydev->mdio.dev,

Review int phy_attach_direct(struct net_device *dev, · line 1775

Quoted 1 line
(quoted diff) +0 -1
-						  DL_FLAG_PM_RUNTIME |

DL_FLAG_STATELESS);

Quoted 8 lines

drivers/net/phy/phy_device.c int phy_attach_direct(struct net_device *dev, struct phy_device *phydev, · line 1779

Quoted 5 lines
drivers/net/phy/phy_device.c +5 -0
+	if (dev && phydev->mdio.bus->parent &&
+	    dev->dev.parent != phydev->mdio.bus->parent) {
+		if (!device_link_add(dev->dev.parent, phydev->mdio.bus->parent,
+				     DL_FLAG_PM_RUNTIME |
+				     DL_FLAG_AUTOREMOVE_SUPPLIER)) {

Don't we have the same problem with SFP ? The struct mii_bus for SFP PHYs also completely disappears when you remove the module.

Quoted 6 lines

Sorry, I'm not familiar with SFP and I do not have a board with SPF module. So I'm not sure whether the MDIO bus will be removed if the SFP module is unplugged. But if the multiple SPF modules share the same MDIO controller, unplugging one SFP module will cause the MDIO bus to be removed. Won't this cause the other SFP modules to stop working?

For SFP, it's not a physical mdiobus that connects to the SFP module, but rather an i2c bus. When the module is inserted, we create an mdiobus that will perform the mdio transfers over i2c :

https://elixir.bootlin.com/linux/v6.19-rc5/source/drivers/net/mdio/mdio-i2c.c#L461

The mdio bus is created when the SFP module is inserted, and we found a PHY device in it :

https://elixir.bootlin.com/linux/v6.19-rc5/source/drivers/net/phy/sfp.c#L791

It is dynamically created and destroyed, its lifetime isn't correlated to the netdevice.

Quoted 32 lines
Quoted 20 lines

That being said, I ran some tests with SFP and this patch, and the netdev actually didn't dissapear under my feet.

I'm a total noob with fw_devlink, but I can see there seems to already be a link between MAC devices and the associated SFP bus :

on cyclone V :

# ls /sys/class/devlink
 [...]
platform:sfp--platform:ff702000.ethernet

on macchiatobin :

# ls /sys/class/devlink
 [...]
platform:sfp-eth3--platform:f4000000.ethernet
Quoted 6 lines

So I guess this is what's preventing the netdev from going away.

Now, these seems to be automagically populated based on DT, but I don't know if it also works with non-DT platforms. If this is DT-specific, then I guess we still can't use DL_FLAG_AUTOREMOVE_SUPPLIER.

Previously, when Sorash attempted to add DL_FLAG_AUTOREMOVE_SUPPLIER to the devlink between the MAC controller and the PHY, you reported that unplugging the PHY would cause the MAC to be removed as well, resulting in system hangs.

Quoted 9 lines

https://lore.kernel.org/imx/[email protected]/

If your guess is correct, removing the SPF module will cause the MDIO bus to be removed, then I think the MAC driver should also be removed. Therefore, based on this patch, you should be able to reproduce the same problem caused by the Sorash's patch. However, your current results seem different from before. Perhaps the MDIO bus was not removed along with the SFP module.

It is removed :

https://elixir.bootlin.com/linux/v6.19-rc5/source/drivers/net/phy/sfp.c#L2648

This line above is called when we unplug the SFP module.

What I am seeing is that there are other devlink links that apply to the net_device, unrelated to the MDIO bus, at least on the devices I am testing this on, and they prevent the net_device from going away.

One of these devlinks is a link between the sfp cage (compatible "sff,sfp") and the netdev. This devlink seems to be automatically populated when OF is parsed.

To solve this, I think we need something more granular that this approach, and make a distinction between when it's OK for a phy_device to dissapear when attached to a MAC (SFP phys), and when it's not (PHYs hardwired to the MAC).

Adding this devlink in phy_attach_direct() is too generic IMO. It's likely that this link should be populated by phylink then.

Maxime

author
#14 lore raw
Quoted 11 lines

It is removed :

https://elixir.boo/
tlin.com%2Flinux%2Fv6.19-rc5%2Fsource%2Fdrivers%2Fnet%2Fphy%2Fsfp.c%2 3L2648&data=05%7C02%7Cwei.fang%40nxp.com%7C2f06a252bfa24532562e0 8de5f1fbe42%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6390527 87105780228%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlY iOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C 0%7C%7C%7C&sdata=FtbZ%2FZWqje7ckLwHvVENVlkus7nsTB3hZ3N05jYuflc%3 D&reserved=0

Quoted 2 lines

This line above is called when we unplug the SFP module.

Many thanks for your detailed explanation, I really know very little about SFP. :(

Quoted 4 lines

What I am seeing is that there are other devlink links that apply to the net_device, unrelated to the MDIO bus, at least on the devices I am testing this on, and they prevent the net_device from going away.

Quoted 8 lines

One of these devlinks is a link between the sfp cage (compatible "sff,sfp") and the netdev. This devlink seems to be automatically populated when OF is parsed.

To solve this, I think we need something more granular that this approach, and make a distinction between when it's OK for a phy_device to dissapear when attached to a MAC (SFP phys), and when it's not (PHYs hardwired to the MAC).

Yes, perhaps we need to add some flags to determine whether to add a devlink between MAC controller and MDIO controller.

Quoted 4 lines

Adding this devlink in phy_attach_direct() is too generic IMO. It's likely that this link should be populated by phylink then.

Some drivers may not use phylink, like the fec driver.