Skip to contentSkip to conversation

net: ethtool: allow MAC drivers to override ethtool get_ts_info

PATCH In review Russell King 5 participants 19 messages Jan 10, 2021 – Jan 21, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net: ethtool: allow MAC drivers to override ethtool get_ts_info

Check whether the MAC driver has implemented the get_ts_info() method first, and call it if present. If this method returns -EOPNOTSUPP, defer to the phylib or default implementation.

This allows network drivers such as mvpp2 to use their more accurate timestamping implementation than using a less accurate implementation in the PHY. Network drivers can opt to defer to phylib by returning -EOPNOTSUPP.

This change will be needed if the Marvell PHY drivers add support for PTP.

Note: this may cause a change for any drivers that use phylib and provide get_ts_info(). It is not obvious if any such cases exist.

---
 net/ethtool/common.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
net/ethtool/common.c +6 -2
index 24036e3055a1..9ec93e24f239 100644
@@ -385,14 +385,18 @@ int __ethtool_get_ts_info(struct net_device *dev, struct ethtool_ts_info *info)
 {
 	const struct ethtool_ops *ops = dev->ethtool_ops;
 	struct phy_device *phydev = dev->phydev;
+	int ret;
 
 	memset(info, 0, sizeof(*info));
 	info->cmd = ETHTOOL_GET_TS_INFO;
 
+	if (ops->get_ts_info) {
+		ret = ops->get_ts_info(dev, info);
+		if (ret != -EOPNOTSUPP)
+			return ret;
+	}
 	if (phy_has_tsinfo(phydev))
 		return phy_ts_info(phydev, info);
-	if (ops->get_ts_info)
-		return ops->get_ts_info(dev, info);
 
 	info->so_timestamping = SOF_TIMESTAMPING_RX_SOFTWARE |
 				SOF_TIMESTAMPING_SOFTWARE;
Signature · 1 line
2.20.1
maintainer
#1 lore raw
Quoted 9 lines · Russell King

Check whether the MAC driver has implemented the get_ts_info() method first, and call it if present. If this method returns -EOPNOTSUPP, defer to the phylib or default implementation.

This allows network drivers such as mvpp2 to use their more accurate timestamping implementation than using a less accurate implementation in the PHY. Network drivers can opt to defer to phylib by returning -EOPNOTSUPP.

Quoted 5 lines

This change will be needed if the Marvell PHY drivers add support for PTP.

Note: this may cause a change for any drivers that use phylib and provide get_ts_info(). It is not obvious if any such cases exist.

Hi Russell

We can detect that condition through? Call both, then do a WARN() if we are changing the order? Maybe we should do that for a couple of cycles?

For netlink ethtool, we can also provide an additional attribute. A MAC, or PHY indicator we can do in the core. A string for the name of the driver would need a bigger change.

Andrew

maintainer
#2 lore raw
Quoted 18 lines · On Sun, 10 Jan 2021 17:35:25 +0100 Andrew Lunn wrote:
Quoted 9 lines · Russell King

Check whether the MAC driver has implemented the get_ts_info() method first, and call it if present. If this method returns -EOPNOTSUPP, defer to the phylib or default implementation.

This allows network drivers such as mvpp2 to use their more accurate timestamping implementation than using a less accurate implementation in the PHY. Network drivers can opt to defer to phylib by returning -EOPNOTSUPP.

Quoted 5 lines

This change will be needed if the Marvell PHY drivers add support for PTP.

Note: this may cause a change for any drivers that use phylib and provide get_ts_info(). It is not obvious if any such cases exist.

Hi Russell

Quoted 7 lines

We can detect that condition through? Call both, then do a WARN() if we are changing the order? Maybe we should do that for a couple of cycles?

For netlink ethtool, we can also provide an additional attribute. A MAC, or PHY indicator we can do in the core. A string for the name of the driver would need a bigger change.

I'm not seeing a response to this suggestion, did vger eat it?

#3 lore raw
In reply to Andrew Lunn
Quoted 13 lines · Andrew Lunn
Quoted 12 lines · Russell King

Check whether the MAC driver has implemented the get_ts_info() method first, and call it if present. If this method returns -EOPNOTSUPP, defer to the phylib or default implementation.

This allows network drivers such as mvpp2 to use their more accurate timestamping implementation than using a less accurate implementation in the PHY. Network drivers can opt to defer to phylib by returning -EOPNOTSUPP.

This change will be needed if the Marvell PHY drivers add support for PTP.

Quoted 8 lines
Quoted 2 lines

Note: this may cause a change for any drivers that use phylib and provide get_ts_info(). It is not obvious if any such cases exist.

Hi Russell

We can detect that condition through? Call both, then do a WARN() if we are changing the order? Maybe we should do that for a couple of cycles?

I guess we could do something, but IMHO this really does not justify using heavy hammers like WARN(). It's pointless producing a backtrace. If we want a large noisy multi-line message that stands out, we should just do that.

I think we can detect with something like:

	if (ops->get_ts_info && phy_has_tsinfo(phydev)) {
		netdev_warn(dev, "Both the PHY and the MAC support PTP. Which you end up with may change.\n");
	}

That said, this is _actually_ a fix.

As the code stands today:

__ethtool_get_ts_info() checks phy_has_tsinfo() and uses phy_ts_info() in preference to ops->get_ts_info(), giving the PHY code first dibs on intercepting this call.

Meanwhile, the ioctl() code gives the network driver first dibs on intercepting the SIOCSHWTSTAMP and SIOCGHWTSTAMP ioctls.

This means that if you have both a PHY supporting timestamping, and a MAC driver, you end up with the ethtool get_ts_info() call giving a response from the PHY implementation but SIOCSHWTSTAMP and SIOCGHWTSTAMP being intercepted by the MAC implementation.

This is exactly what will happen today to mvpp2 if we merge my patches adding PTP support to the Marvell 88e151x PHYs without this patch.

So, my patch merely brings consistency to this.

Quoted 3 lines

For netlink ethtool, we can also provide an additional attribute. A MAC, or PHY indicator we can do in the core. A string for the name of the driver would need a bigger change.

Unfortunately, PTP is not solely controlled through ethtool - it's also controlled via ioctl() where it's not so easy to direct the calls to either the MAC or PHY.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#4 lore raw
Quoted 55 lines · Russell King - ARM Linux admin
Quoted 13 lines · Andrew Lunn
Quoted 12 lines · Russell King

Check whether the MAC driver has implemented the get_ts_info() method first, and call it if present. If this method returns -EOPNOTSUPP, defer to the phylib or default implementation.

This allows network drivers such as mvpp2 to use their more accurate timestamping implementation than using a less accurate implementation in the PHY. Network drivers can opt to defer to phylib by returning -EOPNOTSUPP.

This change will be needed if the Marvell PHY drivers add support for PTP.

Quoted 8 lines
Quoted 2 lines

Note: this may cause a change for any drivers that use phylib and provide get_ts_info(). It is not obvious if any such cases exist.

Hi Russell

We can detect that condition through? Call both, then do a WARN() if we are changing the order? Maybe we should do that for a couple of cycles?

I guess we could do something, but IMHO this really does not justify using heavy hammers like WARN(). It's pointless producing a backtrace. If we want a large noisy multi-line message that stands out, we should just do that.

I think we can detect with something like:

	if (ops->get_ts_info && phy_has_tsinfo(phydev)) {
		netdev_warn(dev, "Both the PHY and the MAC support PTP. Which you end up with may change.\n");
	}

That said, this is _actually_ a fix.

As the code stands today:

__ethtool_get_ts_info() checks phy_has_tsinfo() and uses phy_ts_info() in preference to ops->get_ts_info(), giving the PHY code first dibs on intercepting this call.

Meanwhile, the ioctl() code gives the network driver first dibs on intercepting the SIOCSHWTSTAMP and SIOCGHWTSTAMP ioctls.

This means that if you have both a PHY supporting timestamping, and a MAC driver, you end up with the ethtool get_ts_info() call giving a response from the PHY implementation but SIOCSHWTSTAMP and SIOCGHWTSTAMP being intercepted by the MAC implementation.

This is exactly what will happen today to mvpp2 if we merge my patches adding PTP support to the Marvell 88e151x PHYs without this patch.

So, my patch merely brings consistency to this.

Quoted 7 lines
Quoted 3 lines

For netlink ethtool, we can also provide an additional attribute. A MAC, or PHY indicator we can do in the core. A string for the name of the driver would need a bigger change.

Unfortunately, PTP is not solely controlled through ethtool - it's also controlled via ioctl() where it's not so easy to direct the calls to either the MAC or PHY.

BTW, none of this is new informationm we're just re-covering the same points that were already discussed back in July 2020:

https://lore.kernel.org/netdev/[email protected]/

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#5 lore raw
In reply to Russell King
Quoted 4 lines · Russell King

This allows network drivers such as mvpp2 to use their more accurate timestamping implementation than using a less accurate implementation in the PHY. Network drivers can opt to defer to phylib by returning -EOPNOTSUPP.

My expectation is that PHY time stamping is more accurate than MAC time stamping.

Quoted 2 lines

This change will be needed if the Marvell PHY drivers add support for PTP.

Huh? The mvpp2 appears to be a MAC. If this device has integrated PHYs then I don't see the issue. If your board has the nvpp2 device with the dp83640 PHYTER, then don't you want to actually use the PHYTER?

From my observation of the product offerings, I have yet to see a new PHY (besides the dp83640 PHYTER) that implement time stamping. The PHYTER is 100 megabit only, and my understanding that it is too difficult or even impossible to provide time stamps from within a gigabit+ PHY. So you needn't fear new time stamping PHYs to spoil your setup!

Quoted 2 lines

Note: this may cause a change for any drivers that use phylib and provide get_ts_info(). It is not obvious if any such cases exist.

Up until now, the code always favored PHY devices and devices external to the MAC that snoop on the MII bus. The assumption is that anyone who builds a board with such specialty devices really wants to use them.

Thanks,
Richard

#6 lore raw
Quoted 8 lines · Richard Cochran
Quoted 4 lines · Russell King

This allows network drivers such as mvpp2 to use their more accurate timestamping implementation than using a less accurate implementation in the PHY. Network drivers can opt to defer to phylib by returning -EOPNOTSUPP.

My expectation is that PHY time stamping is more accurate than MAC time stamping.

PHY time stamping may be a "more accurate" location to get timestamps, in terms of the hardware, but when you consider the entire setup, that is not necessarily the case.

Quoted 7 lines
Quoted 2 lines

This change will be needed if the Marvell PHY drivers add support for PTP.

Huh? The mvpp2 appears to be a MAC. If this device has integrated PHYs then I don't see the issue. If your board has the nvpp2 device with the dp83640 PHYTER, then don't you want to actually use the PHYTER?

You seem to be adding more information way beyond what I'm saying.

No, there aren't integrated PHYs. There's an external PHY - a Marvell 88e151x which has what I would call rudimentary stamping abilities, whereas the mvpp2 has advanced stamping abilities.

You implemented the Marvell timestamping in DSA, so you know what the Marvell offering there looks like and what it is capable of. That same hardware appears in some Marvell PHYs.

The mvpp2 hardware (which has support already merged after you acked the TAI part, and failed to provide comments on the mvpp2 part - so davem gave up waiting) is capable of:
- stamping every received packet irrespective of its type
- stamping any transmitted packet or only those we wish to stamp
- inserting a timestamp into the packet (aka one-step, although that isn't implemented that yet)
- correcting the hardware time counter tick rate

There is considerable noise in reading the hardware timestamp counter from the PHYs - caused by latency over the MDIO bus, which makes the achievable accuracy lower. That noise is very much reduced when reading the hardware timestamp counter from mvpp2 - where we can implement mvpp22_tai_gettimex64(). Therefore, the achieved accuracy from mvpp2 is higher than from a PHY.

We had already discussed this patch last year, and you agreed with it then. What has changed?

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#7 lore raw
Quoted 48 lines · Russell King - ARM Linux admin
Quoted 8 lines · Richard Cochran
Quoted 4 lines · Russell King

This allows network drivers such as mvpp2 to use their more accurate timestamping implementation than using a less accurate implementation in the PHY. Network drivers can opt to defer to phylib by returning -EOPNOTSUPP.

My expectation is that PHY time stamping is more accurate than MAC time stamping.

PHY time stamping may be a "more accurate" location to get timestamps, in terms of the hardware, but when you consider the entire setup, that is not necessarily the case.

Quoted 7 lines
Quoted 2 lines

This change will be needed if the Marvell PHY drivers add support for PTP.

Huh? The mvpp2 appears to be a MAC. If this device has integrated PHYs then I don't see the issue. If your board has the nvpp2 device with the dp83640 PHYTER, then don't you want to actually use the PHYTER?

You seem to be adding more information way beyond what I'm saying.

No, there aren't integrated PHYs. There's an external PHY - a Marvell 88e151x which has what I would call rudimentary stamping abilities, whereas the mvpp2 has advanced stamping abilities.

You implemented the Marvell timestamping in DSA, so you know what the Marvell offering there looks like and what it is capable of. That same hardware appears in some Marvell PHYs.

The mvpp2 hardware (which has support already merged after you acked the TAI part, and failed to provide comments on the mvpp2 part - so davem gave up waiting) is capable of:
- stamping every received packet irrespective of its type
- stamping any transmitted packet or only those we wish to stamp
- inserting a timestamp into the packet (aka one-step, although that isn't implemented that yet)
- correcting the hardware time counter tick rate

There is considerable noise in reading the hardware timestamp counter from the PHYs - caused by latency over the MDIO bus, which makes the achievable accuracy lower. That noise is very much reduced when reading the hardware timestamp counter from mvpp2 - where we can implement mvpp22_tai_gettimex64(). Therefore, the achieved accuracy from mvpp2 is higher than from a PHY.

Quoted 2 lines

We had already discussed this patch last year, and you agreed with it then. What has changed?

See the discussion in this sub-thread:

https://lore.kernel.org/netdev/[email protected]/

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#8 lore raw
Quoted 6 lines · Russell King - ARM Linux admin
Quoted 2 lines

We had already discussed this patch last year, and you agreed with it then. What has changed?

See the discussion in this sub-thread:

https://lore.kernel.org/netdev/[email protected]/

Thanks for the reminder. We ended up with having to review the MAC drivers that support phydev.

https://lore.kernel.org/netdev/[email protected]/

There is at least the FEC that supports phydev. I have a board that combines the FEC with the dp83640 PHYTER, and your patch would break this setup. (In the case of this HW combination, the PHYTER is superior in every way.)

Another combination that I have seen twice is the TI am335x with its cpsw MAC and the PHYTER. Unfortunately I don't have one of these boards, but people made them because the cpsw MAC supports time stamping in a way that is inadequate.

I *think* the cpsw/phyter combination would work with your patch, but only if the users disable CONFIG_TI_CPTS at compile time.

Thanks,
Richard

#9 lore raw
Quoted 18 lines · Richard Cochran
Quoted 6 lines · Russell King - ARM Linux admin
Quoted 2 lines

We had already discussed this patch last year, and you agreed with it then. What has changed?

See the discussion in this sub-thread:

https://lore.kernel.org/netdev/[email protected]/

Thanks for the reminder. We ended up with having to review the MAC drivers that support phydev.

   https://lore.kernel.org/netdev/[email protected]/

There is at least the FEC that supports phydev. I have a board that combines the FEC with the dp83640 PHYTER, and your patch would break this setup. (In the case of this HW combination, the PHYTER is superior in every way.)

Quoted 7 lines

Another combination that I have seen twice is the TI am335x with its cpsw MAC and the PHYTER. Unfortunately I don't have one of these boards, but people made them because the cpsw MAC supports time stamping in a way that is inadequate.

I *think* the cpsw/phyter combination would work with your patch, but only if the users disable CONFIG_TI_CPTS at compile time.

I think then the only solution is to move the decision how to handle get_ts_info into each MAC driver and get rid of:

	if (phy_has_tsinfo(phydev))
	        return phy_ts_info(phydev, info);

in __ethtool_get_ts_info().

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#10 lore raw
Quoted 19 lines · Russell King - ARM Linux admin
Quoted 10 lines · Richard Cochran

Thanks for the reminder. We ended up with having to review the MAC drivers that support phydev.

   https://lore.kernel.org/netdev/[email protected]/

There is at least the FEC that supports phydev. I have a board that combines the FEC with the dp83640 PHYTER, and your patch would break this setup. (In the case of this HW combination, the PHYTER is superior in every way.)

Quoted 7 lines

Another combination that I have seen twice is the TI am335x with its cpsw MAC and the PHYTER. Unfortunately I don't have one of these boards, but people made them because the cpsw MAC supports time stamping in a way that is inadequate.

I *think* the cpsw/phyter combination would work with your patch, but only if the users disable CONFIG_TI_CPTS at compile time.

Quoted 7 lines

I think then the only solution is to move the decision how to handle get_ts_info into each MAC driver and get rid of:

	if (phy_has_tsinfo(phydev))
	        return phy_ts_info(phydev, info);

in __ethtool_get_ts_info().

Thinking about this more, that is an impossible task - there's no obvious information around to suggest which ethernet drivers could possibly be attached to a phylib PHY that supports PTP.

So, I think the only way to prevent a regression with the code as it is today is that we _never_ support PTP on Marvell PHYs - because doing so _will_ break the existing MVPP2 driver's implementation and cause a regression.

Right now, there is no option: if a PHY supports PTP, then the only option is to use the PHYs PTP. Which is utterly rediculous.

Unless you can see a way around it. Because I can't.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#11 lore raw
Quoted 4 lines · Russell King - ARM Linux admin

So, I think the only way to prevent a regression with the code as it is today is that we _never_ support PTP on Marvell PHYs - because doing so _will_ break the existing MVPP2 driver's implementation and cause a regression.

The situation isn't as bad as it seems.

For one thing, mvpp2 incorrectly selects NETWORK_PHY_TIMESTAMPING. It really shouldn't. I'm submitting a fix soon.

As long as the new PHY driver (or at least the PTP bit) depends on NETWORK_PHY_TIMESTAMPING, then that allows users who _really_ want that to enable the option at compile time. This option adds extra checks into the networking hot path, and almost everyone should avoid enabling it.

Quoted 4 lines

Right now, there is no option: if a PHY supports PTP, then the only option is to use the PHYs PTP. Which is utterly rediculous.

Unless you can see a way around it. Because I can't.

I still think the original and best method is to hide the two (and with your new driver, three) esoteric PHY time stamping drivers behind a Kconfig option, and structure the code to favor PHY devices.

The idea to favor the MACs, from back in July, doesn't really change the fundamental limitations that

- MAC and PHY time stamping cannot work simultaneously, and

- Users of PHY devices (representing a tiny minority) must enable the otherwise uninteresting NETWORK_PHY_TIMESTAMPING option at compile time.

Thanks,
Richard

#12 lore raw
Quoted 9 lines · Richard Cochran
Quoted 4 lines · Russell King - ARM Linux admin

So, I think the only way to prevent a regression with the code as it is today is that we _never_ support PTP on Marvell PHYs - because doing so _will_ break the existing MVPP2 driver's implementation and cause a regression.

The situation isn't as bad as it seems.

Quoted 8 lines

For one thing, mvpp2 incorrectly selects NETWORK_PHY_TIMESTAMPING. It really shouldn't. I'm submitting a fix soon.

As long as the new PHY driver (or at least the PTP bit) depends on NETWORK_PHY_TIMESTAMPING, then that allows users who _really_ want that to enable the option at compile time. This option adds extra checks into the networking hot path, and almost everyone should avoid enabling it.

As I already explained to you, you can *NOT* use kernel configuration to make the choice. ARM is a multi-platform kernel, and we will not stand for platform choices dictated by kernel configuration options.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#13 lore raw
Quoted 3 lines · Russell King - ARM Linux admin

As I already explained to you, you can *NOT* use kernel configuration to make the choice. ARM is a multi-platform kernel, and we will not stand for platform choices dictated by kernel configuration options.

This has nothing to do with ARM as a platform. The same limitation applies to x86 and all the rest.

The networking stack does not support simultaneous PHY and MAC time stamping.

If you enable both MAC and PHY time stamping, then only one will be reported, and that one is the PHY.

Thanks,
Richard

maintainer
#14 lore raw
Quoted 10 lines · Richard Cochran
Quoted 3 lines · Russell King - ARM Linux admin

As I already explained to you, you can *NOT* use kernel configuration to make the choice. ARM is a multi-platform kernel, and we will not stand for platform choices dictated by kernel configuration options.

This has nothing to do with ARM as a platform. The same limitation applies to x86 and all the rest.

The networking stack does not support simultaneous PHY and MAC time stamping.

Hi Richard

And this appears to be the crux of the problem.

You say PHY timestamping is exotic. I would disagree with that. Many Marvell PHYs support it, using similar hardware as to what is in the Marvell DSA switches. The Aquantia PHYs support it, but have no driver support at the moment. Microsemi PHYs have driver support for it. There are Broadcom PHYs with the needed hardware. So does some of the Qualcom PHYs. There are automotive T1 PHYs with PTP hardware support. Some Realtek devices have the hardware.

There is a growing interesting in PTP, the number of drivers keeps going up. The likelihood of MAC/PHY combination having two timestamping sources is growing all the time. So the stack needs to change to support the selection of the timestamp source.

Andrew

#15 lore raw
Quoted 4 lines · Andrew Lunn

There is a growing interesting in PTP, the number of drivers keeps going up. The likelihood of MAC/PHY combination having two timestamping sources is growing all the time. So the stack needs to change to support the selection of the timestamp source.

Fine, but How should the support look like?

- New/extended time stamping API that delivers multiple time stamps?

- sysctl to select MAC/PHY preference at run time globally?

- per-interface ethtool control?

- per-socket control? (probably not feasible, but heh)

Back of the napkin design ideas appreciated!

Thanks,
Richard

maintainer
#16 lore raw
Quoted 11 lines · Richard Cochran
Quoted 4 lines · Andrew Lunn

There is a growing interesting in PTP, the number of drivers keeps going up. The likelihood of MAC/PHY combination having two timestamping sources is growing all the time. So the stack needs to change to support the selection of the timestamp source.

Fine, but How should the support look like?

- New/extended time stamping API that delivers multiple time stamps?

Quoted 7 lines

- sysctl to select MAC/PHY preference at run time globally?

- per-interface ethtool control?

- per-socket control? (probably not feasible, but heh)

Back of the napkin design ideas appreciated!

Do you know of any realistic uses cases for using two time stampers for the same netdev? If we can eliminate that, then it is down to selecting which one to use. And i would say, the selection needs to be per netdev.

I don't know the ptp subsystem very well, but it seems like ptp_clock_register() does not know which netdev the device is being registered against. Is it guaranteed that parent is actual a MAC? Seems like adding a netdev member to that call could be good, so the core knows what stampers are available per netdev. It then becomes possible to enumerate the stampers associated to a netdev, and to select one to be used.

get_ts_info() is a problem because the MAC directly implements it. It seems like you need all kAPI calls to go into the ptp core. It can then call into the selected driver. So ptp_clock_info might need additional methods, get_ts_info() for example.

Andrew

#17 lore raw
Quoted 16 lines · Andrew Lunn
Quoted 15 lines · Richard Cochran
Quoted 4 lines · Andrew Lunn

There is a growing interesting in PTP, the number of drivers keeps going up. The likelihood of MAC/PHY combination having two timestamping sources is growing all the time. So the stack needs to change to support the selection of the timestamp source.

Fine, but How should the support look like?

- New/extended time stamping API that delivers multiple time stamps?

- sysctl to select MAC/PHY preference at run time globally?

- per-interface ethtool control?

Quoted 8 lines
Quoted 3 lines

- per-socket control? (probably not feasible, but heh)

Back of the napkin design ideas appreciated!

Do you know of any realistic uses cases for using two time stampers for the same netdev? If we can eliminate that, then it is down to selecting which one to use. And i would say, the selection needs to be per netdev.

Yes, I think it needs to be per-netdev.

As I've already demonstrated in previous emails last year on this subject, we have the situation where, on the Macchiatobin board, the ability to sync the PTP clock is quite different between the PHY and the MAC:

- the PHY (slow to access, only event capture which may not be wired, doesn't seem to synchronise well - delay of 58000, frequency changes every second between +/-1500ppb.)

- the Ethernet MAC (fast to access, supports event capture and trigger generation which also may not be wired, synchronises well, delay of 700, frequency changes every second +/- 40ppb.)

These are not theoretical numbers, they are observed measurements on real hardware. The MAC is 37 times more stable than the PHY and 82 times faster to access, which makes the MAC a higher quality clock, and more desirable to use.

Much of the instabillity comes from the need to access the PHY over MDIO - which introduces unpredictable latency, such as due to the bus access locking with other PHYLIB activities delaying PTP accesses.

While it is true that a PHY _may_ be closer to the wire in terms of its ability to timestamp the packet, if the quality of its clock that it is stamping with is significantly less than that which is achievable from the MAC, then "always use a PHY if present because it's more accurate" is a _very_ false assumption to make.

Things could well be different /if/ the PHY that was being used had usable hardware synchronisation (including hardware event capture, and hardware adjustment of the tick rate.) At that point, the PHY would be the device to use.

So, if we have the case where we have timestamping available at both a PHY and a MAC, the choice of which should be used, even for one SoC containing a mvpp2 network device, is not clear-cut "always use mvpp2" or "always use PHY".

Now, what I complain about is what I see as a messed up interface in the PTP world:

- get_ts_info is an ethtool method, where _if_ the PHY supports timestamping, get_ts_info() will _always_ be directed to the PHY, and the MAC driver gets no choice in the matter.

- the IOCTLs to control timestamping go via the .ndo_ioctl interface, where, typically, the MAC driver intercepts the calls that it is interested in, deferring any calls it is not to phy_ioctl() or similar. It is inside phy_ioctl() that we find the rest of the PTP interface.

So, right now, if we end up where, say, we have the mvpp2 driver with a PHY supporting timestamping, we get get_ts_info() reporting what the PHY is capable of, but the IOCTLs that actually control PTP get intercepted and acted upon by the mvpp2 driver.

The solution used by fec_main.c is to detect whether the PHY supports timestamping using phy_has_hwtstamp(phydev), and defer the IOCTLs to phy_ioctl(). That is fine if you want the PHY PTP implementation to always override the MAC PTP implementation.

If you want the other way around, then currently it is impossible because the MAC driver has no choice in whether it sees the get_ts_info() ethtool call.

So, if I were to merge my Marvell PHY PTP code, let's say for argument sake that NETWORK_PHY_TIMESTAMPING dependency was dropped from mvpp2, and a distro comes along and provides its users with a single kernel (as is the case today) which is built with NETWORK_PHY_TIMESTAMPING turned on (because some of the platforms require it) then there is _no_ choice to use the more stable MAC PTP implementation. People would have to suffer an inferior PHY based implementation despite the hardware being capable of better.

We should be giving people the choice of which they want to use.

So, can I suggest that we make a step to _first_ santise how PTP is dealt with in the network layer. Can I suggest that we have a struct net_ptp_ops (or whatever name you care) that is attached to a a net_device structure that contains _all_ the operations to do with a PTP implementation - get_ts_info() and the appropriate PTP ioctls. At that point, we have a much saner definition of how the networking layer talks to the PTP layer, and the decision about whether to use a PHY or a MAC becomes one of which set of operations the pointer to struct net_ptp_ops are pointing at - and we kill the struct ethtool_ops get_ts_info member, and the parsing of the ioctl numbers in network and PHY drivers.

At that point, we have some options. We can think about having a list of PTP clocks attached to a network device, and an API to userspace to select between them. Alternatively, we could do what is done with clocksources, where we provide a rating that determines their "quality" for use, so the "best" can be selected automatically by the kernel. Or whatever other scheme takes our fancy.

The problem right now is we're tied in to the current approach that the PHY always takes precedence, and we must have code in every network driver that has PTP support to defer to the PHY implementation in its ioctl handler.

Does this sound like a sane first step?

(Please note, I've spent since yesterday evening tracking down a horrid KVM regression with 32-bit ARM kernels post 5.6, and we're not yet done with it, so I may not have lots of time to work on this until that is properly sorted - hence my short replies earlier today as I didn't have time for anything else until now. Sorry about that.)

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#18 lore raw
In reply to Richard Cochran
Quoted 7 lines · Richard Cochran
Quoted 3 lines · Russell King - ARM Linux admin

As I already explained to you, you can *NOT* use kernel configuration to make the choice. ARM is a multi-platform kernel, and we will not stand for platform choices dictated by kernel configuration options.

This has nothing to do with ARM as a platform. The same limitation applies to x86 and all the rest.

You don't plainly don't understand, because you just called ARM a platform. ARM isn't a platform, it's an architecture, which is a totally different thing.

A platform is a device, like a phone, or a router, or a computer. However, all platforms may have the same CPU architecture - that being ARM.

ARM as an architecture is in the situation where distributions have pushed hard for it to be like x86. You build one kernel for your distribution, not multiple hardware specific kernels. That means, there is one kernel configuration. Platform specifics are handled through firmware such as device tree.

Having platform specific kernel configuration options, such as "we want to use PHY timestamping" vs "we want to use MAC timestamping" do not work in this environment.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!