Skip to contentSkip to conversation

net: phy: add Marvell PHY PTP support

RFC PATCH In review Russell King 9 participants 74 messages Jul 14, 2020 – Apr 28, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 net: phy: add Marvell PHY PTP support

Add PTP basic support for Marvell 88E151x PHYs. These PHYs support timestamping the egress and ingress of packets, but does not support any packet modification, nor do we support any filtering beyond selecting packets that the hardware recognises as PTP/802.1AS.

The PHYs support hardware pins for providing an external clock for the TAI counter, and a separate pin that can be used for event capture or generation of a trigger (either a pulse or periodic). This code does not support either of these modes.

We currently use a delayed work to poll for the timestamps which is far from ideal, but we also provide a function that can be called from an interrupt handler - which would be good to tie into the main Marvell PHY driver.

The driver takes inspiration from the Marvell 88E6xxx DSA and DP83640 drivers. The hardware is very similar to the implementation found in the 88E6xxx DSA driver, but the access methods are very different, although it may be possible to create a library that both can use along with accessor functions.

---

This does not wire up IRQ support yet - I don't think I have a system I can test that with, but I need to check.

Tested with linuxptp; using phc2sys to synchronise the clock on the PHY with system time, and that seems to work reasonably well - but every so often (randomly) there seems to be a large delay in reading the time, which results in a large shift in the frequency setting, and then it takes a while to re-settle. I'm not sure where that is coming from.

I also haven't checked for other Marvell PHYs supporting this; I know the 88E1111 does not, and the 154x family also seem not to support it. However, the 151x family for x=0,2,4,8 do. The door is partly left open in the code to support a common TAI with multiple ports, but that would require further work - but as the 151x PHYs are only single port PHYs, it seemed pointless to add support immediately.

The 151x family does have two pins that are related to the PTP/TAI support:
- a pin which can be used for event capture, generating a pulse on TAI counter match, or generating a 50% duty cycle clock. In the latter case, defining when the first edge occurs does not seem to be possible.
- a pin which can be supplied an external clock instead of using the internal 125MHz clock to increment the TAI counter.

Getting the Kconfig for this correct has been a struggle - particularly the combination where PTP support is modular. It's rather odd to have the Marvell PTP support asked before the Marvell PHY support. I couldn't work out any other reasonable way to ensure that we always have a valid configuration, without leading to stupidities such as having the PTP and Marvell PTP support modular, but non-functional because Marvell PHY is built-in.

With this driver, there appears to be three instances of a ptp_header() like function in the kernel - I think that ought to become a helper in generic code.

It seems that the Marvell PHY PTP is very similar to that found in their DSA chips, which suggests maybe we should share the code, but different access methods would be required.

There's a hack in here to implement the gettimex64() support, as gettime64() is marked as deprecated - this is the best solution I could come up with to support the new function while still using the timecounter/cyclecounter support.

 drivers/net/phy/Kconfig       |  12 +
 drivers/net/phy/Makefile      |   2 +
 drivers/net/phy/marvell.c     |  28 +-
 drivers/net/phy/marvell_ptp.c | 691 ++++++++++++++++++++++++++++++++++
 drivers/net/phy/marvell_ptp.h |  20 +
 drivers/net/phy/marvell_tai.c | 279 ++++++++++++++
 drivers/net/phy/marvell_tai.h |  36 ++
 7 files changed, 1066 insertions(+), 2 deletions(-)
 create mode 100644 drivers/net/phy/marvell_ptp.c
 create mode 100644 drivers/net/phy/marvell_ptp.h
 create mode 100644 drivers/net/phy/marvell_tai.c
 create mode 100644 drivers/net/phy/marvell_tai.h
drivers/net/phy/Kconfig +12 -0
index dd20c2c27c2f..e5bd5b779a0d 100644
@@ -451,8 +451,20 @@ config LXT_PHY
 	help
 	  Currently supports the lxt970, lxt971
 
+config MARVELL_PHY_PTP
+	tristate "Marvell PHY PTP support"
+	depends on NETWORK_PHY_TIMESTAMPING && PTP_1588_CLOCK
+	help
+	  Support PHY timestamping on Marvell 88E1510, 88E1512, 88E1514
+	  and 88E1518 PHYs.
+
+	  N.B. In order for this to be fully functional, your MAC driver
+	  must call the skb_tx_timestamp() function.
+
 config MARVELL_PHY
 	tristate "Marvell PHYs"
+	# This can't be builtin if the PTP support is modular
+	depends on MARVELL_PHY_PTP=n || MARVELL_PHY_PTP
 	help
 	  Currently has a driver for the 88E1011S
 
drivers/net/phy/Makefile +2 -0
index d84bab489a53..00ba7be63246 100644
@@ -87,6 +87,8 @@ obj-$(CONFIG_INTEL_XWAY_PHY)	+= intel-xway.o
 obj-$(CONFIG_LSI_ET1011C_PHY)	+= et1011c.o
 obj-$(CONFIG_LXT_PHY)		+= lxt.o
 obj-$(CONFIG_MARVELL_PHY)	+= marvell.o
+obj-$(CONFIG_MARVELL_PHY_PTP)	+= marvell-ptp.o
+marvell-ptp-y			:= marvell_ptp.o marvell_tai.o
 obj-$(CONFIG_MARVELL_10G_PHY)	+= marvell10g.o
 obj-$(CONFIG_MESON_GXL_PHY)	+= meson-gxl.o
 obj-$(CONFIG_MICREL_KS8995MA)	+= spi_ks8995.o
drivers/net/phy/marvell.c +26 -2
index bb86ac0bd092..ef1ac619db3c 100644
@@ -37,6 +37,8 @@
 #include <asm/irq.h>
 #include <linux/uaccess.h>
 
+#include "marvell_ptp.h"
+
 #define MII_MARVELL_PHY_PAGE		22
 #define MII_MARVELL_COPPER_PAGE		0x00
 #define MII_MARVELL_FIBER_PAGE		0x01
@@ -2589,6 +2591,27 @@ static int m88e1121_probe(struct phy_device *phydev)
 {
 	int err;
 
+	err = marvell_probe(phydev);
+	if (err)
+		return err;
+
+	err = marvell_ptp_probe(phydev);
+	if (err)
+		return err;
+
+	err = m88e1510_hwmon_probe(phydev);
+	if (err) {
+		marvell_ptp_remove(phydev);
+		return err;
+	}
+
+	return 0;
+}
+
+static int m88e154x_probe(struct phy_device *phydev)
+{
+	int err;
+
 	err = marvell_probe(phydev);
 	if (err)
 		return err;
@@ -2823,6 +2846,7 @@ static struct phy_driver marvell_drivers[] = {
 		.features = PHY_GBIT_FIBRE_FEATURES,
 		.flags = PHY_POLL_CABLE_TEST,
 		.probe = m88e1510_probe,
+		.remove = marvell_ptp_remove,
 		.config_init = m88e1510_config_init,
 		.config_aneg = m88e1510_config_aneg,
 		.read_status = marvell_read_status,
@@ -2851,7 +2875,7 @@ static struct phy_driver marvell_drivers[] = {
 		.name = "Marvell 88E1540",
 		/* PHY_GBIT_FEATURES */
 		.flags = PHY_POLL_CABLE_TEST,
-		.probe = m88e1510_probe,
+		.probe = m88e154x_probe,
 		.config_init = marvell_config_init,
 		.config_aneg = m88e1510_config_aneg,
 		.read_status = marvell_read_status,
@@ -2875,7 +2899,7 @@ static struct phy_driver marvell_drivers[] = {
 		.phy_id = MARVELL_PHY_ID_88E1545,
 		.phy_id_mask = MARVELL_PHY_ID_MASK,
 		.name = "Marvell 88E1545",
-		.probe = m88e1510_probe,
+		.probe = m88e154x_probe,
 		/* PHY_GBIT_FEATURES */
 		.flags = PHY_POLL_CABLE_TEST,
 		.config_init = marvell_config_init,
drivers/net/phy/marvell_ptp.c new file +691 -0

691 lines not on this page · raw message

drivers/net/phy/marvell_ptp.h new file +20 -0
new file mode 100644
index 000000000000..1ae65bb5c298
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef MARVELL_PTP_H
+#define MARVELL_PTP_H
+
+#if IS_ENABLED(CONFIG_MARVELL_PHY_PTP)
+void marvell_ptp_check(struct phy_device *phydev);
+int marvell_ptp_probe(struct phy_device *phydev);
+void marvell_ptp_remove(struct phy_device *phydev);
+#else
+static inline int marvell_ptp_probe(struct phy_device *phydev)
+{
+	return 0;
+}
+
+static inline void marvell_ptp_remove(struct phy_device *phydev)
+{
+}
+#endif
+
+#endif
drivers/net/phy/marvell_tai.c new file +279 -0

279 lines not on this page · raw message

drivers/net/phy/marvell_tai.h new file +36 -0
new file mode 100644
index 000000000000..9752f6cd58da
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef MARVELL_TAI_H
+#define MARVELL_TAI_H
+
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/ptp_clock_kernel.h>
+#include <linux/timecounter.h>
+
+struct phy_device;
+
+struct marvell_tai {
+	struct list_head tai_node;
+	struct phy_device *phydev;
+
+	struct ptp_clock_info caps;
+	struct ptp_clock *ptp_clock;
+
+	u32 cc_mult_num;
+	u32 cc_mult_den;
+	u32 cc_mult;
+
+	struct mutex mutex;
+	struct timecounter timecounter;
+	struct cyclecounter cyclecounter;
+	long half_overflow_period;
+
+	/* Used while reading the TAI */
+	struct ptp_system_timestamp *sts;
+};
+
+u64 marvell_tai_cyc2time(struct marvell_tai *tai, u32 cyc);
+int marvell_tai_get(struct marvell_tai **taip, struct phy_device *phydev);
+void marvell_tai_put(struct marvell_tai *tai);
+
+#endif
Signature · 1 line
2.20.1
maintainer
#1 lore raw
Quoted 7 lines

Getting the Kconfig for this correct has been a struggle - particularly the combination where PTP support is modular. It's rather odd to have the Marvell PTP support asked before the Marvell PHY support. I couldn't work out any other reasonable way to ensure that we always have a valid configuration, without leading to stupidities such as having the PTP and Marvell PTP support modular, but non-functional because Marvell PHY is built-in.

Hi Russell

How much object code is this adding? All the other PHYs which support PTP just make it part of the PHY driver, not a standalone module. That i guess simplifies the conditions.

Looking at DSDT, it lists

        case MAD_88E1340S:
        case MAD_88E1340:
        case MAD_88E1340M:
        case MAD_SWG65G : 
	case MAD_88E151x:

as being MAD_PHY_PTP_TAI_CAPABLE;

and

	case MAD_88E1548
        case MAD_88E1680:
        case MAD_88E1680M:

as MAD_PHY_1STEP_PTP_CAPABLE;

So maybe we can wire this up to a few more PHYs to 'lower' the overhead a bit?

Quoted 3 lines

It seems that the Marvell PHY PTP is very similar to that found in their DSA chips, which suggests maybe we should share the code, but different access methods would be required.

That makes the Kconfig even more complex :-(

Andrew

#2 lore raw
Quoted 8 lines · Andrew Lunn
Quoted 7 lines

Getting the Kconfig for this correct has been a struggle - particularly the combination where PTP support is modular. It's rather odd to have the Marvell PTP support asked before the Marvell PHY support. I couldn't work out any other reasonable way to ensure that we always have a valid configuration, without leading to stupidities such as having the PTP and Marvell PTP support modular, but non-functional because Marvell PHY is built-in.

Quoted 5 lines

Hi Russell

How much object code is this adding? All the other PHYs which support PTP just make it part of the PHY driver, not a standalone module. That i guess simplifies the conditions.

Taking an arm64 build, the PHY driver is 16k and the PTP driver comes in just under 8k.

Quoted 12 lines

Looking at DSDT, it lists

        case MAD_88E1340S:
        case MAD_88E1340:
        case MAD_88E1340M:
        case MAD_SWG65G : 
	case MAD_88E151x:

as being MAD_PHY_PTP_TAI_CAPABLE;

and

Quoted 8 lines
	case MAD_88E1548
        case MAD_88E1680:
        case MAD_88E1680M:

as MAD_PHY_1STEP_PTP_CAPABLE;

So maybe we can wire this up to a few more PHYs to 'lower' the overhead a bit?

That's interesting, the 1548 information (covering 1543 and 1545 as well) I have doesn't mention anything about PTP.

Quoted 5 lines
Quoted 3 lines

It seems that the Marvell PHY PTP is very similar to that found in their DSA chips, which suggests maybe we should share the code, but different access methods would be required.

That makes the Kconfig even more complex :-(

We already have that complexity due to the fact that we are interacting with two subsystems. The 88e6xxx Kconfig entry has:

config NET_DSA_MV88E6XXX_PTP
bool "PTP support for Marvell 88E6xxx"
default n
depends on NET_DSA_MV88E6XXX_GLOBAL2
depends on PTP_1588_CLOCK
imply NETWORK_PHY_TIMESTAMPING

and I've been wondering what that means when PTP_1588_CLOCK=m while NET_DSA_MV88E6XXX_GLOBAL2=y and NET_DSA_MV88E6XXX=y. If this is selectable, then it seems to be misleading the user - you can't have the PTP subsystem modular, and have PTP drivers built-in to the kernel.

Yes, we have the inteligence to be able to make the various PTP calls be basically no-ops, but it's not nice.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#3 lore raw
Quoted 58 lines · Russell King - ARM Linux admin
Quoted 8 lines · Andrew Lunn
Quoted 7 lines

Getting the Kconfig for this correct has been a struggle - particularly the combination where PTP support is modular. It's rather odd to have the Marvell PTP support asked before the Marvell PHY support. I couldn't work out any other reasonable way to ensure that we always have a valid configuration, without leading to stupidities such as having the PTP and Marvell PTP support modular, but non-functional because Marvell PHY is built-in.

Quoted 5 lines

Hi Russell

How much object code is this adding? All the other PHYs which support PTP just make it part of the PHY driver, not a standalone module. That i guess simplifies the conditions.

Taking an arm64 build, the PHY driver is 16k and the PTP driver comes in just under 8k.

Quoted 12 lines

Looking at DSDT, it lists

        case MAD_88E1340S:
        case MAD_88E1340:
        case MAD_88E1340M:
        case MAD_SWG65G : 
	case MAD_88E151x:

as being MAD_PHY_PTP_TAI_CAPABLE;

and

Quoted 8 lines
	case MAD_88E1548
        case MAD_88E1680:
        case MAD_88E1680M:

as MAD_PHY_1STEP_PTP_CAPABLE;

So maybe we can wire this up to a few more PHYs to 'lower' the overhead a bit?

That's interesting, the 1548 information (covering 1543 and 1545 as well) I have doesn't mention anything about PTP.

Quoted 5 lines
Quoted 3 lines

It seems that the Marvell PHY PTP is very similar to that found in their DSA chips, which suggests maybe we should share the code, but different access methods would be required.

That makes the Kconfig even more complex :-(

We already have that complexity due to the fact that we are interacting with two subsystems. The 88e6xxx Kconfig entry has:

config NET_DSA_MV88E6XXX_PTP
bool "PTP support for Marvell 88E6xxx"
default n
depends on NET_DSA_MV88E6XXX_GLOBAL2
depends on PTP_1588_CLOCK
imply NETWORK_PHY_TIMESTAMPING

Quoted 8 lines

and I've been wondering what that means when PTP_1588_CLOCK=m while NET_DSA_MV88E6XXX_GLOBAL2=y and NET_DSA_MV88E6XXX=y. If this is selectable, then it seems to be misleading the user - you can't have the PTP subsystem modular, and have PTP drivers built-in to the kernel.

Yes, we have the inteligence to be able to make the various PTP calls be basically no-ops, but it's not nice.

Sure enough:

CONFIG_NET_DSA_MV88E6XXX=y
CONFIG_NET_DSA_MV88E6XXX_GLOBAL2=y
CONFIG_NET_DSA_MV88E6XXX_PTP=y
CONFIG_PTP_1588_CLOCK=m

is a possible configuration, but all the PTP calls from mv88e6xxx are stubbed out (since the IS_REACHABLE() in linux/ptp_clock_kernel.h is false.)

The DP83640 PHY works around this by introducing a hard dependency on PTP:

config DP83640_PHY
tristate "Driver for the National Semiconductor DP83640 PHYTER" depends on NETWORK_PHY_TIMESTAMPING
depends on PHYLIB
depends on PTP_1588_CLOCK

A possible solution to this would be for the PTP core code to be a separate Kconfig symbol that is not offered to the user, but is selected by the various drivers and Kconfig entries that need that support. That way, it would automatically be modular or built-in as required by its users.

PTP_1588_CLOCK could then be used as a global enable for PTP support where appropriate. We can also avoid all the Kconfig complexity introduced by having two independent subsystems either of which could be modular.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#4 lore raw
Quoted 5 lines · Russell King - ARM Linux admin

PTP_1588_CLOCK could then be used as a global enable for PTP support where appropriate. We can also avoid all the Kconfig complexity introduced by having two independent subsystems either of which could be modular.

I'm sorry about the Kconfig confusion. I think it all started with the tiny-fication work and the new "select" Kconfig keyword. The purpose of all this was, IIRC, to allow leaving dynamic posix clocks out of the kernel, but it still makes my head spin.

Thanks,
Richard

#5 lore raw
In reply to Russell King
Quoted 3 lines · Russell King

With this driver, there appears to be three instances of a ptp_header() like function in the kernel - I think that ought to become a helper in generic code.

Yes, and the hellcreek DSA driver will add yet another. I'll be happy if patches that refactor everything this magically appear!

Quoted 5 lines
drivers/net/phy/marvell_ptp.c new file +5 -0
+/* Read the status, timestamp and PTP common header sequence from the PHY.
+ * Apparently, reading these are atomic, but there is no mention how the
+ * PHY treats this access as atomic. So, we set the DisTSOverwrite bit
+ * when configuring the PHY.
+ */

drivers/net/phy/marvell_ptp.c line 96

Quoted 10 lines
drivers/net/phy/marvell_ptp.c new file +10 -0
+static int marvell_read_tstamp(struct phy_device *phydev,
+			       struct marvell_ts *ts,
+			       uint8_t page, uint8_t reg)
+{
+	int oldpage;
+	int ret;
+
+	/* Read status register */
+	oldpage = phy_select_page(phydev, page);
+	if (oldpage >= 0) {

Suggest avoiding the IfOk anti-pattern.

Quoted 84 lines
drivers/net/phy/marvell_ptp.c new file +82 -0
+		ret = __phy_read(phydev, reg);
+		if (ret < 0)
+			goto restore;
+
+		ts->stat = ret;
+		if (!(ts->stat & MV_STATUS_VALID)) {
+			ret = 0;
+			goto restore;
+		}
+
+		/* Read low timestamp */
+		ret = __phy_read(phydev, reg + 1);
+		if (ret < 0)
+			goto restore;
+
+		ts->time = ret;
+
+		/* Read high timestamp */
+		ret = __phy_read(phydev, reg + 2);
+		if (ret < 0)
+			goto restore;
+
+		ts->time |= ret << 16;
+
+		/* Read sequence */
+		ret = __phy_read(phydev, reg + 3);
+		if (ret < 0)
+			goto restore;
+
+		ts->seq = ret;
+
+		/* Clear valid */
+		__phy_write(phydev, reg, 0);
+	}
+restore:
+	return phy_restore_page(phydev, oldpage, ret);
+}
 
 
+/* Check whether the packet is suitable for timestamping, and if so,
+ * try to find a pending timestamp for it. If no timestamp is found,
+ * queue the packet with a timeout.
+ */
+static bool marvell_ptp_rxtstamp(struct mii_timestamper *mii_ts,
+				 struct sk_buff *skb, int type)
+{
+	struct marvell_ptp *ptp = mii_ts_to_ptp(mii_ts);
+	struct marvell_rxts *rxts;
+	bool found = false;
+	u8 *ptp_hdr;
+	u16 seqid;
+	u64 ns;
+
+	if (ptp->rx_filter == HWTSTAMP_FILTER_NONE)
+		return false;
+
+	ptp_hdr = ptp_header(skb, type);
+	if (!ptp_hdr)
+		return false;
+
+	seqid = ptp_seqid(ptp_hdr);
+
+	mutex_lock(&ptp->rx_mutex);
+
+	/* Search the pending receive timestamps for a matching seqid */
+	list_for_each_entry(rxts, &ptp->rx_pend, node) {
+		if (rxts->seq == seqid) {
+			found = true;
+			ns = rxts->ns;
+			/* Move this timestamp entry to the free list */
+			list_move_tail(&rxts->node, &ptp->rx_free);
+			break;
+		}
+	}
+
+	if (!found) {
+		/* Store the seqid and queue the skb. Do this under the lock
+		 * to ensure we don't miss any timestamps appended to the
+		 * rx_pend list.
+		 */
+		MARVELL_PTP_CB(skb)->seq = seqid;
+		MARVELL_PTP_CB(skb)->timeout = jiffies +
+			msecs_to_jiffies(RX_TIMEOUT_MS);
+		__skb_queue_tail(&ptp->rx_queue, skb);

drivers/net/phy/marvell_ptp.c line 308

Quoted 10 lines
drivers/net/phy/marvell_ptp.c new file +10 -0
+	}
+
+	mutex_unlock(&ptp->rx_mutex);
+
+	if (found) {
+		/* We found the corresponding timestamp. If we can add the
+		 * timestamp, do we need to go through the netif_rx_ni()
+		 * path, or would it be more efficient to add the timestamp
+		 * and return "false" here?
+		 */

The caller expects the driver to deliver the skb. If you return the skb with your hwtstamp value, there is no guarantee that the caller won't clobber it.

drivers/net/phy/marvell_ptp.c line 311

Quoted 3 lines
drivers/net/phy/marvell_ptp.c new file +3 -0
+		marvell_ptp_rx(skb, ns);
+	} else {
+		schedule_delayed_work(&ptp->ts_work, 2);

Instead of using a separate work item, consider combining this with the counter overflow logic in caps.do_aux_work = marvell_tai_aux_work. The advantage of do_aux_work is that you get your own kernel thread which can be given scheduling priority via chrt on busy systems.

Quoted 26 lines
drivers/net/phy/marvell_ptp.c new file +24 -0
+	}
+
+	return true;
+}
 
 
+/* Check whether the skb will be timestamped on transmit; we only support
+ * a single outstanding skb. Add it if the slot is available.
+ */
+static bool marvell_ptp_do_txtstamp(struct mii_timestamper *mii_ts,
+				    struct sk_buff *skb, int type)
+{
+	struct marvell_ptp *ptp = mii_ts_to_ptp(mii_ts);
+	u8 *ptp_hdr;
+
+	if (ptp->tx_type != HWTSTAMP_TX_ON)
+		return false;
+
+	if (!(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP))
+		return false;
+
+	ptp_hdr = ptp_header(skb, type);
+	if (!ptp_hdr)
+		return false;
+
+	MARVELL_PTP_CB(skb)->seq = ptp_seqid(ptp_hdr);

drivers/net/phy/marvell_ptp.c line 429

Quoted 10 lines
drivers/net/phy/marvell_ptp.c new file +10 -0
+	MARVELL_PTP_CB(skb)->timeout = jiffies +
+		msecs_to_jiffies(TX_TIMEOUT_MS);
+
+	if (cmpxchg(&ptp->tx_skb, NULL, skb) != NULL)
+		return false;
+
+	/* DP83640 marks the skb for hw timestamping. Since the MAC driver
+	 * may call skb_tx_timestamp() but may not support timestamping
+	 * itself, it may not set this flag. So, we need to do this here.
+	 */

The purpose of this flag is to prevent a SW time stamp from being delivered when the user dialed HW time stamps only. The PHY driver is expected to set this flag all by itself regardless of the MAC driver. Unfortunately the stack doesn't support simultaneous MAC and PHY time stamping, and so it is up to the MAC driver to avoid setting this flag when a time stamping PHY is attached.

Quoted 52 lines
drivers/net/phy/marvell_ptp.c new file +52 -0
+	skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
+	schedule_delayed_work(&ptp->ts_work, 2);
+
+	return true;
+}
+
+static void marvell_ptp_txtstamp(struct mii_timestamper *mii_ts,
+				 struct sk_buff *skb, int type)
+{
+	if (!marvell_ptp_do_txtstamp(mii_ts, skb, type))
+		kfree_skb(skb);
+}
+
+static int marvell_ptp_hwtstamp(struct mii_timestamper *mii_ts,
+				struct ifreq *ifreq)
+{
+	struct marvell_ptp *ptp = mii_ts_to_ptp(mii_ts);
+	struct hwtstamp_config config;
+	u16 cfg0 = PTP1_PORT_CONFIG_0_DISPTP;
+	u16 cfg2 = 0;
+	int err;
+
+	if (copy_from_user(&config, ifreq->ifr_data, sizeof(config)))
+		return -EFAULT;
+
+	if (config.flags)
+		return -EINVAL;
+
+	switch (config.tx_type) {
+	case HWTSTAMP_TX_OFF:
+		break;
+
+	case HWTSTAMP_TX_ON:
+		cfg0 = 0;
+		cfg2 |= PTP1_PORT_CONFIG_2_DEPINTEN;
+		break;
+
+	default:
+		return -ERANGE;
+	}
+
+	switch (config.rx_filter) {
+	case HWTSTAMP_FILTER_NONE:
+		break;
+
+	case HWTSTAMP_FILTER_PTP_V1_L4_EVENT:
+	case HWTSTAMP_FILTER_PTP_V1_L4_SYNC:
+	case HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ:
+	case HWTSTAMP_FILTER_PTP_V2_L4_EVENT:
+	case HWTSTAMP_FILTER_PTP_V2_L4_SYNC:
+	case HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ:
+	case HWTSTAMP_FILTER_PTP_V2_L2_EVENT:

drivers/net/phy/marvell_ptp.c line 491

Quoted 10 lines
drivers/net/phy/marvell_ptp.c new file +10 -0
+	case HWTSTAMP_FILTER_PTP_V2_L2_SYNC:
+	case HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ:
+	case HWTSTAMP_FILTER_PTP_V2_EVENT:
+	case HWTSTAMP_FILTER_PTP_V2_SYNC:
+	case HWTSTAMP_FILTER_PTP_V2_DELAY_REQ:
+		/* We accept 802.1AS, IEEE 1588v1 and IEEE 1588v2. We could
+		 * filter on 802.1AS using the transportSpecific field, but
+		 * that affects the transmit path too.
+		 */
+		config.rx_filter = HWTSTAMP_FILTER_SOME;

I think you want HWTSTAMP_FILTER_PTP_V2_EVENT here.

From Documentation/networking/timestamping.rst...

Drivers are free to use a more permissive configuration than the requested configuration. It is expected that drivers should only implement directly the most generic mode that can be supported. For example if the hardware can support HWTSTAMP_FILTER_V2_EVENT, then it should generally always upscale HWTSTAMP_FILTER_V2_L2_SYNC_MESSAGE, and so forth, as HWTSTAMP_FILTER_V2_EVENT is more generic (and more useful to applications).

A driver which supports hardware time stamping shall update the struct with the actual, possibly more permissive configuration. If the requested packets cannot be time stamped, then nothing should be changed and ERANGE shall be returned (in contrast to EINVAL, which indicates that SIOCSHWTSTAMP is not supported at all).

Quoted 69 lines
drivers/net/phy/marvell_ptp.c new file +67 -0
+		Cfg0 = 0;
+		cfg2 |= PTP1_PORT_CONFIG_2_ARRINTEN;
+		break;
+
+	default:
+		return -ERANGE;
+	}
+
+	err = phy_modify_paged(ptp->phydev, MARVELL_PAGE_PTP_PORT_1,
+			       PTP1_PORT_CONFIG_0,
+			       PTP1_PORT_CONFIG_0_DISPTP, cfg0);
+	if (err)
+		return err;
+
+	err = phy_write_paged(ptp->phydev, MARVELL_PAGE_PTP_PORT_1,
+			      PTP1_PORT_CONFIG_2, cfg2);
+	if (err)
+		return err;
+
+	ptp->tx_type = config.tx_type;
+	ptp->rx_filter = config.rx_filter;
+
+	return copy_to_user(ifreq->ifr_data, &config, sizeof(config)) ?
+		-EFAULT : 0;
+}
 
 
+int marvell_tai_get(struct marvell_tai **taip, struct phy_device *phydev)
+{
+	struct marvell_tai *tai;
+	unsigned long overflow_ms;
+	int err;
+
+	err = marvell_tai_global_config(phydev);
+	if (err < 0)
+		return err;
+
+	tai = kzalloc(sizeof(*tai), GFP_KERNEL);
+	if (!tai)
+		return -ENOMEM;
+
+	mutex_init(&tai->mutex);
+
+	tai->phydev = phydev;
+
+	/* This assumes a 125MHz clock */
+	tai->cc_mult = 8 << 28;
+	tai->cc_mult_num = 1 << 9;
+	tai->cc_mult_den = 15625U;
+
+	tai->cyclecounter.read = marvell_tai_clock_read;
+	tai->cyclecounter.mask = CYCLECOUNTER_MASK(32);
+	tai->cyclecounter.mult = tai->cc_mult;
+	tai->cyclecounter.shift = 28;
+
+	overflow_ms = (1ULL << 32 * tai->cc_mult * 1000) >>
+			tai->cyclecounter.shift;
+	tai->half_overflow_period = msecs_to_jiffies(overflow_ms / 2);
+
+	timecounter_init(&tai->timecounter, &tai->cyclecounter,
+			 ktime_to_ns(ktime_get_real()));
+
+	tai->caps.owner = THIS_MODULE;
+	snprintf(tai->caps.name, sizeof(tai->caps.name), "Marvell PHY");
+	/* max_adj of 1000000 is what MV88E6xxx DSA uses */
+	tai->caps.max_adj = 1000000;
+	tai->caps.adjfine = marvell_tai_adjfine;
+	tai->caps.adjtime = marvell_tai_adjtime;
+	tai->caps.gettimex64 = marvell_tai_gettimex64;

drivers/net/phy/marvell_ptp.c

Quoted 10 lines
drivers/net/phy/marvell_ptp.c new file +10 -0
+	tai->caps.settime64 = marvell_tai_settime64;
+	tai->caps.do_aux_work = marvell_tai_aux_work;
+
+	tai->ptp_clock = ptp_clock_register(&tai->caps, &phydev->mdio.dev);
+	if (IS_ERR(tai->ptp_clock)) {
+		kfree(tai);
+		return PTR_ERR(tai->ptp_clock);
+	}
+
+	ptp_schedule_worker(tai->ptp_clock, tai->half_overflow_period);

ptp_clock_register() can return NULL, and so you should check for that case before passing tai->ptp_clock here.

* ptp_clock_register() - register a PTP hardware clock driver *
* @info: Structure describing the new clock.
* @parent: Pointer to the parent device of the new clock. *
* Returns a valid pointer on success or PTR_ERR on failure. If PHC
* support is missing at the configuration level, this function
* returns NULL, and drivers are expected to gracefully handle that
* case separately.

Thanks,
Richard

#6 lore raw

Hi Richard,

Quoted 8 lines · Richard Cochran
Quoted 3 lines · Russell King

With this driver, there appears to be three instances of a ptp_header() like function in the kernel - I think that ought to become a helper in generic code.

Yes, and the hellcreek DSA driver will add yet another. I'll be happy if patches that refactor everything this magically appear!

I'll post the next version of the hellcreek DSA driver probably next week. I can include a generic ptp_header() function if you like in that patch series. But, where to put it? ptp core or maybe ptp_classify?

Thanks,
Kurt

#7 lore raw
Quoted 3 lines · Kurt Kanzenbach

I'll post the next version of the hellcreek DSA driver probably next week. I can include a generic ptp_header() function if you like in that patch series. But, where to put it? ptp core or maybe ptp_classify?

Either place is fine with me. Maybe it makes most sense in ptp_classify? Please put the re-factoring in a separate patches, before the new driver.

Thanks,
Richard

#8 lore raw
Quoted 8 lines · Richard Cochran
Quoted 3 lines · Kurt Kanzenbach

I'll post the next version of the hellcreek DSA driver probably next week. I can include a generic ptp_header() function if you like in that patch series. But, where to put it? ptp core or maybe ptp_classify?

Either place is fine with me. Maybe it makes most sense in ptp_classify? Please put the re-factoring in a separate patches, before the new driver.

And maybe the new header parsing routine should provide a pointer to a structure with the message layout. Something similar to this (from the linuxptp user stack) might work.

struct ptp_header {
	uint8_t             tsmt; /* transportSpecific | messageType */
	uint8_t             ver;  /* reserved          | versionPTP  */
	UInteger16          messageLength;
	UInteger8           domainNumber;
	Octet               reserved1;
	Octet               flagField[2];
	Integer64           correction;
	UInteger32          reserved2;
	struct PortIdentity sourcePortIdentity;
	UInteger16          sequenceId;
	UInteger8           control;
	Integer8            logMessageInterval;
} PACKED;

Of course, the structure should use the kernel types that include the big endian annotations.

Thanks,
Richard

#9 lore raw
Quoted 14 lines · Richard Cochran
Quoted 8 lines · Richard Cochran
Quoted 3 lines · Kurt Kanzenbach

I'll post the next version of the hellcreek DSA driver probably next week. I can include a generic ptp_header() function if you like in that patch series. But, where to put it? ptp core or maybe ptp_classify?

Either place is fine with me. Maybe it makes most sense in ptp_classify? Please put the re-factoring in a separate patches, before the new driver.

And maybe the new header parsing routine should provide a pointer to a structure with the message layout. Something similar to this (from the linuxptp user stack) might work.

Quoted 14 lines
struct ptp_header {
	uint8_t             tsmt; /* transportSpecific | messageType */
	uint8_t             ver;  /* reserved          | versionPTP  */
	UInteger16          messageLength;
	UInteger8           domainNumber;
	Octet               reserved1;
	Octet               flagField[2];
	Integer64           correction;
	UInteger32          reserved2;
	struct PortIdentity sourcePortIdentity;
	UInteger16          sequenceId;
	UInteger8           control;
	Integer8            logMessageInterval;
} PACKED;

Yes, makes sense. It should work.

Quoted 4 lines

Of course, the structure should use the kernel types that include the big endian annotations.

Sure.

Thanks,
Kurt

#10 lore raw
In reply to Russell King
Quoted 5 lines · Russell King

The driver takes inspiration from the Marvell 88E6xxx DSA and DP83640 drivers. The hardware is very similar to the implementation found in the 88E6xxx DSA driver, but the access methods are very different, although it may be possible to create a library that both can use along with accessor functions.

I think this definitely needs to become a library - I've just been looking at mvneta in Armada 388. One of the three interfaces has PTP support, and it looks like the register set is again very similar to the 88E151x and 88E6xxx PTP/TAI register layouts, but has yet another different access method.

We certainly don't want three copies of almost identical code...

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#11 lore raw
In reply to Russell King
Quoted 4 lines · Russell King

Add PTP basic support for Marvell 88E151x PHYs. These PHYs support timestamping the egress and ingress of packets, but does not support any packet modification, nor do we support any filtering beyond selecting packets that the hardware recognises as PTP/802.1AS.

A question has come up concerning the selection of PTP timestamping sources within a network device.

I have the situation on a couple of devices where there are multiple places that can do PTP timestamping:

- the PHY (slow to access, only event capture which may not be wired, doesn't seem to synchronise well - delay of 58000, frequency changes every second between +/-1500ppb.)
- the Ethernet MAC (fast to access, supports event capture and trigger generation which also may not be wired, synchronises well, delay of 700, frequency changes every second +/- 40ppb.)

How do we deal with this situation - from what I can see from the ethtool API, we have to make a choice about which to use. How do we make that choice?

It's not a case of "just implement one" since hardware may have both available on a particular ethernet interface or just one available.

Do we need a property to indicate whether we wish to use the PHY or MAC PTP stamping, or something more elaborate?

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#12 lore raw
Quoted 3 lines · Russell King - ARM Linux admin

How do we deal with this situation - from what I can see from the ethtool API, we have to make a choice about which to use. How do we make that choice?

Unfortunately the stack does not implement simultaneous MAC + PHY time stamping. If your board has both, then you make the choice to use the PHY by selecting NETWORK_PHY_TIMESTAMPING at kernel compile time.

(Also some MAC drivers do not defer to the PHY properly. Sometimes you can work around that by de-selecting the MAC's PTP function in the Kconfig if possible, but otherwise you need to patch the MAC driver.)

Quoted 2 lines

Do we need a property to indicate whether we wish to use the PHY or MAC PTP stamping, or something more elaborate?

To do this at run time would require quite some work, I expect.

Thanks,
Richard

#13 lore raw
Quoted 8 lines · Richard Cochran
Quoted 3 lines · Russell King - ARM Linux admin

How do we deal with this situation - from what I can see from the ethtool API, we have to make a choice about which to use. How do we make that choice?

Unfortunately the stack does not implement simultaneous MAC + PHY time stamping. If your board has both, then you make the choice to use the PHY by selecting NETWORK_PHY_TIMESTAMPING at kernel compile time.

Which is more or less what I said in my email. However, the important question about how to select between the two, which is really what I'm after, has not been addressed.

Quoted 3 lines

(Also some MAC drivers do not defer to the PHY properly. Sometimes you can work around that by de-selecting the MAC's PTP function in the Kconfig if possible, but otherwise you need to patch the MAC driver.)

... which really doesn't work if you have a board where only some network interfaces have a PHY with PTP support, but all have PTP support in the MAC.

If all MACs or the majority of MACs use a common PTP clock, it seems to me that you would want to use the MACs rather than the PHY, especially if the PHY doesn't offer as good a quality PTP clock as is available from the MAC.

Randomly patching the kernel is out of the question, for arm based systems we want one kernel that works correctly across as many platforms as possible, and Kconfig choices to set platform specific details are basically unacceptable, let alone patching the kernel to make those decisions.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#14 lore raw
Quoted 27 lines · Russell King - ARM Linux admin
Quoted 8 lines · Richard Cochran
Quoted 3 lines · Russell King - ARM Linux admin

How do we deal with this situation - from what I can see from the ethtool API, we have to make a choice about which to use. How do we make that choice?

Unfortunately the stack does not implement simultaneous MAC + PHY time stamping. If your board has both, then you make the choice to use the PHY by selecting NETWORK_PHY_TIMESTAMPING at kernel compile time.

Which is more or less what I said in my email. However, the important question about how to select between the two, which is really what I'm after, has not been addressed.

Quoted 3 lines

(Also some MAC drivers do not defer to the PHY properly. Sometimes you can work around that by de-selecting the MAC's PTP function in the Kconfig if possible, but otherwise you need to patch the MAC driver.)

... which really doesn't work if you have a board where only some network interfaces have a PHY with PTP support, but all have PTP support in the MAC.

If all MACs or the majority of MACs use a common PTP clock, it seems to me that you would want to use the MACs rather than the PHY, especially if the PHY doesn't offer as good a quality PTP clock as is available from the MAC.

Quoted 5 lines

Randomly patching the kernel is out of the question, for arm based systems we want one kernel that works correctly across as many platforms as possible, and Kconfig choices to set platform specific details are basically unacceptable, let alone patching the kernel to make those decisions.

The way PHY PTP support is handled is _really_ not nice.

If we have a phylib driver that supports timestamping, and a MAC driver that also supports timestamping, then what we end up with is very messed up.

1. The SIOCGHWTSTAMP/SIOCSHWTSTAMP calls are directed through ndo_do_ioctl(). The MAC driver gets first call on whether to intercept these or not. See dev_ifsioc().

2. The ethtool ETHTOOL_GET_TS_INFO call, however, is given to the PHY in preference to the MAC driver - there is no way for the MAC driver to gain preference. See __ethtool_get_ts_info().

So, if we have this situation (and yes, I do), then the SIOC*HWTSTAMP calls get implemented by the MAC driver, and takes effect at the MAC driver, while the ethtool ETHTOOL_GET_TS_INFO call returns results from the PHY driver.

That means the MAC driver's timestamping will be controllable from userspace, but userspace sees the abilities of the PHY driver's timestamping, and potentially directed to the wrong PTP clock instance.

What I see elsewhere in ethtool is that the MAC has the ability to override the phylib provided functionality - for example, __ethtool_get_sset_count(), __ethtool_get_strings(), and ethtool_get_phy_stats(). Would it be possible to do the same in
__ethtool_get_ts_info(), so at least a MAC driver can then decide
whether to propagate the ethtool request to phylib or not, just like it can do with the SIOC*HWTSTAMP ioctls? Essentially, reversing the order of:

        if (phy_has_tsinfo(phydev))
                return phy_ts_info(phydev, info);
        if (ops->get_ts_info)
                return ops->get_ts_info(dev, info);

?

Thanks.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#15 lore raw
Quoted 34 lines · Russell King - ARM Linux admin

The way PHY PTP support is handled is _really_ not nice.

If we have a phylib driver that supports timestamping, and a MAC driver that also supports timestamping, then what we end up with is very messed up.

1. The SIOCGHWTSTAMP/SIOCSHWTSTAMP calls are directed through ndo_do_ioctl(). The MAC driver gets first call on whether to intercept these or not. See dev_ifsioc().

2. The ethtool ETHTOOL_GET_TS_INFO call, however, is given to the PHY in preference to the MAC driver - there is no way for the MAC driver to gain preference. See __ethtool_get_ts_info().

So, if we have this situation (and yes, I do), then the SIOC*HWTSTAMP calls get implemented by the MAC driver, and takes effect at the MAC driver, while the ethtool ETHTOOL_GET_TS_INFO call returns results from the PHY driver.

That means the MAC driver's timestamping will be controllable from userspace, but userspace sees the abilities of the PHY driver's timestamping, and potentially directed to the wrong PTP clock instance.

What I see elsewhere in ethtool is that the MAC has the ability to override the phylib provided functionality - for example, __ethtool_get_sset_count(), __ethtool_get_strings(), and ethtool_get_phy_stats(). Would it be possible to do the same in
__ethtool_get_ts_info(), so at least a MAC driver can then decide
whether to propagate the ethtool request to phylib or not, just like it can do with the SIOC*HWTSTAMP ioctls? Essentially, reversing the order of:

Quoted 12 lines
        if (phy_has_tsinfo(phydev))
                return phy_ts_info(phydev, info);
        if (ops->get_ts_info)
                return ops->get_ts_info(dev, info);

?

Thanks.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

When we were discussing this in the other thread:
https://www.spinics.net/lists/netdev/msg669699.html
what I proposed was to introduce a NETIF_F_PHY_HWTSTAMP net_device flag that could get set when the driver is capable of doing the right thing with PHY timestamping. Then ethtool could look at that flag. Additionally, if we could make this flag configurable as a netdev feature with ethtool, then you could turn it off and even if you have an attached PTP PHY, you'll use the PHC of the MAC.
You're free to leave your thoughts in that thread.

Thanks,
-Vladimir

#16 lore raw
In reply to Russell King - ARM Linux admin
Quoted 9 lines · Russell King - ARM Linux admin

What I see elsewhere in ethtool is that the MAC has the ability to override the phylib provided functionality - for example, __ethtool_get_sset_count(), __ethtool_get_strings(), and ethtool_get_phy_stats(). Would it be possible to do the same in
__ethtool_get_ts_info(), so at least a MAC driver can then decide
whether to propagate the ethtool request to phylib or not, just like it can do with the SIOC*HWTSTAMP ioctls? Essentially, reversing the order of:

Quoted 6 lines
        if (phy_has_tsinfo(phydev))
                return phy_ts_info(phydev, info);
        if (ops->get_ts_info)
                return ops->get_ts_info(dev, info);

?

I don't see a simple solution. I think no matter what, the MAC drivers need work to allow PHY time stamping, and the great majority of users and driver authors are happy with MAC time stamping.

Thanks,
Richard

#17 lore raw
Quoted 15 lines · Richard Cochran
Quoted 14 lines · Russell King - ARM Linux admin

What I see elsewhere in ethtool is that the MAC has the ability to override the phylib provided functionality - for example, __ethtool_get_sset_count(), __ethtool_get_strings(), and ethtool_get_phy_stats(). Would it be possible to do the same in
__ethtool_get_ts_info(), so at least a MAC driver can then decide
whether to propagate the ethtool request to phylib or not, just like it can do with the SIOC*HWTSTAMP ioctls? Essentially, reversing the order of:

        if (phy_has_tsinfo(phydev))
                return phy_ts_info(phydev, info);
        if (ops->get_ts_info)
                return ops->get_ts_info(dev, info);
Quoted 5 lines
Quoted 1 line

?

I don't see a simple solution. I think no matter what, the MAC drivers need work to allow PHY time stamping, and the great majority of users and driver authors are happy with MAC time stamping.

What I ended up doing was:

        if (ops->get_ts_info) {
                ret = ops->get_ts_info(dev, info);
                if (ret != -EOPNOTSUPP)
                        return ret;
        }
        if (phy_has_tsinfo(phydev))
                return phy_ts_info(phydev, info);

...

which gives the MAC first refusal. If the MAC wishes to defer to phylib or the default, it can just return -EOPNOTSUPP.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#18 lore raw
Quoted 9 lines · Russell King - ARM Linux admin

What I ended up doing was:

        if (ops->get_ts_info) {
                ret = ops->get_ts_info(dev, info);
                if (ret != -EOPNOTSUPP)
                        return ret;
        }
        if (phy_has_tsinfo(phydev))
                return phy_ts_info(phydev, info);
Quoted 4 lines

...

which gives the MAC first refusal. If the MAC wishes to defer to phylib or the default, it can just return -EOPNOTSUPP.

I guess that makes sense. If someone designs a board that happens to have a PHY with unwanted time stamping fcunctionality, then at least the MAC time stamping function will work. If the designers really want PHY time stamping, then they are likely to have to patch the MAC driver in any case.

So I'm not against such a change. It would be important to keep the current "PHY-friendly" MAC drivers still friendly, and so they would need patching as part of the change.

Thanks,
Richard

#19 lore raw
Quoted 21 lines · Richard Cochran
Quoted 9 lines · Russell King - ARM Linux admin

What I ended up doing was:

        if (ops->get_ts_info) {
                ret = ops->get_ts_info(dev, info);
                if (ret != -EOPNOTSUPP)
                        return ret;
        }
        if (phy_has_tsinfo(phydev))
                return phy_ts_info(phydev, info);
Quoted 4 lines

...

which gives the MAC first refusal. If the MAC wishes to defer to phylib or the default, it can just return -EOPNOTSUPP.

I guess that makes sense. If someone designs a board that happens to have a PHY with unwanted time stamping fcunctionality, then at least the MAC time stamping function will work. If the designers really want PHY time stamping, then they are likely to have to patch the MAC driver in any case.

Quoted 3 lines

So I'm not against such a change. It would be important to keep the current "PHY-friendly" MAC drivers still friendly, and so they would need patching as part of the change.

That would only be necessary if they also provide the get_ts_info method.

So, I guess I need to find all drivers that refer to phylink or phylib functions, that also implement get_ts_info method and review them. I would expect that to be very small, since there's currently little point implementing PTP at both the PHY and the MAC for the reason I've raised earlier in this thread.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#20 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Richard Cochran
Quoted 9 lines · Richard Cochran
Quoted 3 lines · Russell King - ARM Linux admin

How do we deal with this situation - from what I can see from the ethtool API, we have to make a choice about which to use. How do we make that choice?

Unfortunately the stack does not implement simultaneous MAC + PHY time stamping. If your board has both, then you make the choice to use the PHY by selecting NETWORK_PHY_TIMESTAMPING at kernel compile time.

Quoted 8 lines

(Also some MAC drivers do not defer to the PHY properly. Sometimes you can work around that by de-selecting the MAC's PTP function in the Kconfig if possible, but otherwise you need to patch the MAC driver.)

Quoted 2 lines

Do we need a property to indicate whether we wish to use the PHY or MAC PTP stamping, or something more elaborate?

To do this at run time would require quite some work, I expect.

Okay, I'm falling into horrible multicast issues with DSA switches while trying to test.

Some of my platforms have IP_MULTICAST=y, others have IP_MULTICAST=n. This causes some to send IGMP messages when binding to the multicast address, others do not.

Those that do cause the DSA switch to add a static database entry causing all traffic for that multicast address to be only directed to the port(s) that the machine(s) with IP_MULTICAST=y kernels are connected to, depriving all IP_MULTICAST=n machines from seeing those packets.

Maybe, with modern networking technology, it's about time that the kernel configuration help recommended that kernels should be built with IP_MULTICAST=y ?

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#21 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 32 lines · Russell King - ARM Linux admin
Quoted 9 lines · Richard Cochran
Quoted 3 lines · Russell King - ARM Linux admin

How do we deal with this situation - from what I can see from the ethtool API, we have to make a choice about which to use. How do we make that choice?

Unfortunately the stack does not implement simultaneous MAC + PHY time stamping. If your board has both, then you make the choice to use the PHY by selecting NETWORK_PHY_TIMESTAMPING at kernel compile time.

Quoted 8 lines

(Also some MAC drivers do not defer to the PHY properly. Sometimes you can work around that by de-selecting the MAC's PTP function in the Kconfig if possible, but otherwise you need to patch the MAC driver.)

Quoted 2 lines

Do we need a property to indicate whether we wish to use the PHY or MAC PTP stamping, or something more elaborate?

To do this at run time would require quite some work, I expect.

Okay, I'm falling into horrible multicast issues with DSA switches while trying to test.

Some of my platforms have IP_MULTICAST=y, others have IP_MULTICAST=n. This causes some to send IGMP messages when binding to the multicast address, others do not.

Those that do cause the DSA switch to add a static database entry causing all traffic for that multicast address to be only directed to the port(s) that the machine(s) with IP_MULTICAST=y kernels are connected to, depriving all IP_MULTICAST=n machines from seeing those packets.

Quoted 3 lines

Maybe, with modern networking technology, it's about time that the kernel configuration help recommended that kernels should be built with IP_MULTICAST=y ?

Hmm, and even with IP_MULTICAST=y, I still can't get the "timestamping" program from the kernel sources to work.

On two different machines, I'm running:

# ./timestamping32 eno0 SOF_TIMESTAMPING_RX_HARDWARE SOF_TIMESTAMPING_RAW_HARDWARE

On one machine (arm32) this works - it can see the traffic it generates and receives from the other machine.

On the other machine (arm64), it sees _no_ traffic at all, but tcpdump on that machine can see the traffic being received:

12:36:40.002065 00:51:82:11:33:03 > 01:00:5e:00:01:82, ethertype IPv4 (0x0800), length 166: (tos 0x0, ttl 1, id 15045, offset 0, flags [DF], proto UDP (17), length 152)

    192.168.3.1.319 > 224.0.1.130.319: [bad udp cksum 0xa5c1 -> 0x7aaa!] UDP, length 124
12:36:41.105391 00:50:43:02:03:02 > 01:00:5e:00:01:82, ethertype IPv4 (0x0800),
length 166: (tos 0x0, ttl 1, id 9715, offset 0, flags [DF], proto UDP (17), length 152)
    192.168.3.2.319 > 224.0.1.130.319: [udp sum ok] UDP, length 124

The bad udp cksum is due to checksum offloadong on transmit - 3.1 is the arm64 host running that tcpdump.

When I look at /proc/net/snmp, I can see the IP InReceives incrementing but not the IP InDelivers nor UDP InDatagrams counter:

Ip: 2 64 1602 0 3 0 0 0 297 531 0 46 0 0 0 0 0 0 0
Udp: 572 0 0 480 0 0 0 4
Ip: 2 64 1603 0 3 0 0 0 297 531 0 46 0 0 0 0 0 0 0
Udp: 572 0 0 480 0 0 0 4

I don't have any firewall rules on the machine.

I've checked rp_filter on the appropriate interfaces... it's disabled on the ARM64 machine. The only other difference in interface config is accept_source_route, which is enabled on the ARM64 machine.

So, something in the IPv4 layer on ARM64 is silently discarding multicast UDP PTP packets, and I've no idea what... and I'm coming to the conclusion that this is all way too much effort and way too unreliable to be worth spending any more time trying to make work.

I'll send out what I have in the hope that maybe someone will find it useful and maybe able to complete the work. However, with these problems, it is totally unusable for me, and hence I can't test it.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#22 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 74 lines · Russell King - ARM Linux admin
Quoted 32 lines · Russell King - ARM Linux admin
Quoted 9 lines · Richard Cochran
Quoted 3 lines · Russell King - ARM Linux admin

How do we deal with this situation - from what I can see from the ethtool API, we have to make a choice about which to use. How do we make that choice?

Unfortunately the stack does not implement simultaneous MAC + PHY time stamping. If your board has both, then you make the choice to use the PHY by selecting NETWORK_PHY_TIMESTAMPING at kernel compile time.

Quoted 8 lines

(Also some MAC drivers do not defer to the PHY properly. Sometimes you can work around that by de-selecting the MAC's PTP function in the Kconfig if possible, but otherwise you need to patch the MAC driver.)

Quoted 2 lines

Do we need a property to indicate whether we wish to use the PHY or MAC PTP stamping, or something more elaborate?

To do this at run time would require quite some work, I expect.

Okay, I'm falling into horrible multicast issues with DSA switches while trying to test.

Some of my platforms have IP_MULTICAST=y, others have IP_MULTICAST=n. This causes some to send IGMP messages when binding to the multicast address, others do not.

Those that do cause the DSA switch to add a static database entry causing all traffic for that multicast address to be only directed to the port(s) that the machine(s) with IP_MULTICAST=y kernels are connected to, depriving all IP_MULTICAST=n machines from seeing those packets.

Quoted 3 lines

Maybe, with modern networking technology, it's about time that the kernel configuration help recommended that kernels should be built with IP_MULTICAST=y ?

Hmm, and even with IP_MULTICAST=y, I still can't get the "timestamping" program from the kernel sources to work.

On two different machines, I'm running:

# ./timestamping32 eno0 SOF_TIMESTAMPING_RX_HARDWARE SOF_TIMESTAMPING_RAW_HARDWARE

On one machine (arm32) this works - it can see the traffic it generates and receives from the other machine.

On the other machine (arm64), it sees _no_ traffic at all, but tcpdump on that machine can see the traffic being received:

12:36:40.002065 00:51:82:11:33:03 > 01:00:5e:00:01:82, ethertype IPv4 (0x0800), length 166: (tos 0x0, ttl 1, id 15045, offset 0, flags [DF], proto UDP (17), length 152)

    192.168.3.1.319 > 224.0.1.130.319: [bad udp cksum 0xa5c1 -> 0x7aaa!] UDP, length 124
12:36:41.105391 00:50:43:02:03:02 > 01:00:5e:00:01:82, ethertype IPv4 (0x0800),
length 166: (tos 0x0, ttl 1, id 9715, offset 0, flags [DF], proto UDP (17), length 152)
    192.168.3.2.319 > 224.0.1.130.319: [udp sum ok] UDP, length 124

The bad udp cksum is due to checksum offloadong on transmit - 3.1 is the arm64 host running that tcpdump.

When I look at /proc/net/snmp, I can see the IP InReceives incrementing but not the IP InDelivers nor UDP InDatagrams counter:

Ip: 2 64 1602 0 3 0 0 0 297 531 0 46 0 0 0 0 0 0 0
Udp: 572 0 0 480 0 0 0 4
Ip: 2 64 1603 0 3 0 0 0 297 531 0 46 0 0 0 0 0 0 0
Udp: 572 0 0 480 0 0 0 4

I don't have any firewall rules on the machine.

I've checked rp_filter on the appropriate interfaces... it's disabled on the ARM64 machine. The only other difference in interface config is accept_source_route, which is enabled on the ARM64 machine.

Quoted 8 lines

So, something in the IPv4 layer on ARM64 is silently discarding multicast UDP PTP packets, and I've no idea what... and I'm coming to the conclusion that this is all way too much effort and way too unreliable to be worth spending any more time trying to make work.

I'll send out what I have in the hope that maybe someone will find it useful and maybe able to complete the work. However, with these problems, it is totally unusable for me, and hence I can't test it.

To prove the point...

Two arm32 machines connected back to back through eno2, both running the basic test:

# ./timestamping eno2

see each other.

Try that same test between arm32 and arm64 connected in a similar manner, and the arm32 machine can see the UDP packets from the arm64 machine, but the arm64 machine discards the UDP packets from the arm32 machine in the IP layer running the exact same commands.

Both kernels and timestamping programs built from the same sources.

So, to summarise where I am:

- TAI (PHC) support:
* Works on Marvell 88E1512 PHY, with one event capture.
* Works on mvneta on Armada 388, no event capture implemented.
* Works on mvpp2.2 on Armada 8040. Event capture is possible, but interferes with the operation of the driver; no way to mask the external event trigger interference. Generating a "trigger" which can be set to a defined time and pulse with uses the same pin, and is also always seen as an external event, interfering with the operation of the hardware. There are two "clock" generation capabilities which one of which is described as "PPS", but these can't be aligned to a second boundary.

- PTP transmit timestamping:
* Works on Marvell 88E1512 PHY.
* Unimplemented on mvneta on Armada 388 - I am unable to get the PTP port registers to respond, and the documentation gives no hints; the PTP global registers and TAI registers respond however. No errata hint at this being a problem.
* Unimplemented on mvpp2.2 on Armada 8040 - I haven't got far enough to implement this.

- PTP receive timestamping:
* I think this works on Marvell 88E1512 PHY, but I'm now not sure.
* Implemented on mvpp2.2 on Armada 8040 - but unable to test (see above issues with the IP layer discarding what seem to be totally correct packets on ARM64.)
* Unimplemented on mvneta on Armada 388 - see above (PTP tx).

- DSA switches can effectively block multicast packets if only some kernels on the network have IP_MULTICAST=y and others have IP_MULTICAST=n, which adds a level of complexity and unreliability. (The answer is likely to always have IP_MULTICAST=y, and maybe the kernel should default to that - or in this day and age, the option ought to be removed.)

Hence why I'm at the point of giving up; I don't see that PTP will be of very limited benefit on my network with all these issues, and in any case, NTP has been "good enough" for the last 20+ years. Given that only a limited number of machines will be able to implement PTP support anyway, NTP will have to run along side it.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#23 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Hello RMK,

Quoted 5 lines

Hence why I'm at the point of giving up; I don't see that PTP will be of very limited benefit on my network with all these issues, and in any case, NTP has been "good enough" for the last 20+ years. Given that only a limited number of machines will be able to implement PTP support anyway, NTP will have to run along side it.

I see this patch has been abandoned.
I am testing it with a ZynqMP board (macb ethernet) and it seems to more or less work. It got tx timestamp timeout at initialization but after some times (~20 seconds) ptp4l manages to set it working. Also the IEEE 802.3 network PTP mode is not working, it constantly throw rx timestamp overrun errors.
I will aim at fixing these issues and adding support to interrupts. It would be good to have it accepted mainline. What do you think is missing for that? I see you faced issues with few Armada SOM and IP_MULTICAST, is it still the case?

I am new to the PTP and Ethernet APIs I will try to not speak nonsense but please correct me if I do.

Köry

#24 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 8 lines · Köry Maincent

Hello RMK,

Quoted 5 lines

Hence why I'm at the point of giving up; I don't see that PTP will be of very limited benefit on my network with all these issues, and in any case, NTP has been "good enough" for the last 20+ years. Given that only a limited number of machines will be able to implement PTP support anyway, NTP will have to run along side it.

Quoted 8 lines

I see this patch has been abandoned.
I am testing it with a ZynqMP board (macb ethernet) and it seems to more or less work. It got tx timestamp timeout at initialization but after some times (~20 seconds) ptp4l manages to set it working. Also the IEEE 802.3 network PTP mode is not working, it constantly throw rx timestamp overrun errors.
I will aim at fixing these issues and adding support to interrupts. It would be good to have it accepted mainline. What do you think is missing for that?

It isn't formally abandoned, but is permanently on-hold as merging Marvell PHY PTP support into mainline _will_ regress the superior PTP support on the Macchiatobin platform for the reasons outlined in:

https://lore.kernel.org/netdev/[email protected]/

Attempting to fix this problem was basically rejected by the PTP maintainer, and thus we're at a deadlock over the issue, and Marvell PHY PTP support can never be merged into mainline.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#25 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 14 lines · On Mon, 27 Feb 2023 15:20:05 +0000 "Russell King (Oracle)"…
Quoted 9 lines

I see this patch has been abandoned.
I am testing it with a ZynqMP board (macb ethernet) and it seems to more or less work. It got tx timestamp timeout at initialization but after some times (~20 seconds) ptp4l manages to set it working. Also the IEEE 802.3 network PTP mode is not working, it constantly throw rx timestamp overrun errors.
I will aim at fixing these issues and adding support to interrupts. It would be good to have it accepted mainline. What do you think is missing for that?

It isn't formally abandoned, but is permanently on-hold as merging Marvell PHY PTP support into mainline _will_ regress the superior PTP support on the Macchiatobin platform for the reasons outlined in:

Quoted 5 lines

https://lore.kernel.org/netdev/[email protected]/

Attempting to fix this problem was basically rejected by the PTP maintainer, and thus we're at a deadlock over the issue, and Marvell PHY PTP support can never be merged into mainline.

As I understand, if the PHY support PTP, it is prioritize to the PTP of the MAC. As quote in the mail thread it seems there was discussion in netdev about moving phy_has_hwtstamp to core and allowing ethtool to choose which one to use. I don't know if the decision have been made about it since, but it seems nothing has been sent to mainline. Meanwhile, why do we not move forward on this patch with the current PTP behavior and updates it when new core PTP change will be sent mailine?

Regards,
Köry

#26 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 23 lines · Köry Maincent
Quoted 14 lines · On Mon, 27 Feb 2023 15:20:05 +0000 "Russell King (Oracle)"…
Quoted 9 lines

I see this patch has been abandoned.
I am testing it with a ZynqMP board (macb ethernet) and it seems to more or less work. It got tx timestamp timeout at initialization but after some times (~20 seconds) ptp4l manages to set it working. Also the IEEE 802.3 network PTP mode is not working, it constantly throw rx timestamp overrun errors.
I will aim at fixing these issues and adding support to interrupts. It would be good to have it accepted mainline. What do you think is missing for that?

It isn't formally abandoned, but is permanently on-hold as merging Marvell PHY PTP support into mainline _will_ regress the superior PTP support on the Macchiatobin platform for the reasons outlined in:

Quoted 5 lines

https://lore.kernel.org/netdev/[email protected]/

Attempting to fix this problem was basically rejected by the PTP maintainer, and thus we're at a deadlock over the issue, and Marvell PHY PTP support can never be merged into mainline.

Quoted 7 lines

As I understand, if the PHY support PTP, it is prioritize to the PTP of the MAC. As quote in the mail thread it seems there was discussion in netdev about moving phy_has_hwtstamp to core and allowing ethtool to choose which one to use. I don't know if the decision have been made about it since, but it seems nothing has been sent to mainline. Meanwhile, why do we not move forward on this patch with the current PTP behavior and updates it when new core PTP change will be sent mailine?

Clearly, we have an issue with communication. Let me repeat:

Merging support for Marvell PHY PTP *will* *regress* the Macchiatobin platform, making PTP *unusable* because *some* API calls will go to the *PHY* PTP instance while *other* API calls go to the *MAC* PTP instance.

Since merging Marvell PHY PTP support will cause a regression, the only way to fix that regression at the moment is by reverting the merge of Marvell PHY PTP support as there is no acceptable solution to the above problem - I've attempted to fix it and the patch was rejected.

Sorry, but no, mainline does not get Marvell PHY PTP support before the generic code is fixed for this blocking issue.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#27 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Russell King (Oracle)
Quoted 3 lines · Russell King (Oracle)

Attempting to fix this problem was basically rejected by the PTP maintainer, and thus we're at a deadlock over the issue, and Marvell PHY PTP support can never be merged into mainline.

FWIW, here was my attempt to solve the issue by making the PHY/MAC layer selectable at run time, while preserving PHY as the default.

https://lore.kernel.org/netdev/[email protected]/

There was a fair amount of discussion, and it seemed to me that everyone wanted a pony.

If anyone wants to take up that series, or present a more creative solution, then by all means, please go ahead. I still have a working board with the PHYTER, and so I would be happy to help validate the new feature.

Thanks,
Richard

#28 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 10 lines · Richard Cochran
Quoted 3 lines · Russell King (Oracle)

Attempting to fix this problem was basically rejected by the PTP maintainer, and thus we're at a deadlock over the issue, and Marvell PHY PTP support can never be merged into mainline.

FWIW, here was my attempt to solve the issue by making the PHY/MAC layer selectable at run time, while preserving PHY as the default.

https://lore.kernel.org/netdev/[email protected]/

Hi Richard,

Looking at that link, I'm only seeing that message, with none of the patches nor the discussion. Digging back in my mailbox, I find that the patches weren't threaded to the cover message, which makes it quite difficult to go back and review the discussion.

Patch 1 (no comments)
https://lore.kernel.org/netdev/[email protected]
Patch 2 (one comment from me suggesting moving a variable)
https://lore.kernel.org/netdev/[email protected]
Patch 3 (lots of comments)
https://lore.kernel.org/netdev/[email protected]
Patch 4 (no comments)
https://lore.kernel.org/netdev/[email protected]

Looking back briefly at the discussion on patch 3, was the reason this approach died due to the request to have something more flexible, supporting multiple hardware timestamps per packet?

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#29 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 4 lines · Russell King (Oracle)

Looking at that link, I'm only seeing that message, with none of the patches nor the discussion. Digging back in my mailbox, I find that the patches weren't threaded to the cover message, which makes it quite difficult to go back and review the discussion.

Sorry about that. By accident I omitted --thread=shallow that time.

Quoted 3 lines

Looking back briefly at the discussion on patch 3, was the reason this approach died due to the request to have something more flexible, supporting multiple hardware timestamps per packet?

I still think the approach will work, but I guess I got distracted with other stuff and forgot about it.

The "multiple hardware timestamps per packet" is a nice idea, but it would require a new user API, and so selectable MAC/PHY on the existing API is still needed.

Thanks,
Richard

#30 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 13 lines · Richard Cochran
Quoted 4 lines · Russell King (Oracle)

Looking at that link, I'm only seeing that message, with none of the patches nor the discussion. Digging back in my mailbox, I find that the patches weren't threaded to the cover message, which makes it quite difficult to go back and review the discussion.

Sorry about that. By accident I omitted --thread=shallow that time.

Quoted 3 lines

Looking back briefly at the discussion on patch 3, was the reason this approach died due to the request to have something more flexible, supporting multiple hardware timestamps per packet?

Quoted 6 lines

I still think the approach will work, but I guess I got distracted with other stuff and forgot about it.

The "multiple hardware timestamps per packet" is a nice idea, but it would require a new user API, and so selectable MAC/PHY on the existing API is still needed.

I agree - even when we have support for multiple hardware timestamps, we still need the existing API to work in a sensible way, and we need a way to choose which hardware timestamp we want the existing API to report.

So yes, it's a nice idea to support multiple hardware timestamps, but I think that's an entirely separate problem to solving the current issue, which is a blocking issue to adding support for PTP on some platforms.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#31 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 4 lines · On Tue, 28 Feb 2023 12:07:09 +0000 "Russell King (Oracle)"…

So yes, it's a nice idea to support multiple hardware timestamps, but I think that's an entirely separate problem to solving the current issue, which is a blocking issue to adding support for PTP on some platforms.

Alright, Richard can I continue your work on it and send new revisions of your patch series or do you prefer to continue on your own?
Also your series rise the question of which timestamping should be the default, MAC or PHY, without breaking any past or future compatibilities. There is question of using Kconfig or devicetree but each of them seems to have drawbacks:
https://lore.kernel.org/netdev/[email protected]/

Do you or Russell have any new thought about it?

Regards,
Köry

#32 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 15 lines · Köry Maincent
Quoted 4 lines · On Tue, 28 Feb 2023 12:07:09 +0000 "Russell King (Oracle)"…

So yes, it's a nice idea to support multiple hardware timestamps, but I think that's an entirely separate problem to solving the current issue, which is a blocking issue to adding support for PTP on some platforms.

Alright, Richard can I continue your work on it and send new revisions of your patch series or do you prefer to continue on your own?
Also your series rise the question of which timestamping should be the default, MAC or PHY, without breaking any past or future compatibilities. There is question of using Kconfig or devicetree but each of them seems to have drawbacks:
https://lore.kernel.org/netdev/[email protected]/

Quoted 1 line

Do you or Russell have any new thought about it?

The only thought I have is that maybe a MAC driver should be able to override the default, then at least we have a way to deal with this on a case by case basis. However, that's just pulling an idea out of the air.

I think what might be useful as a first step is to go through the various networking devices to work out which support PTP today, and tabulate the result. There shouldn't be any cases where we have both the MAC and PHY having PTP support (for the API reasons I've already stated) but if there are, that needs to be highlighted.

Then we can see what the default should be, and then which MAC drivers would need to override the default.

It would probably be a good idea to document that in the kernel's Documentation subdirectory so when e.g. a PHY driver gains PTP support, we have some idea which MAC drivers may be impacted.

Does that sound sensible?

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#33 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 13 lines · On Tue, 28 Feb 2023 13:36:36 +0000 "Russell King (Oracle)"…
Quoted 7 lines

Also your series rise the question of which timestamping should be the default, MAC or PHY, without breaking any past or future compatibilities. There is question of using Kconfig or devicetree but each of them seems to have drawbacks:
https://lore.kernel.org/netdev/[email protected]/

Do you or Russell have any new thought about it?

The only thought I have is that maybe a MAC driver should be able to override the default, then at least we have a way to deal with this on a case by case basis. However, that's just pulling an idea out of the air.

Quoted 5 lines

I think what might be useful as a first step is to go through the various networking devices to work out which support PTP today, and tabulate the result. There shouldn't be any cases where we have both the MAC and PHY having PTP support (for the API reasons I've already stated) but if there are, that needs to be highlighted.

The only defconfig on that case present on mainline is socfpga_defconfig (MICREL_PHY and STMMAC_ETH). In fact PHY timestamping is really not spread. There is only one defconfig which support PTP on both MAC and PHY, therefore setting the PTP on MAC by default seems the best choice. It won't change the default behavior when adding new PHY PTP support. Does adding documentation about socfpga_defconfig is sufficient or should we need to add something specific to change the default PTP on this case.

Regards,
Köry

#34 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Köry Maincent
Quoted 9 lines · Köry Maincent
Quoted 4 lines · On Tue, 28 Feb 2023 12:07:09 +0000 "Russell King (Oracle)"…

So yes, it's a nice idea to support multiple hardware timestamps, but I think that's an entirely separate problem to solving the current issue, which is a blocking issue to adding support for PTP on some platforms.

Alright, Richard can I continue your work on it and send new revisions of your patch series or do you prefer to continue on your own?

If you can work this, please do. I can help review and test.

Quoted 7 lines

Also your series rise the question of which timestamping should be the default, MAC or PHY, without breaking any past or future compatibilities. There is question of using Kconfig or devicetree but each of them seems to have drawbacks:
https://lore.kernel.org/netdev/[email protected]/

Do you or Russell have any new thought about it?

The overall default must be PHY, because that is the legacy condition. The options to change the default are:

1. device tree: Bad because the default is a configuration and does not describe the hardware.

2. Kconfig: Override PHY default at compile time.

3. Module Param: Configure default on kernel command line.

4. Letting drivers override PHY at run time.

5. other?

It would be possible to support both 2 and 3, with command line having the last word.

I don't like #4 because it would cleaner if every time stamping driver would simply implement the internal APIs, without having special hooks for various boards.

Thanks,
Richard

maintainer
#35 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 5 lines

The overall default must be PHY, because that is the legacy condition. The options to change the default are:

1. device tree: Bad because the default is a configuration and does not describe the hardware.

There are also lots of systems which don't use DT, e.g. ACPI, or are just plain PCI or USB devices which don't have any configuration at all.

Quoted 1 line

2. Kconfig: Override PHY default at compile time.

Per MAC/PHY combination? That does not scale.

Quoted 1 line

3. Module Param: Configure default on kernel command line.

Module params are consider bad in the networking stack. DaveM always NACKs them.

Quoted 1 line

4. Letting drivers override PHY at run time.

This is what Russell suggested.

Quoted 8 lines

5. other?

It would be possible to support both 2 and 3, with command line having the last word.

I don't like #4 because it would cleaner if every time stamping driver would simply implement the internal APIs, without having special hooks for various boards.

I agree that stamping drivers should implement the API. All i think Russell is suggesting is that MAC drivers have an API call they can make on the PTP core to tell it to direct all calls to the MAC implementation of the time stamper, not the PHY implementation of the time stamper. This might need some changes to the internal APIs, such that all kernel API calls should go through the PTP core, and the PTP core then directs them by default to the PHY, but on request sends it to the MAC stamper.

Andrew

#36 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 3 lines · Andrew Lunn
Quoted 1 line

2. Kconfig: Override PHY default at compile time.

Per MAC/PHY combination? That does not scale.

No, I meant a single, global default that overrides the implicit default of using the PHY.

Quoted 4 lines
Quoted 1 line

3. Module Param: Configure default on kernel command line.

Module params are consider bad in the networking stack. DaveM always NACKs them.

It could be a regular command line option?

Thanks,
Richard

#37 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Richard Cochran
Quoted 19 lines · Richard Cochran
Quoted 9 lines · Köry Maincent
Quoted 4 lines · On Tue, 28 Feb 2023 12:07:09 +0000 "Russell King (Oracle)"…

So yes, it's a nice idea to support multiple hardware timestamps, but I think that's an entirely separate problem to solving the current issue, which is a blocking issue to adding support for PTP on some platforms.

Alright, Richard can I continue your work on it and send new revisions of your patch series or do you prefer to continue on your own?

If you can work this, please do. I can help review and test.

Quoted 6 lines

Also your series rise the question of which timestamping should be the default, MAC or PHY, without breaking any past or future compatibilities. There is question of using Kconfig or devicetree but each of them seems to have drawbacks:
https://lore.kernel.org/netdev/[email protected]/

Quoted 7 lines
Quoted 1 line

Do you or Russell have any new thought about it?

The overall default must be PHY, because that is the legacy condition. The options to change the default are:

1. device tree: Bad because the default is a configuration and does not describe the hardware.

I'm quite sure the DT maintainers will frown upon that.

Quoted 1 line

2. Kconfig: Override PHY default at compile time.

That doesn't work for Arm (32 or 64 bit) kernels as we want a single kernel image that will work correctly on any platform, so as soon as we have two platforms that needs the Kconfig set differently, this becomes a problem.

Quoted 1 line

3. Module Param: Configure default on kernel command line.

I understand module parameters are frowned upon by netdev.

Quoted 1 line

4. Letting drivers override PHY at run time.

I think this is the only sensible solution - we know for example that mvpp2 will prefer its PTP implementation as it is (a) higher resolution and (b) has more flexibility than what can be provided by the Marvell PHYs that it is often used with.

Quoted 1 line

5. other?

Another possible solution to this would be to introduce a rating for each PTP clock in a similar way that we do for the kernel's clocksources, and the one with the highest rating becomes the default.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#38 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 6 lines
Quoted 1 line

4. Letting drivers override PHY at run time.

I think this is the only sensible solution - we know for example that mvpp2 will prefer its PTP implementation as it is (a) higher resolution and (b) has more flexibility than what can be provided by the Marvell PHYs that it is often used with.

Please also consider that there might be one switch with a shared PHC and multiple PHYs, each with its own PHC. In this case, it is a property of the board wether PHY timestamping actually works, because it will need some kind of synchronization between all the PHYs. Whereas the MAC timestamping just works. Or we dodge that problem now and these kind of drivers might not use PHY timestamping. But AFAIK microchip is working on PHY timestamping for their quad PHYs which are used together with a LAN9668 switch.

-michael

#39 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 9 lines · Michael Walle
Quoted 6 lines
Quoted 1 line

4. Letting drivers override PHY at run time.

I think this is the only sensible solution - we know for example that mvpp2 will prefer its PTP implementation as it is (a) higher resolution and (b) has more flexibility than what can be provided by the Marvell PHYs that it is often used with.

Please also consider that there might be one switch with a shared PHC and multiple PHYs, each with its own PHC.

Doesn't the PTP API already allow that? The PHC is a separate API from the network hardware timestamping - and the netdev/PHY is required to implement the ethtool get_ts_info API that provides userspace with the index to the PHC associated with the interface.

Quoted 3 lines

In this case, it is a
property of the board wether PHY timestamping actually works, because it will need some kind of synchronization between all the PHYs.

How is this any different from e.g. a platform where there are multiple network interfaces each with their own independent PHC such as Macchiatobin, where there are two CP110 dies, each with their own group of three ethernet adapters, and each die has its own PHC shared between the three ethernet adapters?

Hardware synchronisation between the two PHCs isn't possible, but they might tick at the same rate (it's something that hasn't been checked.) However, the hardware signals aren't that helpful because there's no way to make e.g. the rising edge always be at the start of a second. So the synchronisation has to be done in software.

I don't think PHCs need to be synchronised in hardware to "actually work". Take an example of a PC with two network cards, both having their own independent PHC.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#40 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Am 2023-02-28 17:58, schrieb Russell King (Oracle):

Quoted 8 lines · Michael Walle
Quoted 7 lines
Quoted 6 lines
Quoted 1 line

4. Letting drivers override PHY at run time.

I think this is the only sensible solution - we know for example that mvpp2 will prefer its PTP implementation as it is (a) higher resolution and (b) has more flexibility than what can be provided by the Marvell PHYs that it is often used with.

Quoted 7 lines
Quoted 2 lines

Please also consider that there might be one switch with a shared PHC and multiple PHYs, each with its own PHC.

Doesn't the PTP API already allow that? The PHC is a separate API from the network hardware timestamping - and the netdev/PHY is required to implement the ethtool get_ts_info API that provides userspace with the index to the PHC associated with the interface.

Yes, but the source for the timestamp is the PHC. If the PHCs are not synchronized, the timestamps won't be either. With a shared PHC, the synchronization is already a given.

Quoted 16 lines
Quoted 3 lines

In this case, it is a
property of the board wether PHY timestamping actually works, because it will need some kind of synchronization between all the PHYs.

How is this any different from e.g. a platform where there are multiple network interfaces each with their own independent PHC such as Macchiatobin, where there are two CP110 dies, each with their own group of three ethernet adapters, and each die has its own PHC shared between the three ethernet adapters?

Hardware synchronisation between the two PHCs isn't possible, but they might tick at the same rate (it's something that hasn't been checked.) However, the hardware signals aren't that helpful because there's no way to make e.g. the rising edge always be at the start of a second. So the synchronisation has to be done in software.

Quoted 3 lines

I don't think PHCs need to be synchronised in hardware to "actually work". Take an example of a PC with two network cards, both having their own independent PHC.

That might be true if you just want to use PTP as a time sync protocol, but keep in mind that there is also the time aware scheduler which uses the PHC as its time source, too. If you want to use PHY timestamping in this case, the PHCs needs to be synchronized. Honestly, I'm not really sure, how that is supposed to work.
All I'm trying to say, is there might also be some board constraints so the MAC driver might not always be telling what is best, PHY or MAC.

-michael

#41 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 2 lines · Michael Walle

All I'm trying to say, is there might also be some board constraints so the MAC driver might not always be telling what is best, PHY or MAC.

+1

I can't see how the MAC, in general, could possibly determine whether it should be preferred.

Thanks,
Richard

#42 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Russell King (Oracle)
Quoted 3 lines · Russell King (Oracle)

Another possible solution to this would be to introduce a rating for each PTP clock in a similar way that we do for the kernel's clocksources, and the one with the highest rating becomes the default.

Hm, that is an idea.

It would only need a command line override in case the user disagrees with the kernel's hard coded rating.

Thanks,
Richard

maintainer
#43 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Russell King (Oracle)
Quoted 5 lines · On Tue, 28 Feb 2023 16:27:10 +0000 Russell King (Oracle) wrote:
Quoted 1 line

5. other?

Another possible solution to this would be to introduce a rating for each PTP clock in a similar way that we do for the kernel's clocksources, and the one with the highest rating becomes the default.

Why not ethtool? Sorry if I'm missing something obvious..

#44 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 8 lines · Jakub Kicinski
Quoted 5 lines · On Tue, 28 Feb 2023 16:27:10 +0000 Russell King (Oracle) wrote:
Quoted 1 line

5. other?

Another possible solution to this would be to introduce a rating for each PTP clock in a similar way that we do for the kernel's clocksources, and the one with the highest rating becomes the default.

Why not ethtool? Sorry if I'm missing something obvious..

If we make the "default" fixed, such as "we always default to PHY" then merge Marvell PHY PTP support, then on Macchiatobin, we will end up switching from the current mvpp2-based PTP support to using the inferior PHY PTP support, resulting in a loss of precision and accuracy. That's a regression.

So, while we generally want PHY PTP support to be the default, there are legitimate reasons for wanting in some circumstances for the MAC to be the default.

The problem with an ethtool control is it's an extra step that the user has to do to restore the accuracy that they had under today's kernels, and I don't think that's something that they should be involved in doing.

Providing controls to userspace is all very well _if_ there is a way for users to make sensible decisions - which means giving them information such as "on this hardware, MAC PTP is preferable because it has better accuracy and precision, but on some other hardware, PHY PTP is preferable" and such a document would get very very long.

IMHO, it's better if the kernel automatically selects a sensible default _and_ gives the user the ability to override it e.g. via ethtool.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
maintainer
#45 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 3 lines · On Tue, 28 Feb 2023 22:40:05 +0000 Russell King (Oracle) wrote:

IMHO, it's better if the kernel automatically selects a sensible default _and_ gives the user the ability to override it e.g. via ethtool.

Oh, I see, makes sense.

#46 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 6 lines · On Tue, 28 Feb 2023 14:59:11 -0800 Jakub Kicinski <[email protected]> wrote:
Quoted 3 lines · On Tue, 28 Feb 2023 22:40:05 +0000 Russell King (Oracle) wrote:

IMHO, it's better if the kernel automatically selects a sensible default _and_ gives the user the ability to override it e.g. via ethtool.

Oh, I see, makes sense.

I suppose the idea of Russell to rate each PTP clocks may be the best one, as all others solutions have drawbacks. Does using the PTP clock period value (in picoseconds) is enough to decide which PTP is the best one? It is hardware specific therefore it is legitimate to be set by the MAC and PHY drivers.

Regards,

#47 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 4 lines · Köry Maincent

I suppose the idea of Russell to rate each PTP clocks may be the best one, as all others solutions have drawbacks. Does using the PTP clock period value (in picoseconds) is enough to decide which PTP is the best one? It is hardware specific therefore it is legitimate to be set by the MAC and PHY drivers.

It is not that simple. In fact, I've never seen an objective comparision of different HW. The vendors surely have no reason to conduct such a study. Also, the data sheets never make any claim about synchronization quality.

Thanks,
Richard

#48 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 10 lines · Richard Cochran
Quoted 4 lines · Köry Maincent

I suppose the idea of Russell to rate each PTP clocks may be the best one, as all others solutions have drawbacks. Does using the PTP clock period value (in picoseconds) is enough to decide which PTP is the best one? It is hardware specific therefore it is legitimate to be set by the MAC and PHY drivers.

It is not that simple. In fact, I've never seen an objective comparision of different HW. The vendors surely have no reason to conduct such a study. Also, the data sheets never make any claim about synchronization quality.

mvpp2 (MAC) hardware has a counter in hardware that we can read the timestamps from, and it supports being fine-tuned and stepped in hardware. So the hardware gives us the timestamps with no shenanigans. The hardware timestamp is incremented by a programmable amount (including a 32-bit fractional part) every 3ns.

The mvpp2 hardware captures the timestamp when receiving packets, and places the timestamp in the buffer descriptors - there is no filtering and this timestamping essentially comes for free from the software perspective. There is no requirement to read any registers before the next packet that needs to be timestamped, and the timestamps can not be confused which packet they are intended for. Since these timestamps are in the buffer descriptors, they are available as soon as the packet is ready to be passed to the networking layers.

Timestamps can be inserted into transmitted packets and the checksums updated by the hardware. The hardware has capability to do a number of operations on the packets when inserting the timestamp, although I never wrote software support for this (I wanted some way to positively test these but I don't think I had a way to do that.) Multiple transmit packets can be queued for timestamping and the hardware will cope.

Hardware signals are supported. The hardware has event capture, capabilities, which snapshots the counter, and this should give high accuracy. However, for generation of events, e.g. PPS, I have observed that there is no way to ensure that the PPS signal is aligned to the start of a second. That said, the signal is subject to the fine adjustments of the hardware counter - so increasing the hardware counter's increment correctly shortens the PPS signal period.

All accesses to the hardware are fast, being MMIO, which gives a very stable reading when the hardware clock is adjusted by ptp4l.

The hardware timestamp counter is shared across all three ethernet interfaces of the CP110 die, and there can be more than one CP110 die in a SoC.

Marvell PHY hardware is qutie different. The counter is updated every 8ns, and merely increments by one each time. There is no fractional adjustment, meaning that there is no fine adjustment of the hardware. Fine adjustment is performed by using the kernel's timecounter infrastructure.

The counter is accessed over MDIO which appears to introduce a lot of variability in latency, which transfers over to read accesses to the counter, and thus the read timestamp. This makes it harder for ptp4l to synchronise the counter, and from my testing, introduces a lot of noise.

The PHY can't modify the packet in any way, but merely captures the counter when the packet is transmitted. The PHY needs to be programmed with the byte offset from the start of the packet to the ethernet protocol, and the ethernet protocol to the IP header - suggesting that the PTP hardware needs to know whether packets will or will not be incorporating a VLAN header (this is explicitly mentioned in e.g. the 88e151x manual.)

The PHY's filtering applies to both the transmit and receive paths, and as there is only one set of capture registers for a transmit packet, this may become a problem, especially if we are not using interrupts for the PHY. Without interrupts, at best we poll the PHY every 2ms (but could be longer due to MDIO access times.)

For receive, the PHY implement two capture register sets. The PHY needs to parse the packet (using the offsets into the packet above) in order to retrieve the MessageID field which is then used to work out which receive capture register to write the packet's counter value to. This means if one receives several PTP packets in quick succession, there is a chance to lose timestamps.

The situation with hardware signals will be similar to PP2, in that event capture takes a snapshot of the hardware counter, which can then be translated. However, as the hardware counter does not get adjusted, signal generation is tied to the unadjusted rate at which the hardware ticks at, which means that asking for a PPS, the generated signal will drift as the PHY 125MHz clock drifts.

In conclusion, not only does PP2 have a 3ns tick vs 8ns for PHY, PP2 can more reliably capture the timestamps, reading the hardware counter value has less noise, and thus can be synchronised by ptp4l with far less random variance than the PHY implementation.

Therefore, I believe that the Marvell PHY PTP implementation is all round inferior to that found in the Marvell PP2 MAC, and hence why I believe that the PP2 MAC implementation should be used by default over the PHY.

(In essence, because of all the noise when trying the Marvell PHY with ptp4l, I came to the conlusion that NTP was a far better solution to time synchronisation between machines than PTP would ever be due to the nose induced by MDIO access. However, I should also state that I basically gave up with PTP in the end because hardware support is overall poor, and NTP just works - and I'd still have to run NTP for the machines that have no PTP capabilities. PTP probably only makes sense if one has a nice expensive grand master PTP clock on ones network, and all the machines one wants to synchronise have decent PTP implementations.)

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
maintainer
#49 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 10 lines · On Thu, 2 Mar 2023 11:49:26 +0000 Russell King (Oracle) wrote:

(In essence, because of all the noise when trying the Marvell PHY with ptp4l, I came to the conlusion that NTP was a far better solution to time synchronisation between machines than PTP would ever be due to the nose induced by MDIO access. However, I should also state that I basically gave up with PTP in the end because hardware support is overall poor, and NTP just works - and I'd still have to run NTP for the machines that have no PTP capabilities. PTP probably only makes sense if one has a nice expensive grand master PTP clock on ones network, and all the machines one wants to synchronise have decent PTP implementations.)

Don't wanna waste too much of your time with the questions since I haven't done much research but - wouldn't MAC timestamp be a better choice more often (as long as it's a real, to-spec PTP stamp)? Are we picking PHY for historical reasons?

Not that flipping the default would address the problem of regressing some setups..

#50 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 12 lines · On Thu, 2 Mar 2023 08:49:32 -0800 Jakub Kicinski <[email protected]> wrote:
Quoted 10 lines · On Thu, 2 Mar 2023 11:49:26 +0000 Russell King (Oracle) wrote:

(In essence, because of all the noise when trying the Marvell PHY with ptp4l, I came to the conlusion that NTP was a far better solution to time synchronisation between machines than PTP would ever be due to the nose induced by MDIO access. However, I should also state that I basically gave up with PTP in the end because hardware support is overall poor, and NTP just works - and I'd still have to run NTP for the machines that have no PTP capabilities. PTP probably only makes sense if one has a nice expensive grand master PTP clock on ones network, and all the machines one wants to synchronise have decent PTP implementations.)

Quoted 7 lines

Don't wanna waste too much of your time with the questions since I haven't done much research but - wouldn't MAC timestamp be a better choice more often (as long as it's a real, to-spec PTP stamp)? Are we picking PHY for historical reasons?

Not that flipping the default would address the problem of regressing some setups..

I have measured it with the Marvell PHY and MACB MAC but it is the contrary on my side:
https://lkml.kernel.org/netdev/20230302113752.057a3213@kmaincent-XPS-13-7390/
Also PHY default seems more logical as it is nearer to the physical link, but still I am interesting by the answer as I am not a PTP expert. Is really PTP MAC often more precise than PTP PHY?

Regards,
Köry

maintainer
#51 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 6 lines · On Thu, 2 Mar 2023 18:06:16 +0100 Köry Maincent wrote:

I have measured it with the Marvell PHY and MACB MAC but it is the contrary on my side:
https://lkml.kernel.org/netdev/20230302113752.057a3213@kmaincent-XPS-13-7390/
Also PHY default seems more logical as it is nearer to the physical link, but still I am interesting by the answer as I am not a PTP expert. Is really PTP MAC often more precise than PTP PHY?

Do you happen to have a datasheet for MACB? The time stamping capability is suspiciously saved in a variable called hw_dma_cap which may indicate it's a DMA time stamp not a true PTP stamp.

Quite a few NICs/MACs support DMA time stamps because it's far easier to take the stamp where the descriptor writing logic is (DMA block) than take it at the end of the MAC and haul it all the way thru the pipeline back to the DMA block.

#52 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 8 lines · On Thu, 2 Mar 2023 09:23:20 -0800 Jakub Kicinski <[email protected]> wrote:

Do you happen to have a datasheet for MACB? The time stamping capability is suspiciously saved in a variable called hw_dma_cap which may indicate it's a DMA time stamp not a true PTP stamp.

Quite a few NICs/MACs support DMA time stamps because it's far easier to take the stamp where the descriptor writing logic is (DMA block) than take it at the end of the MAC and haul it all the way thru the pipeline back to the DMA block.

I don't have the datasheet but indeed you seem to have right. From the support commit message:

Quoted 1 line

Time stamps are obtained from the dma buffer descriptors

I suppose it is less precise as using true PTP stamp as it is an hardware block further. Is that right?

Regards,
Köry

maintainer
#53 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 4 lines · On Fri, 3 Mar 2023 14:12:47 +0100 Köry Maincent wrote:
Quoted 4 lines

Do you happen to have a datasheet for MACB? The time stamping capability is suspiciously saved in a variable called hw_dma_cap which may indicate it's a DMA time stamp not a true PTP stamp.

Quoted 8 lines
Quoted 4 lines

Quite a few NICs/MACs support DMA time stamps because it's far easier to take the stamp where the descriptor writing logic is (DMA block) than take it at the end of the MAC and haul it all the way thru the pipeline back to the DMA block.

I don't have the datasheet but indeed you seem to have right. From the support commit message:

Quoted 1 line

Time stamps are obtained from the dma buffer descriptors

That's just saying that the time stamp is communicated via a descriptor rather than a register, so not a strong proof of where it's taken.

Quoted 2 lines

I suppose it is less precise as using true PTP stamp as it is an hardware block further. Is that right?

#54 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Jakub Kicinski
Quoted 12 lines · Jakub Kicinski
Quoted 10 lines · On Thu, 2 Mar 2023 11:49:26 +0000 Russell King (Oracle) wrote:

(In essence, because of all the noise when trying the Marvell PHY with ptp4l, I came to the conlusion that NTP was a far better solution to time synchronisation between machines than PTP would ever be due to the nose induced by MDIO access. However, I should also state that I basically gave up with PTP in the end because hardware support is overall poor, and NTP just works - and I'd still have to run NTP for the machines that have no PTP capabilities. PTP probably only makes sense if one has a nice expensive grand master PTP clock on ones network, and all the machines one wants to synchronise have decent PTP implementations.)

Quoted 7 lines

Don't wanna waste too much of your time with the questions since I haven't done much research but - wouldn't MAC timestamp be a better choice more often (as long as it's a real, to-spec PTP stamp)? Are we picking PHY for historical reasons?

Not that flipping the default would address the problem of regressing some setups..

There is the argument in PTP that having the PHY do the timestamping results in the timestamps being "closer to the wire" meaning that they will not be subject to any delays between the packet leaving the MAC and being transmitted on the wire. Some PHYs have FIFOs or other buffering which introduces a delay which PTP doesn't desire.

TI has a blog on this:

https://e2e.ti.com/blogs_/b/analogwire/posts/how-to-implement-ieee-1588-time-stamping-in-an-ethernet-transceiver

However, what is failed to be mentioned there is that yes, doing PTP at the PHY means one can accurately trigger a capture of the timestamp from the PHC when the SFD is detected on the media. That is a great advantage, but is really only half the story.

If the PHC (hardware counter) in the PHY can't be accurately synchronised, for example, it has a 1.5ppm second-to-second variability, then the resulting timestamps will also have that same variability. Another way to look at it is they will appear to have 1.5ppm of noise. If this noise is random, then it becomes difficult to filter that noise out, and results in jitter.

However, timestamping at the MAC may have only 40ppb of variability, but have a resulting delay through the PHY. The delay through the PHY will likely be constant, so the timestamps gathered at the MAC will have a constant error but have much less noise.

Things change if one can use hardware signals to synchronise the PHC, because then we become less reliant on a variable latency accessing the PHY over the MDIO bus. The hardware event capture allows the PHC to be captured on a hardware signal, software can then read that timestamp, and if the hardware event is a PPS, then that can be used to ensure that the PHC is ticking at the correct rate. If the PPS is also aligned to a second boundary, then the hardware PHC can also be aligned. With both, the latency of the MDIO bus becomes irrelevant, and PTP at the PHY becomes a far more preferable option.

Note that things which can influence the latency over the MDIO bus include how many PHYs or other devices are also on it, and the rate at which accesses to those PHYs are performed. Then there's latency from kernel locking and scheduling. Maybe interrupt latency if the MDIO bus driver uses interrupts.

When I created the generic Marvell PHY PTP layer (my patch set for Marvell PHYs) I tried very hard to eliminate as many of these sources of variable latency as possible - such as avoiding taking and releasing locks on every MDIO bus access. Even with that, I could not get the PHY's PHC to synchronise any better than 1.5ppm, vs 40ppb for the PP2 MAC's PHC.

The last bit of consideration is whether the PHCs can be synchronised in hardware. If one has multiple ethernet interfaces, no hardware synchronisation of PHY PHCs, but also have MAC PTP support which is shared across the ethernet interfaces, why would one want to use the PHY with software based synchronisation of their individual PHCs.

So, to wrap this email up... if one has hardware purposely designed for PTP support, which uses the PTP hardware signals from the various PHCs, then one would want to use the PHY based timestamping. If one doesn't have that, then one would want to use something that gives the best timestamps, which may not necessarily be the PHY.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#55 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

(In essence, because of all the noise when trying the Marvell PHY with ptp4l, I came to the conlusion that NTP was a far better solution to time synchronisation between machines than PTP would ever be due to the nose induced by MDIO access. However, I should also state that I basically gave up with PTP in the end because hardware support is overall poor, and NTP just works - and I'd still have to run NTP for the machines that have no PTP capabilities. PTP probably only makes sense if one has a nice expensive grand master PTP clock on ones network, and all the machines one wants to synchronise have decent PTP implementations.)

Quoted 7 lines
Quoted 7 lines

Don't wanna waste too much of your time with the questions since I haven't done much research but - wouldn't MAC timestamp be a better choice more often (as long as it's a real, to-spec PTP stamp)? Are we picking PHY for historical reasons?

Not that flipping the default would address the problem of regressing some setups..

Another thing I pointed out some time ago is that you might have a working setup with MAC timestamping. Then comes along a patch which adds PHY timestamping for the PHY you are using and it might break your setup, because of the "PHY timestamping first" rule.

Quoted 15 lines

There is the argument in PTP that having the PHY do the timestamping results in the timestamps being "closer to the wire" meaning that they will not be subject to any delays between the packet leaving the MAC and being transmitted on the wire. Some PHYs have FIFOs or other buffering which introduces a delay which PTP doesn't desire.

TI has a blog on this:

https://e2e.ti.com/blogs_/b/analogwire/posts/how-to-implement-ieee-1588-time-stamping-in-an-ethernet-transceiver

However, what is failed to be mentioned there is that yes, doing PTP at the PHY means one can accurately trigger a capture of the timestamp from the PHC when the SFD is detected on the media. That is a great advantage, but is really only half the story.

Quoted 6 lines

If the PHC (hardware counter) in the PHY can't be accurately synchronised, for example, it has a 1.5ppm second-to-second variability, then the resulting timestamps will also have that same variability. Another way to look at it is they will appear to have 1.5ppm of noise. If this noise is random, then it becomes difficult to filter that noise out, and results in jitter.

Exactly this.

Quoted 4 lines

However, timestamping at the MAC may have only 40ppb of variability, but have a resulting delay through the PHY. The delay through the PHY will likely be constant, so the timestamps gathered at the MAC will have a constant error but have much less noise.

One thing to consider is also the quality of the oscillator of each part. E.g. we design boards which have special temperture controlled oscillators. How would you know where that one is connected to, MAC? PHY? Both?

Quoted 30 lines

Things change if one can use hardware signals to synchronise the PHC, because then we become less reliant on a variable latency accessing the PHY over the MDIO bus. The hardware event capture allows the PHC to be captured on a hardware signal, software can then read that timestamp, and if the hardware event is a PPS, then that can be used to ensure that the PHC is ticking at the correct rate. If the PPS is also aligned to a second boundary, then the hardware PHC can also be aligned. With both, the latency of the MDIO bus becomes irrelevant, and PTP at the PHY becomes a far more preferable option.

Note that things which can influence the latency over the MDIO bus include how many PHYs or other devices are also on it, and the rate at which accesses to those PHYs are performed. Then there's latency from kernel locking and scheduling. Maybe interrupt latency if the MDIO bus driver uses interrupts.

When I created the generic Marvell PHY PTP layer (my patch set for Marvell PHYs) I tried very hard to eliminate as many of these sources of variable latency as possible - such as avoiding taking and releasing locks on every MDIO bus access. Even with that, I could not get the PHY's PHC to synchronise any better than 1.5ppm, vs 40ppb for the PP2 MAC's PHC.

The last bit of consideration is whether the PHCs can be synchronised in hardware. If one has multiple ethernet interfaces, no hardware synchronisation of PHY PHCs, but also have MAC PTP support which is shared across the ethernet interfaces, why would one want to use the PHY with software based synchronisation of their individual PHCs.

Quoted 5 lines

So, to wrap this email up... if one has hardware purposely designed for PTP support, which uses the PTP hardware signals from the various PHCs, then one would want to use the PHY based timestamping. If one doesn't have that, then one would want to use something that gives the best timestamps, which may not necessarily be the PHY.

Which would make that a property of the board and not something the software could figure out most of the time.

IMHO it really depends on how precise one want to go. Just saying PHY timestamping is better isn't true. Vendors might go great lengths to design their boards to do very well regarding precision including PHY timestamping and maybe some kind of synchronization.

But in genernal, MAC timestamping - if available and not totally broken - is the better choice IMHO. If it wasn't for the legacy rule, I'd prefer MAC timestamping and just use PHY timestamping if MAC timestamping is not available. If both are available, PHY timestamping should be an opt-in.

-michael

maintainer
#56 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

I'm not sure we are making much progress here...

Lets divide an conquer. As far as i can see we have the following bits of work to do:

1) Kernel internal plumbing to allow multiple time stampers for one netdev. The PTP core probably needs to be the mux for all kAPI calls, and any internal calling between components. This might mean changes to all MAC drivers supporting PTP and time stampers. But i don't think there is anything too controversial here, just plumbing work.

2) Some method to allow user space to control which time stamper is used. Either an extension of the existing IOCTL interface, or maybe ethtool. Depending on how ambitious we want to be, add a netlink API to eventually replace the IOCTL interface?

3) Add a device tree binding to control which time stamper is used. Probably a MAC property. Also probably not too controversial.

4) Some solution to the default choice if there is no DT property.

Andrew

#57 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 11 lines · On Fri, 3 Mar 2023 14:20:20 +0100 Andrew Lunn <[email protected]> wrote:

I'm not sure we are making much progress here...

Lets divide an conquer. As far as i can see we have the following bits of work to do:

1) Kernel internal plumbing to allow multiple time stampers for one netdev. The PTP core probably needs to be the mux for all kAPI calls, and any internal calling between components. This might mean changes to all MAC drivers supporting PTP and time stampers. But i don't think there is anything too controversial here, just plumbing work.

Quoted 4 lines

2) Some method to allow user space to control which time stamper is used. Either an extension of the existing IOCTL interface, or maybe ethtool. Depending on how ambitious we want to be, add a netlink API to eventually replace the IOCTL interface?

Isn't the patch series (with small revisions) from Richard sufficient for this two points?
https://lkml.kernel.org/netdev/Y%[email protected]/

Quoted 4 lines

3) Add a device tree binding to control which time stamper is used. Probably a MAC property. Also probably not too controversial.

4) Some solution to the default choice if there is no DT property.

And in cases of architectures which do not support DT how do we deal with it?

Regards,
Köry

maintainer
#58 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 14 lines · Köry Maincent
Quoted 11 lines · On Fri, 3 Mar 2023 14:20:20 +0100 Andrew Lunn <[email protected]> wrote:

I'm not sure we are making much progress here...

Lets divide an conquer. As far as i can see we have the following bits of work to do:

1) Kernel internal plumbing to allow multiple time stampers for one netdev. The PTP core probably needs to be the mux for all kAPI calls, and any internal calling between components. This might mean changes to all MAC drivers supporting PTP and time stampers. But i don't think there is anything too controversial here, just plumbing work.

Quoted 8 lines
Quoted 4 lines

2) Some method to allow user space to control which time stamper is used. Either an extension of the existing IOCTL interface, or maybe ethtool. Depending on how ambitious we want to be, add a netlink API to eventually replace the IOCTL interface?

Isn't the patch series (with small revisions) from Richard sufficient for this two points?
https://lkml.kernel.org/netdev/Y%[email protected]/

I've not looked at it. How about you make the small revisions and post it. So long as the Marvell PHY PTP code is not merged, it should be safe to work on these parts and not cause regressions.

Quoted 6 lines
Quoted 4 lines

3) Add a device tree binding to control which time stamper is used. Probably a MAC property. Also probably not too controversial.

4) Some solution to the default choice if there is no DT property.

And in cases of architectures which do not support DT how do we deal with it?

x86 has ACPI. It seems like ACPI folks are happy to simply stuff DT properties into ACPI nodes, rather than come up with a native ACPI way. But we don't really need to support ACPI until somebody has an ACPI board which needs to configure the time stamper.

Harder is things like USB, which mostly never has DT properties. But a USB dongle tends to have known components, both the MAC and the PHY is known, and so the dongle driver could make a call into the PTP core to select which time stamper should be used. But i would suggest we solve this problem when we actually reach it, not now.

Andrew

#59 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Andrew Lunn
Quoted 16 lines · Andrew Lunn

I'm not sure we are making much progress here...

Lets divide an conquer. As far as i can see we have the following bits of work to do:

1) Kernel internal plumbing to allow multiple time stampers for one netdev. The PTP core probably needs to be the mux for all kAPI calls, and any internal calling between components. This might mean changes to all MAC drivers supporting PTP and time stampers. But i don't think there is anything too controversial here, just plumbing work.

2) Some method to allow user space to control which time stamper is used. Either an extension of the existing IOCTL interface, or maybe ethtool. Depending on how ambitious we want to be, add a netlink API to eventually replace the IOCTL interface?

Quoted 4 lines

3) Add a device tree binding to control which time stamper is used. Probably a MAC property. Also probably not too controversial.

4) Some solution to the default choice if there is no DT property.

I thought (4) was what we have been discussing... but the problem seems to be that folk want to drill down into the fine detail about whether PHY or MAC timestamping is better than the other.

As I've already stated, I doubt DT maintainers will wear having a DT property for this, on the grounds that it's very much a software control rather than a hardware description.

On the other hand, if DT described the connectivity of the PTP hardware signals, then one could probably make a decision based upon that, and as that's describing hardware, it would be less problematical during DT review.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
maintainer
#60 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 9 lines
Quoted 1 line

4) Some solution to the default choice if there is no DT property.

I thought (4) was what we have been discussing... but the problem seems to be that folk want to drill down into the fine detail about whether PHY or MAC timestamping is better than the other.

As I've already stated, I doubt DT maintainers will wear having a DT property for this, on the grounds that it's very much a software control rather than a hardware description.

We can argue it is describing the hardware. In that hardware blob X has a better implementation of PTP than hardware blob Y. That could be because of the basic features of the blob, like resolution, adjustable clocks, or board specific features, like temperature compensated crystals, etc.

There could also be a linkage to the default choice. If we go with the idea of giving each PTP stamper a quality indicator, and the highest quality wins, we can have the DT property change this quality property because it knows of external things, like the lack of a temperature compensated crystal.

Andrew

#61 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 10 lines · Andrew Lunn
Quoted 9 lines
Quoted 1 line

4) Some solution to the default choice if there is no DT property.

I thought (4) was what we have been discussing... but the problem seems to be that folk want to drill down into the fine detail about whether PHY or MAC timestamping is better than the other.

As I've already stated, I doubt DT maintainers will wear having a DT property for this, on the grounds that it's very much a software control rather than a hardware description.

Quoted 5 lines

We can argue it is describing the hardware. In that hardware blob X has a better implementation of PTP than hardware blob Y. That could be because of the basic features of the blob, like resolution, adjustable clocks, or board specific features, like temperature compensated crystals, etc.

IMO a DTS property is the most user friendly solution, even if it isn't strictly hardware description.

Thanks,
Richard

#62 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 11 lines · Richard Cochran
Quoted 10 lines · Andrew Lunn
Quoted 9 lines
Quoted 1 line

4) Some solution to the default choice if there is no DT property.

I thought (4) was what we have been discussing... but the problem seems to be that folk want to drill down into the fine detail about whether PHY or MAC timestamping is better than the other.

As I've already stated, I doubt DT maintainers will wear having a DT property for this, on the grounds that it's very much a software control rather than a hardware description.

Quoted 8 lines
Quoted 5 lines

We can argue it is describing the hardware. In that hardware blob X has a better implementation of PTP than hardware blob Y. That could be because of the basic features of the blob, like resolution, adjustable clocks, or board specific features, like temperature compensated crystals, etc.

IMO a DTS property is the most user friendly solution, even if it isn't strictly hardware description.

Afte all, the engineer who designs the board will know which is the better choice for use case the board targets.

Thanks,
Richard

maintainer
#63 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Russell King (Oracle)
Quoted 5 lines · On Fri, 3 Mar 2023 14:03:03 +0000 Russell King (Oracle) wrote:
Quoted 1 line

4) Some solution to the default choice if there is no DT property.

I thought (4) was what we have been discussing... but the problem seems to be that folk want to drill down into the fine detail about whether PHY or MAC timestamping is better than the other.

There seems to be no solid heuristic. Maybe it's better to give up and force the user to pick (for new drivers)?

#64 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Jakub Kicinski
Quoted 4 lines · Jakub Kicinski

Don't wanna waste too much of your time with the questions since I haven't done much research but - wouldn't MAC timestamp be a better choice more often (as long as it's a real, to-spec PTP stamp)? Are we picking PHY for historical reasons?

I think the default rating for MACs can be higher, but in the case of at least one PHY, the TI PHYTER (dp83640), the PHY is the far better choice. This PHY surpasses any MAC on the market, and so it should have a high rating.

For example, I've seen boards with the TI am335x, initially selected because of the built in CPTS PTP feature, which the designers were forced augment with the PHYTERs simply because of inadequate SoC implementation.

In other words, if you find a board with the PHYTER, then it is there for a reason.

Thanks,
Richard

#65 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Russell King (Oracle)
Quoted 4 lines · Russell King (Oracle)

Therefore, I believe that the Marvell PHY PTP implementation is all round inferior to that found in the Marvell PP2 MAC, and hence why I believe that the PP2 MAC implementation should be used by default over the PHY.

Yeah, that phy sure sounds like a lemon.

Quoted 10 lines

(In essence, because of all the noise when trying the Marvell PHY with ptp4l, I came to the conlusion that NTP was a far better solution to time synchronisation between machines than PTP would ever be due to the nose induced by MDIO access. However, I should also state that I basically gave up with PTP in the end because hardware support is overall poor, and NTP just works - and I'd still have to run NTP for the machines that have no PTP capabilities. PTP probably only makes sense if one has a nice expensive grand master PTP clock on ones network, and all the machines one wants to synchronise have decent PTP implementations.)

Yes, NTP is really what most people need, and with PTP you really must carefully select the hardware. There is lots of PTP junk on the market.

Thanks,
Richard

#66 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Russell King (Oracle)

Hello Russell,

Quoted 23 lines · On Mon, 27 Feb 2023 15:20:05 +0000 "Russell King (Oracle)"…
Quoted 8 lines · Köry Maincent

Hello RMK,

Quoted 5 lines

Hence why I'm at the point of giving up; I don't see that PTP will be of very limited benefit on my network with all these issues, and in any case, NTP has been "good enough" for the last 20+ years. Given that only a limited number of machines will be able to implement PTP support anyway, NTP will have to run along side it.

Quoted 9 lines

I see this patch has been abandoned.
I am testing it with a ZynqMP board (macb ethernet) and it seems to more or less work. It got tx timestamp timeout at initialization but after some times (~20 seconds) ptp4l manages to set it working. Also the IEEE 802.3 network PTP mode is not working, it constantly throw rx timestamp overrun errors.
I will aim at fixing these issues and adding support to interrupts. It would be good to have it accepted mainline. What do you think is missing for that?

It isn't formally abandoned, but is permanently on-hold as merging Marvell PHY PTP support into mainline _will_ regress the superior PTP support on the Macchiatobin platform for the reasons outlined in:

Quoted 5 lines

https://lore.kernel.org/netdev/[email protected]/

Attempting to fix this problem was basically rejected by the PTP maintainer, and thus we're at a deadlock over the issue, and Marvell PHY PTP support can never be merged into mainline.

As we are currently moving forward on PTP core to resolve this issue, I would like to investigate your PHY PTP patch in parallel. Indeed it does not work very well on my side.

The PTP UDP v4 and v6 work only if I add "--tx_timestamp_timeout 20" and the PTP IEEE 802.3 (802.1AS) does not work at all.
On PTP IEEE 802.3 network transport ("ptp4l -2") I get continuously rx timestamp overrun:
Marvell 88E1510 ff0d0000.ethernet-ffffffff:01: rx timestamp overrun (5) Marvell 88E1510 ff0d0000.ethernet-ffffffff:01: rx timestamp overrun (5)

I know it's been a long time but does it ring a bell on your memory?

maintainer
#67 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 4 lines

As we are currently moving forward on PTP core to resolve this issue, I would like to investigate your PHY PTP patch in parallel. Indeed it does not work very well on my side.

Quoted 8 lines

The PTP UDP v4 and v6 work only if I add "--tx_timestamp_timeout 20" and the PTP IEEE 802.3 (802.1AS) does not work at all.
On PTP IEEE 802.3 network transport ("ptp4l -2") I get continuously rx timestamp overrun:
Marvell 88E1510 ff0d0000.ethernet-ffffffff:01: rx timestamp overrun (5) Marvell 88E1510 ff0d0000.ethernet-ffffffff:01: rx timestamp overrun (5)

I know it's been a long time but does it ring a bell on your memory?

How are you talking to the PHY? I had issues with slow MDIO busses, especially those embedded within an Ethernet switch. You end up with MDIO over MDIO which has a lot of latency. I _think_ i added some patches to ptp4l to deal with this, but i forget exactly what landed.

Andrew

#68 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

Quoted 11 lines · On Thu, 27 Apr 2023 18:50:47 +0200 Andrew Lunn <[email protected]> wrote:
Quoted 11 lines

As we are currently moving forward on PTP core to resolve this issue, I would like to investigate your PHY PTP patch in parallel. Indeed it does not work very well on my side.

The PTP UDP v4 and v6 work only if I add "--tx_timestamp_timeout 20" and the PTP IEEE 802.3 (802.1AS) does not work at all.
On PTP IEEE 802.3 network transport ("ptp4l -2") I get continuously rx timestamp overrun:
Marvell 88E1510 ff0d0000.ethernet-ffffffff:01: rx timestamp overrun (5) Marvell 88E1510 ff0d0000.ethernet-ffffffff:01: rx timestamp overrun (5)

Quoted 6 lines
Quoted 1 line

I know it's been a long time but does it ring a bell on your memory?

How are you talking to the PHY? I had issues with slow MDIO busses, especially those embedded within an Ethernet switch. You end up with MDIO over MDIO which has a lot of latency. I _think_ i added some patches to ptp4l to deal with this, but i forget exactly what landed.

I am talking to the PHY through a simple MDIO bus.
The current PTP support from Russell does not support interrupts. Could it be the cause of needing bigger tx timestamp timeout?

#69 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Russell King - ARM Linux admin
Quoted 4 lines · Russell King - ARM Linux admin

So, something in the IPv4 layer on ARM64 is silently discarding multicast UDP PTP packets, and I've no idea what... and I'm coming to the conclusion that this is all way too much effort and way too unreliable to be worth spending any more time trying to make work.

I've also seen this on the Marvell switch. I'm not sure if the root cause is in the Linux stack or in the mcast forwarding logic in the switch. Using the Marvell, I can make a layer2 TC or a UDPv4 BC, but not a UDPv4 TC.

I haven't had time to dig deeper, and I think many people are happy with a layer2 TC as that is called out in the TSN world.

Thanks,
Richard

maintainer
#70 lore raw

Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support [multicast/DSA issues]

In reply to Russell King - ARM Linux admin
Quoted 26 lines

Hmm, and even with IP_MULTICAST=y, I still can't get the "timestamping" program from the kernel sources to work.

On two different machines, I'm running:

# ./timestamping32 eno0 SOF_TIMESTAMPING_RX_HARDWARE SOF_TIMESTAMPING_RAW_HARDWARE

On one machine (arm32) this works - it can see the traffic it generates and receives from the other machine.

On the other machine (arm64), it sees _no_ traffic at all, but tcpdump on that machine can see the traffic being received:

12:36:40.002065 00:51:82:11:33:03 > 01:00:5e:00:01:82, ethertype IPv4 (0x0800), length 166: (tos 0x0, ttl 1, id 15045, offset 0, flags [DF], proto UDP (17), length 152)

    192.168.3.1.319 > 224.0.1.130.319: [bad udp cksum 0xa5c1 -> 0x7aaa!] UDP, length 124
12:36:41.105391 00:50:43:02:03:02 > 01:00:5e:00:01:82, ethertype IPv4 (0x0800),
length 166: (tos 0x0, ttl 1, id 9715, offset 0, flags [DF], proto UDP (17), length 152)
    192.168.3.2.319 > 224.0.1.130.319: [udp sum ok] UDP, length 124

The bad udp cksum is due to checksum offloadong on transmit - 3.1 is the arm64 host running that tcpdump.

When I look at /proc/net/snmp, I can see the IP InReceives incrementing but not the IP InDelivers nor UDP InDatagrams counter:

Quoted 6 lines

Ip: 2 64 1602 0 3 0 0 0 297 531 0 46 0 0 0 0 0 0 0
Udp: 572 0 0 480 0 0 0 4
Ip: 2 64 1603 0 3 0 0 0 297 531 0 46 0 0 0 0 0 0 0
Udp: 572 0 0 480 0 0 0 4

I don't have any firewall rules on the machine.

Hi Russell

A shot in the dark....

I remember somewhere in the stack BPF is used to identify PTP packets. Is it maybe getting JITed wrongly on arm64? Maybe disable the BPF JIT and see if it starts working?

Andrew

#71 lore raw
In reply to Russell King - ARM Linux admin

Hello Russell,

Quoted 9 lines

I have the situation on a couple of devices where there are multiple places that can do PTP timestamping:

- the PHY (slow to access, only event capture which may not be wired, doesn't seem to synchronise well - delay of 58000, frequency changes every second between +/-1500ppb.)
- the Ethernet MAC (fast to access, supports event capture and trigger generation which also may not be wired, synchronises well, delay of 700, frequency changes every second +/- 40ppb.)

Does this test have been made with the marvell 88E151x PHY?
On my side with a zynqMP SOM (cadence MACB MAC) the synchronization of the PHY PTP clock is way better: +/-50ppb. Do you have an idea about the difference? Which link partner were you using? stm32mp157 hardware PTP on my side.

On the thread mail we are talking about PTP rate scale but as we can see you and I have really not the same synchronisation on this PHY PTP. How could we set a PTP rate scale in the driver if on two different board the result vary a lot? Although I don't understand why.

I have also differences with the PTP MAC but it is expected as it is not the same MAC.

Regards,
Köry

#72 lore raw
Quoted 5 lines · Köry Maincent

Hello Russell,

Quoted 3 lines

I have the situation on a couple of devices where there are multiple places that can do PTP timestamping:

Quoted 8 lines
Quoted 6 lines

- the PHY (slow to access, only event capture which may not be wired, doesn't seem to synchronise well - delay of 58000, frequency changes every second between +/-1500ppb.)
- the Ethernet MAC (fast to access, supports event capture and trigger generation which also may not be wired, synchronises well, delay of 700, frequency changes every second +/- 40ppb.)

Does this test have been made with the marvell 88E151x PHY?

Yes. Remember, however, that there are many factors that influence the access latency (as I detailed in my chronologically previous reply on the other thread.)

Quoted 3 lines

On my side with a zynqMP SOM (cadence MACB MAC) the synchronization of the PHY PTP clock is way better: +/-50ppb. Do you have an idea about the difference? Which link partner were you using? stm32mp157 hardware PTP on my side.

Both of my results were through synchronising the PHC with the machine's local clock (which had already been synchronised via NTP), thereby creating a "grand master" for my network. That's the only way to setup a grand master short of purchasing specific kit for that job (and from what I can see, that's $$$$$ - and out of reach for your average volunteer kernel maintainer).

I also rigged up the PP2 to provide a PPS signal, connected that to my TXCO-based frequency counter, measuring the PPS period - which gives independent measurement of the stability of the PHC. It wasn't possible to do that with the PHY since the hardware signals aren't exposed.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#73 lore raw
In reply to Köry Maincent
Quoted 3 lines · Köry Maincent

On my side with a zynqMP SOM (cadence MACB MAC) the synchronization of the PHY PTP clock is way better: +/-50ppb. Do you have an idea about the difference? Which link partner were you using? stm32mp157 hardware PTP on my side.

The predecessor Zynq 7000 SoC had a PTP MAC that was unusable. So much so, that the vendor simply deleted the PTP chapters from the data sheet without a word. IIRC, the newer MP has a different implementation, but maybe they also flubbed that one?

In any case, this is another example of how a good PHY implementation covers for a poor MAC one.

Thanks,
Richard