Skip to contentSkip to conversation

ethtool: add setting frame preemption of traffic classes

PATCH In review Po Liu 8 participants 38 messages Nov 27, 2019 – May 13, 2020

Subsystems (4)
lore
Share & tools
lore

Po Liu

author
#0 lore raw

PATCH 1/2 ethtool: add setting frame preemption of traffic classes

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

---
 include/linux/netdev_features.h | 5 ++++-
 include/uapi/linux/ethtool.h    | 5 ++++-
 net/core/ethtool.c              | 1 +
 3 files changed, 9 insertions(+), 2 deletions(-)
include/linux/netdev_features.h +4 -1
index 4b19c544c59a..299750a8b414 100644
@@ -80,6 +80,7 @@ enum {
 
 	NETIF_F_GRO_HW_BIT,		/* Hardware Generic receive offload */
 	NETIF_F_HW_TLS_RECORD_BIT,	/* Offload TLS record */
+	NETIF_F_HW_PREEMPTION_BIT,	/* Hardware TC frame preemption */
 
 	/*
 	 * Add your fresh new feature above and remember to update
@@ -150,6 +151,7 @@ enum {
 #define NETIF_F_GSO_UDP_L4	__NETIF_F(GSO_UDP_L4)
 #define NETIF_F_HW_TLS_TX	__NETIF_F(HW_TLS_TX)
 #define NETIF_F_HW_TLS_RX	__NETIF_F(HW_TLS_RX)
+#define NETIF_F_PREEMPTION	__NETIF_F(HW_PREEMPTION)
 
 /* Finds the next feature with the highest number of the range of start till 0.
  */
@@ -175,7 +177,8 @@ static inline int find_next_netdev_feature(u64 feature, unsigned long start)
 /* Features valid for ethtool to change */
 /* = all defined minus driver/device-class-related */
 #define NETIF_F_NEVER_CHANGE	(NETIF_F_VLAN_CHALLENGED | \
-				 NETIF_F_LLTX | NETIF_F_NETNS_LOCAL)
+				 NETIF_F_LLTX | NETIF_F_NETNS_LOCAL | \
+				 NETIF_F_PREEMPTION)
 
 /* remember that ((t)1 << t_BITS) is undefined in C99 */
 #define NETIF_F_ETHTOOL_BITS	((__NETIF_F_BIT(NETDEV_FEATURE_COUNT - 1) | \
include/uapi/linux/ethtool.h +4 -1
index d4591792f0b4..12ffb34afbfa 100644
@@ -1776,6 +1776,8 @@ enum ethtool_reset_flags {
 };
 #define ETH_RESET_SHARED_SHIFT	16
 
+/* Disable preemtion. */
+#define PREEMPTION_DISABLE     0x0
 
 /**
  * struct ethtool_link_settings - link control and status
@@ -1886,7 +1888,8 @@ struct ethtool_link_settings {
 	__s8	link_mode_masks_nwords;
 	__u8	transceiver;
 	__u8	reserved1[3];
-	__u32	reserved[7];
+	__u32	preemption;
+	__u32	reserved[6];
 	__u32	link_mode_masks[0];
 	/* layout of link_mode_masks fields:
 	 * __u32 map_supported[link_mode_masks_nwords];
net/core/ethtool.c +1 -0
index cd9bc67381b2..6ffcd8a602b8 100644
@@ -111,6 +111,7 @@ static const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN]
 	[NETIF_F_HW_TLS_RECORD_BIT] =	"tls-hw-record",
 	[NETIF_F_HW_TLS_TX_BIT] =	 "tls-hw-tx-offload",
 	[NETIF_F_HW_TLS_RX_BIT] =	 "tls-hw-rx-offload",
+	[NETIF_F_HW_PREEMPTION_BIT] =	 "tx-preemption",
 };
 
 static const char
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH 2/2 enetc: implement the enetc 802.1Qbu hardware function

The interface follow up the ethtool 'preemption' set/get. Hardware features also need to set hw_features with NETIF_F_PREEMPTION flag. So ethtool could check kernel link features if there is preemption capability of port.

There are two MACs in ENETC. One is express MAC which traffic classes in it are advanced transmition. Another is preemptable MAC which traffic classes are frame preemptable.

The hardware need to initialize the MACs at initial stage. And then set the preemption enable registers of traffic classes when ethtool set .get_link_ksettings/.set_link_ksettings stage.

To test the ENETC preemption capability, user need to set mqprio or taprio to mapping the traffic classes with priorities. Then use ethtool command to set 'preemption' with a 8 bits value. MSB represent high number traffic class.

---
 drivers/net/ethernet/freescale/enetc/enetc.c  |   3 +
 drivers/net/ethernet/freescale/enetc/enetc.h  |   4 +
 .../ethernet/freescale/enetc/enetc_ethtool.c  | 142 ++++++++++++++++--
 .../net/ethernet/freescale/enetc/enetc_hw.h   |  17 +++
 .../net/ethernet/freescale/enetc/enetc_pf.c   |  15 +-
 .../net/ethernet/freescale/enetc/enetc_qos.c  |   4 +
 6 files changed, 174 insertions(+), 11 deletions(-)
drivers/net/ethernet/freescale/enetc/enetc.c +3 -0
index 9db1b96ed9b9..be0d9916e6ea 100644
@@ -750,6 +750,8 @@ void enetc_get_si_caps(struct enetc_si *si)
 
 	if (val & ENETC_SIPCAPR0_QBV)
 		si->hw_features |= ENETC_SI_F_QBV;
+	if (val & ENETC_SIPCAPR0_QBU)
+		si->hw_features |= ENETC_SI_F_QBU;
 }
 
 static int enetc_dma_alloc_bdr(struct enetc_bdr *r, size_t bd_size)
@@ -1448,6 +1450,7 @@ static int enetc_setup_tc_mqprio(struct net_device *ndev, void *type_data)
 	num_tc = mqprio->num_tc;
 
 	if (!num_tc) {
+		enetc_preemption_set(ndev, 0);
 		netdev_reset_tc(ndev);
 		netif_set_real_num_tx_queues(ndev, priv->num_tx_rings);
 
drivers/net/ethernet/freescale/enetc/enetc.h +4 -0
index 7ee0da6d0015..cfa74fa326e8 100644
@@ -119,6 +119,7 @@ enum enetc_errata {
 };
 
 #define ENETC_SI_F_QBV BIT(0)
+#define ENETC_SI_F_QBU BIT(1)
 
 /* PCI IEP device data */
 struct enetc_si {
@@ -177,6 +178,7 @@ enum enetc_active_offloads {
 	ENETC_F_RX_TSTAMP	= BIT(0),
 	ENETC_F_TX_TSTAMP	= BIT(1),
 	ENETC_F_QBV             = BIT(2),
+	ENETC_F_QBU		= BIT(3),
 };
 
 struct enetc_ndev_priv {
@@ -261,3 +263,5 @@ int enetc_setup_tc_cbs(struct net_device *ndev, void *type_data);
 #define enetc_sched_speed_set(ndev) (void)0
 #define enetc_setup_tc_cbs(ndev, type_data) -EOPNOTSUPP
 #endif
+
+int enetc_preemption_set(struct net_device *ndev, u32 ptvector);
drivers/net/ethernet/freescale/enetc/enetc_ethtool.c +133 -9

180 lines not on this page · raw message

drivers/net/ethernet/freescale/enetc/enetc_hw.h +17 -0
index 51f543ef37a8..b609ec095710 100644
@@ -19,6 +19,7 @@
 #define ENETC_SICTR1	0x1c
 #define ENETC_SIPCAPR0	0x20
 #define ENETC_SIPCAPR0_QBV	BIT(4)
+#define ENETC_SIPCAPR0_QBU	BIT(3)
 #define ENETC_SIPCAPR0_RSS	BIT(8)
 #define ENETC_SIPCAPR1	0x24
 #define ENETC_SITGTGR	0x30
@@ -176,6 +177,7 @@ enum enetc_bdr_type {TX, RX};
 #define ENETC_PCAPR0_RXBDR(val)	((val) >> 24)
 #define ENETC_PCAPR0_TXBDR(val)	(((val) >> 16) & 0xff)
 #define ENETC_PCAPR1		0x0904
+#define ENETC_NUM_TCS_MASK	GENMASK(6, 4)
 #define ENETC_PSICFGR0(n)	(0x0940 + (n) * 0xc)  /* n = SI index */
 #define ENETC_PSICFGR0_SET_TXBDR(val)	((val) & 0xff)
 #define ENETC_PSICFGR0_SET_RXBDR(val)	(((val) & 0xff) << 16)
@@ -223,6 +225,7 @@ enum enetc_bdr_type {TX, RX};
 #define ENETC_SET_TX_MTU(val)	((val) << 16)
 #define ENETC_SET_MAXFRM(val)	((val) & 0xffff)
 #define ENETC_PM0_IF_MODE	0x8300
+#define ENETC_PM1_IF_MODE       0x9300
 #define ENETC_PMO_IFM_RG	BIT(2)
 #define ENETC_PM0_IFM_RLP	(BIT(5) | BIT(11))
 #define ENETC_PM0_IFM_RGAUTO	(BIT(15) | ENETC_PMO_IFM_RG | BIT(1))
@@ -276,6 +279,15 @@ enum enetc_bdr_type {TX, RX};
 #define ENETC_PM0_TSCOL		0x82E0
 #define ENETC_PM0_TLCOL		0x82E8
 #define ENETC_PM0_TECOL		0x82F0
+#define ENETC_PM1_RFRM		0x9120
+#define ENETC_PM1_RDRP		0x9158
+#define ENETC_PM1_RPKT		0x9160
+#define ENETC_PM1_RFRG		0x91B8
+#define ENETC_PM1_TFRM		0x9220
+#define ENETC_PM1_TERR		0x9238
+#define ENETC_PM1_TPKT		0x9260
+#define ENETC_MAC_MERGE_MMFCRXR	0x1f14
+#define ENETC_MAC_MERGE_MMFCTXR	0x1f18
 
 /* Port counters */
 #define ENETC_PICDR(n)		(0x0700 + (n) * 8) /* n = [0..3] */
@@ -615,3 +627,8 @@ struct enetc_cbd {
 /* Port time gating capability register */
 #define ENETC_QBV_PTGCAPR_OFFSET	0x11a08
 #define ENETC_QBV_MAX_GCL_LEN_MASK	GENMASK(15, 0)
+
+#define ENETC_QBU_TC_MASK	GENMASK(7, 0)
+
+#define ENETC_PTCFPR(n)         (0x1910 + (n) * 4)
+#define ENETC_FPE               BIT(31)
drivers/net/ethernet/freescale/enetc/enetc_pf.c +13 -2
index e7482d483b28..f1873c4da77f 100644
@@ -523,10 +523,15 @@ static void enetc_configure_port_mac(struct enetc_hw *hw)
 		      ENETC_PM0_CMD_TXP	| ENETC_PM0_PROMISC |
 		      ENETC_PM0_TX_EN | ENETC_PM0_RX_EN);
 	/* set auto-speed for RGMII */
-	if (enetc_port_rd(hw, ENETC_PM0_IF_MODE) & ENETC_PMO_IFM_RG)
+	if (enetc_port_rd(hw, ENETC_PM0_IF_MODE) & ENETC_PMO_IFM_RG) {
 		enetc_port_wr(hw, ENETC_PM0_IF_MODE, ENETC_PM0_IFM_RGAUTO);
-	if (enetc_global_rd(hw, ENETC_G_EPFBLPR(1)) == ENETC_G_EPFBLPR1_XGMII)
+		enetc_port_wr(hw, ENETC_PM1_IF_MODE, ENETC_PM0_IFM_RGAUTO);
+	}
+
+	if (enetc_global_rd(hw, ENETC_G_EPFBLPR(1)) == ENETC_G_EPFBLPR1_XGMII) {
 		enetc_port_wr(hw, ENETC_PM0_IF_MODE, ENETC_PM0_IFM_XGMII);
+		enetc_port_wr(hw, ENETC_PM1_IF_MODE, ENETC_PM0_IFM_XGMII);
+	}
 }
 
 static void enetc_configure_port_pmac(struct enetc_hw *hw)
@@ -745,6 +750,12 @@ static void enetc_pf_netdev_setup(struct enetc_si *si, struct net_device *ndev,
 	if (si->hw_features & ENETC_SI_F_QBV)
 		priv->active_offloads |= ENETC_F_QBV;
 
+	if (si->hw_features & ENETC_SI_F_QBU) {
+		ndev->hw_features |= NETIF_F_PREEMPTION;
+		ndev->features |= NETIF_F_PREEMPTION;
+		priv->active_offloads |= ENETC_F_QBU;
+	}
+
 	/* pick up primary MAC address from SI */
 	enetc_get_primary_mac_addr(&si->hw, ndev->dev_addr);
 }
drivers/net/ethernet/freescale/enetc/enetc_qos.c +4 -0
index 2e99438cb1bf..94dde847d052 100644
@@ -169,6 +169,10 @@ int enetc_setup_tc_taprio(struct net_device *ndev, void *type_data)
 					   priv->tx_ring[i]->index,
 					   taprio->enable ? 0 : i);
 
+	/* preemption off if TC priority is all 0 */
+	if ((err && taprio->enable) || !(err || taprio->enable))
+		enetc_preemption_set(ndev, 0);
+
 	return err;
 }
 
Signature · 1 line
2.17.1
#2 lore raw

Re: [v1,net-next, 2/2] enetc: implement the enetc 802.1Qbu hardware function

Hi Po,

Quoted 15 lines · Po Liu

The interface follow up the ethtool 'preemption' set/get. Hardware features also need to set hw_features with NETIF_F_PREEMPTION flag. So ethtool could check kernel link features if there is preemption capability of port.

There are two MACs in ENETC. One is express MAC which traffic classes in it are advanced transmition. Another is preemptable MAC which traffic classes are frame preemptable.

The hardware need to initialize the MACs at initial stage. And then set the preemption enable registers of traffic classes when ethtool set .get_link_ksettings/.set_link_ksettings stage.

Quoted 7 lines

To test the ENETC preemption capability, user need to set mqprio or taprio to mapping the traffic classes with priorities. Then use ethtool command to set 'preemption' with a 8 bits value. MSB represent high number traffic class.

---

Just to clarify, the net-next tree is closed during the following 2 weeks or so, for the 5.5 merge window:
http://vger.kernel.org/~davem/net-next.html
Only bugfix and RFC patches (aka for the sake of discussion) patches should be sent during this time.
So let's treat this series as RFC.

Quoted 349 lines
 drivers/net/ethernet/freescale/enetc/enetc.c  |   3 +
 drivers/net/ethernet/freescale/enetc/enetc.h  |   4 +
 .../ethernet/freescale/enetc/enetc_ethtool.c  | 142 ++++++++++++++++--
 .../net/ethernet/freescale/enetc/enetc_hw.h   |  17 +++
 .../net/ethernet/freescale/enetc/enetc_pf.c   |  15 +-
 .../net/ethernet/freescale/enetc/enetc_qos.c  |   4 +
 6 files changed, 174 insertions(+), 11 deletions(-)
drivers/net/ethernet/freescale/enetc/enetc.c +3 -0
index 9db1b96ed9b9..be0d9916e6ea 100644
@@ -750,6 +750,8 @@ void enetc_get_si_caps(struct enetc_si *si)
 
        if (val & ENETC_SIPCAPR0_QBV)
                si->hw_features |= ENETC_SI_F_QBV;
+       if (val & ENETC_SIPCAPR0_QBU)
+               si->hw_features |= ENETC_SI_F_QBU;
 }
 
 static int enetc_dma_alloc_bdr(struct enetc_bdr *r, size_t bd_size)
@@ -1448,6 +1450,7 @@ static int enetc_setup_tc_mqprio(struct net_device *ndev, void *type_data)
        num_tc = mqprio->num_tc;
 
        if (!num_tc) {
+               enetc_preemption_set(ndev, 0);
                netdev_reset_tc(ndev);
                netif_set_real_num_tx_queues(ndev, priv->num_tx_rings);
 
drivers/net/ethernet/freescale/enetc/enetc.h +4 -0
index 7ee0da6d0015..cfa74fa326e8 100644
@@ -119,6 +119,7 @@ enum enetc_errata {
 };
 
 #define ENETC_SI_F_QBV BIT(0)
+#define ENETC_SI_F_QBU BIT(1)
 
 /* PCI IEP device data */
 struct enetc_si {
@@ -177,6 +178,7 @@ enum enetc_active_offloads {
        ENETC_F_RX_TSTAMP       = BIT(0),
        ENETC_F_TX_TSTAMP       = BIT(1),
        ENETC_F_QBV             = BIT(2),
+       ENETC_F_QBU             = BIT(3),
 };
 
 struct enetc_ndev_priv {
@@ -261,3 +263,5 @@ int enetc_setup_tc_cbs(struct net_device *ndev, void *type_data);
 #define enetc_sched_speed_set(ndev) (void)0
 #define enetc_setup_tc_cbs(ndev, type_data) -EOPNOTSUPP
 #endif
+
+int enetc_preemption_set(struct net_device *ndev, u32 ptvector);
drivers/net/ethernet/freescale/enetc/enetc_ethtool.c +133 -9
index 880a8ed8bb47..4c7425539280 100644
@@ -183,6 +183,21 @@ static const struct {
        { ENETC_PICDR(3),   "ICM DR3 discarded frames" },
 };
 
+static const struct {
+       int reg;
+       char name[ETH_GSTRING_LEN];
+} enetc_pmac_counters[] = {
+       { ENETC_PM1_RFRM,   "PMAC rx frames" },
+       { ENETC_PM1_RPKT,   "PMAC rx packets" },
+       { ENETC_PM1_RDRP,   "PMAC rx dropped packets" },
+       { ENETC_PM1_RFRG,   "PMAC rx fragment packets" },
+       { ENETC_PM1_TFRM,   "PMAC tx frames" },
+       { ENETC_PM1_TERR,   "PMAC tx error frames" },
+       { ENETC_PM1_TPKT,   "PMAC tx packets" },
+       { ENETC_MAC_MERGE_MMFCRXR,   "MAC merge fragment rx counter" },
+       { ENETC_MAC_MERGE_MMFCTXR,   "MAC merge fragment tx counter"},
+};
+
 static const char rx_ring_stats[][ETH_GSTRING_LEN] = {
        "Rx ring %2d frames",
        "Rx ring %2d alloc errors",
@@ -195,15 +210,24 @@ static const char tx_ring_stats[][ETH_GSTRING_LEN] = {
 static int enetc_get_sset_count(struct net_device *ndev, int sset)
 {
        struct enetc_ndev_priv *priv = netdev_priv(ndev);
+       int len;
+
+       if (sset != ETH_SS_STATS)
+               return -EOPNOTSUPP;
 
-       if (sset == ETH_SS_STATS)
-               return ARRAY_SIZE(enetc_si_counters) +
-                       ARRAY_SIZE(tx_ring_stats) * priv->num_tx_rings +
-                       ARRAY_SIZE(rx_ring_stats) * priv->num_rx_rings +
-                       (enetc_si_is_pf(priv->si) ?
-                       ARRAY_SIZE(enetc_port_counters) : 0);
+       len = ARRAY_SIZE(enetc_si_counters) +
+             ARRAY_SIZE(tx_ring_stats) * priv->num_tx_rings +
+             ARRAY_SIZE(rx_ring_stats) * priv->num_rx_rings;
 
-       return -EOPNOTSUPP;
+       if (!enetc_si_is_pf(priv->si))
+               return len;
+
+       len += ARRAY_SIZE(enetc_port_counters);
+
+       if (priv->active_offloads & ENETC_F_QBU)
+               len += ARRAY_SIZE(enetc_pmac_counters);
+
+       return len;
 }
 
 static void enetc_get_strings(struct net_device *ndev, u32 stringset, u8 *data)
@@ -241,6 +265,16 @@ static void enetc_get_strings(struct net_device *ndev, u32 stringset, u8 *data)
                                ETH_GSTRING_LEN);
                        p += ETH_GSTRING_LEN;
                }
+
+               if (!(priv->active_offloads & ENETC_F_QBU))
+                       break;
+
+               for (i = 0; i < ARRAY_SIZE(enetc_pmac_counters); i++) {
+                       strlcpy(p, enetc_pmac_counters[i].name,
+                               ETH_GSTRING_LEN);
+                       p += ETH_GSTRING_LEN;
+               }
+
                break;
        }
 }
@@ -268,6 +302,12 @@ static void enetc_get_ethtool_stats(struct net_device *ndev,
 
        for (i = 0; i < ARRAY_SIZE(enetc_port_counters); i++)
                data[o++] = enetc_port_rd(hw, enetc_port_counters[i].reg);
+
+       if (!(priv->active_offloads & ENETC_F_QBU))
+               return;
+
+       for (i = 0; i < ARRAY_SIZE(enetc_pmac_counters); i++)
+               data[o++] = enetc_port_rd(hw, enetc_pmac_counters[i].reg);
 }
 
 #define ENETC_RSSHASH_L3 (RXH_L2DA | RXH_VLAN | RXH_L3_PROTO | RXH_IP_SRC | \
@@ -609,6 +649,90 @@ static int enetc_set_wol(struct net_device *dev,
        return ret;
 }
 
+static u8 enetc_get_tc_num(struct enetc_si *si)
+{
+       struct net_device *ndev = si->ndev;
+       u8 tc_num;
+
+       tc_num = (enetc_port_rd(&si->hw, ENETC_PCAPR1)
+                 & ENETC_NUM_TCS_MASK) >> 4;
+
+       return min(netdev_get_num_tc(ndev), tc_num + 1);
+}
+
+int enetc_preemption_set(struct net_device *ndev, u32 ptvector)
+{
+       struct enetc_ndev_priv *priv = netdev_priv(ndev);
+       u8 tc_num;
+       u32 temp;
+       int i;
+
+       if (ptvector & ~ENETC_QBU_TC_MASK)
+               return -EINVAL;
+
+       temp = enetc_rd(&priv->si->hw, ENETC_QBV_PTGCR_OFFSET);
+       if (temp & ENETC_QBV_TGE)
+               enetc_wr(&priv->si->hw, ENETC_QBV_PTGCR_OFFSET,
+                        temp & (~ENETC_QBV_TGPE));
+
+       tc_num = enetc_get_tc_num(priv->si);
+
+       for (i = 0; i < tc_num; i++) {
+               temp = enetc_port_rd(&priv->si->hw, ENETC_PTCFPR(i));
+
+               if ((ptvector >> i) & 0x1)
+                       enetc_port_wr(&priv->si->hw,
+                                     ENETC_PTCFPR(i),
+                                     temp | ENETC_FPE);
+               else
+                       enetc_port_wr(&priv->si->hw,
+                                     ENETC_PTCFPR(i),
+                                     temp & ~ENETC_FPE);
+       }
+
+       return 0;
+}
+
+static u32 enetc_preemption_get(struct net_device *ndev)
+{
+       struct enetc_ndev_priv *priv = netdev_priv(ndev);
+       u32 ptvector = 0;
+       u8 tc_num;
+       int i;
+
+       /* If preemptable MAC is not enable return 0 */
+       if (!(enetc_port_rd(&priv->si->hw, ENETC_PFPMR) & ENETC_PFPMR_PMACE))
+               return 0;
+
+       tc_num = enetc_get_tc_num(priv->si);
+
+       for (i = 0; i < tc_num; i++)
+               if (enetc_port_rd(&priv->si->hw, ENETC_PTCFPR(i)) & ENETC_FPE)
+                       ptvector |= 1 << i;
+
+       return ptvector;
+}
+
+static int enetc_get_link_ksettings(struct net_device *ndev,
+                                   struct ethtool_link_ksettings *cmd)
+{
+       cmd->base.preemption = enetc_preemption_get(ndev);
+
+       return phy_ethtool_get_link_ksettings(ndev, cmd);
+}
+
+static int enetc_set_link_ksettings(struct net_device *ndev,
+                                   const struct ethtool_link_ksettings *cmd)
+{
+       int err;
+
+       err = enetc_preemption_set(ndev, cmd->base.preemption);
+       if (err)
+               return err;
+
+       return phy_ethtool_set_link_ksettings(ndev, cmd);
+}
+
 static const struct ethtool_ops enetc_pf_ethtool_ops = {
        .get_regs_len = enetc_get_reglen,
        .get_regs = enetc_get_regs,
@@ -622,8 +746,8 @@ static const struct ethtool_ops enetc_pf_ethtool_ops = {
        .get_rxfh = enetc_get_rxfh,
        .set_rxfh = enetc_set_rxfh,
        .get_ringparam = enetc_get_ringparam,
-       .get_link_ksettings = phy_ethtool_get_link_ksettings,
-       .set_link_ksettings = phy_ethtool_set_link_ksettings,
+       .get_link_ksettings = enetc_get_link_ksettings,
+       .set_link_ksettings = enetc_set_link_ksettings,
        .get_link = ethtool_op_get_link,
        .get_ts_info = enetc_get_ts_info,
        .get_wol = enetc_get_wol,
drivers/net/ethernet/freescale/enetc/enetc_hw.h +17 -0
index 51f543ef37a8..b609ec095710 100644
@@ -19,6 +19,7 @@
 #define ENETC_SICTR1   0x1c
 #define ENETC_SIPCAPR0 0x20
 #define ENETC_SIPCAPR0_QBV     BIT(4)
+#define ENETC_SIPCAPR0_QBU     BIT(3)
 #define ENETC_SIPCAPR0_RSS     BIT(8)
 #define ENETC_SIPCAPR1 0x24
 #define ENETC_SITGTGR  0x30
@@ -176,6 +177,7 @@ enum enetc_bdr_type {TX, RX};
 #define ENETC_PCAPR0_RXBDR(val)        ((val) >> 24)
 #define ENETC_PCAPR0_TXBDR(val)        (((val) >> 16) & 0xff)
 #define ENETC_PCAPR1           0x0904
+#define ENETC_NUM_TCS_MASK     GENMASK(6, 4)
 #define ENETC_PSICFGR0(n)      (0x0940 + (n) * 0xc)  /* n = SI index */
 #define ENETC_PSICFGR0_SET_TXBDR(val)  ((val) & 0xff)
 #define ENETC_PSICFGR0_SET_RXBDR(val)  (((val) & 0xff) << 16)
@@ -223,6 +225,7 @@ enum enetc_bdr_type {TX, RX};
 #define ENETC_SET_TX_MTU(val)  ((val) << 16)
 #define ENETC_SET_MAXFRM(val)  ((val) & 0xffff)
 #define ENETC_PM0_IF_MODE      0x8300
+#define ENETC_PM1_IF_MODE       0x9300
 #define ENETC_PMO_IFM_RG       BIT(2)
 #define ENETC_PM0_IFM_RLP      (BIT(5) | BIT(11))
 #define ENETC_PM0_IFM_RGAUTO   (BIT(15) | ENETC_PMO_IFM_RG | BIT(1))
@@ -276,6 +279,15 @@ enum enetc_bdr_type {TX, RX};
 #define ENETC_PM0_TSCOL                0x82E0
 #define ENETC_PM0_TLCOL                0x82E8
 #define ENETC_PM0_TECOL                0x82F0
+#define ENETC_PM1_RFRM         0x9120
+#define ENETC_PM1_RDRP         0x9158
+#define ENETC_PM1_RPKT         0x9160
+#define ENETC_PM1_RFRG         0x91B8
+#define ENETC_PM1_TFRM         0x9220
+#define ENETC_PM1_TERR         0x9238
+#define ENETC_PM1_TPKT         0x9260
+#define ENETC_MAC_MERGE_MMFCRXR        0x1f14
+#define ENETC_MAC_MERGE_MMFCTXR        0x1f18
 
 /* Port counters */
 #define ENETC_PICDR(n)         (0x0700 + (n) * 8) /* n = [0..3] */
@@ -615,3 +627,8 @@ struct enetc_cbd {
 /* Port time gating capability register */
 #define ENETC_QBV_PTGCAPR_OFFSET       0x11a08
 #define ENETC_QBV_MAX_GCL_LEN_MASK     GENMASK(15, 0)
+
+#define ENETC_QBU_TC_MASK      GENMASK(7, 0)
+
+#define ENETC_PTCFPR(n)         (0x1910 + (n) * 4)
+#define ENETC_FPE               BIT(31)
drivers/net/ethernet/freescale/enetc/enetc_pf.c +13 -2
index e7482d483b28..f1873c4da77f 100644
@@ -523,10 +523,15 @@ static void enetc_configure_port_mac(struct enetc_hw *hw)
                      ENETC_PM0_CMD_TXP | ENETC_PM0_PROMISC |
                      ENETC_PM0_TX_EN | ENETC_PM0_RX_EN);
        /* set auto-speed for RGMII */
-       if (enetc_port_rd(hw, ENETC_PM0_IF_MODE) & ENETC_PMO_IFM_RG)
+       if (enetc_port_rd(hw, ENETC_PM0_IF_MODE) & ENETC_PMO_IFM_RG) {
                enetc_port_wr(hw, ENETC_PM0_IF_MODE, ENETC_PM0_IFM_RGAUTO);
-       if (enetc_global_rd(hw, ENETC_G_EPFBLPR(1)) == ENETC_G_EPFBLPR1_XGMII)
+               enetc_port_wr(hw, ENETC_PM1_IF_MODE, ENETC_PM0_IFM_RGAUTO);
+       }
+
+       if (enetc_global_rd(hw, ENETC_G_EPFBLPR(1)) == ENETC_G_EPFBLPR1_XGMII) {
                enetc_port_wr(hw, ENETC_PM0_IF_MODE, ENETC_PM0_IFM_XGMII);
+               enetc_port_wr(hw, ENETC_PM1_IF_MODE, ENETC_PM0_IFM_XGMII);
+       }
 }
 
 static void enetc_configure_port_pmac(struct enetc_hw *hw)
@@ -745,6 +750,12 @@ static void enetc_pf_netdev_setup(struct enetc_si *si, struct net_device *ndev,
        if (si->hw_features & ENETC_SI_F_QBV)
                priv->active_offloads |= ENETC_F_QBV;
 
+       if (si->hw_features & ENETC_SI_F_QBU) {
+               ndev->hw_features |= NETIF_F_PREEMPTION;
+               ndev->features |= NETIF_F_PREEMPTION;
+               priv->active_offloads |= ENETC_F_QBU;
+       }
+
        /* pick up primary MAC address from SI */
        enetc_get_primary_mac_addr(&si->hw, ndev->dev_addr);
 }
drivers/net/ethernet/freescale/enetc/enetc_qos.c +4 -1
index 2e99438cb1bf..94dde847d052 100644
@@ -169,6 +169,10 @@ int enetc_setup_tc_taprio(struct net_device *ndev, void *type_data)
                                           priv->tx_ring[i]->index,
                                           taprio->enable ? 0 : i);
 
+       /* preemption off if TC priority is all 0 */
+       if ((err && taprio->enable) || !(err || taprio->enable))
+               enetc_preemption_set(ndev, 0);
+
        return err;
 }
 
--
Quoted 2 lines

2.17.1

Thanks,
-Vladimir

#3 lore raw

Re: [v1,net-next, 2/2] enetc: implement the enetc 802.1Qbu hardware function

In reply to Po Liu

On Wed, Nov 27, 2019 at 09:59:30AM +0000, Po Liu wrote:

Hi, Po Liu

Quoted 27 lines

The interface follow up the ethtool 'preemption' set/get. Hardware features also need to set hw_features with NETIF_F_PREEMPTION flag. So ethtool could check kernel link features if there is preemption capability of port.

There are two MACs in ENETC. One is express MAC which traffic classes in it are advanced transmition. Another is preemptable MAC which traffic classes are frame preemptable.

The hardware need to initialize the MACs at initial stage. And then set the preemption enable registers of traffic classes when ethtool set .get_link_ksettings/.set_link_ksettings stage.

To test the ENETC preemption capability, user need to set mqprio or taprio to mapping the traffic classes with priorities. Then use ethtool command to set 'preemption' with a 8 bits value. MSB represent high number traffic class.

---

drivers/net/ethernet/freescale/enetc/enetc.c  |   3 +
drivers/net/ethernet/freescale/enetc/enetc.h  |   4 +
.../ethernet/freescale/enetc/enetc_ethtool.c  | 142 ++++++++++++++++--
.../net/ethernet/freescale/enetc/enetc_hw.h   |  17 +++
.../net/ethernet/freescale/enetc/enetc_pf.c   |  15 +-
.../net/ethernet/freescale/enetc/enetc_qos.c  |   4 +

[...]

Quoted 7 lines
drivers/net/ethernet/freescale/enetc/enetc_ethtool.c +7 -0
+
+static u32 enetc_preemption_get(struct net_device *ndev)
+{
+	struct enetc_ndev_priv *priv = netdev_priv(ndev);
+	u32 ptvector = 0;
+	u8 tc_num;
+	int i;

drivers/net/ethernet/freescale/enetc/enetc_ethtool.c static int enetc_set_wol(struct net_device *dev, · line 711

Quoted 10 lines
drivers/net/ethernet/freescale/enetc/enetc_ethtool.c +10 -0
+
+	/* If preemptable MAC is not enable return 0 */
+	if (!(enetc_port_rd(&priv->si->hw, ENETC_PFPMR) & ENETC_PFPMR_PMACE))
+		return 0;
+
+	tc_num = enetc_get_tc_num(priv->si);
+
+	for (i = 0; i < tc_num; i++)
+		if (enetc_port_rd(&priv->si->hw, ENETC_PTCFPR(i)) & ENETC_FPE)
+			ptvector |= 1 << i;

Would be better to replace above on just priv var?

Quoted 10 lines
drivers/net/ethernet/freescale/enetc/enetc_ethtool.c +10 -0
+
+	return ptvector;
+}
+
+static int enetc_get_link_ksettings(struct net_device *ndev,
+				    struct ethtool_link_ksettings *cmd)
+{
+	cmd->base.preemption = enetc_preemption_get(ndev);
+
+	return phy_ethtool_get_link_ksettings(ndev, cmd);

drivers/net/ethernet/freescale/enetc/enetc_ethtool.c static int enetc_set_wol(struct net_device *dev, · line 731

Quoted 10 lines
drivers/net/ethernet/freescale/enetc/enetc_ethtool.c +10 -0
+}
+
+static int enetc_set_link_ksettings(struct net_device *ndev,
+				    const struct ethtool_link_ksettings *cmd)
+{
+	int err;
+
+	err = enetc_preemption_set(ndev, cmd->base.preemption);
+	if (err)
+		return err;

Shouldn't it be after
phy_ethtool_set_link_ksettings() ?

I mean after potential phy restart, does it have impact on it?

Quoted 124 lines
drivers/net/ethernet/freescale/enetc/enetc_ethtool.c +4 -0
+
+	return phy_ethtool_set_link_ksettings(ndev, cmd);
+}
+

static const struct ethtool_ops enetc_pf_ethtool_ops = {
.get_regs_len = enetc_get_reglen, .get_regs = enetc_get_regs,

drivers/net/ethernet/freescale/enetc/enetc_ethtool.c +2 -2
@@ -622,8 +746,8 @@ static const struct ethtool_ops enetc_pf_ethtool_ops = {
 	.get_rxfh = enetc_get_rxfh,
 	.set_rxfh = enetc_set_rxfh,
 	.get_ringparam = enetc_get_ringparam,
-	.get_link_ksettings = phy_ethtool_get_link_ksettings,
-	.set_link_ksettings = phy_ethtool_set_link_ksettings,
+	.get_link_ksettings = enetc_get_link_ksettings,
+	.set_link_ksettings = enetc_set_link_ksettings,
 	.get_link = ethtool_op_get_link,
 	.get_ts_info = enetc_get_ts_info,
 	.get_wol = enetc_get_wol,
drivers/net/ethernet/freescale/enetc/enetc_hw.h +17 -0
index 51f543ef37a8..b609ec095710 100644
@@ -19,6 +19,7 @@
 #define ENETC_SICTR1	0x1c
 #define ENETC_SIPCAPR0	0x20
 #define ENETC_SIPCAPR0_QBV	BIT(4)
+#define ENETC_SIPCAPR0_QBU	BIT(3)
 #define ENETC_SIPCAPR0_RSS	BIT(8)
 #define ENETC_SIPCAPR1	0x24
 #define ENETC_SITGTGR	0x30
@@ -176,6 +177,7 @@ enum enetc_bdr_type {TX, RX};
 #define ENETC_PCAPR0_RXBDR(val)	((val) >> 24)
 #define ENETC_PCAPR0_TXBDR(val)	(((val) >> 16) & 0xff)
 #define ENETC_PCAPR1		0x0904
+#define ENETC_NUM_TCS_MASK	GENMASK(6, 4)
 #define ENETC_PSICFGR0(n)	(0x0940 + (n) * 0xc)  /* n = SI index */
 #define ENETC_PSICFGR0_SET_TXBDR(val)	((val) & 0xff)
 #define ENETC_PSICFGR0_SET_RXBDR(val)	(((val) & 0xff) << 16)
@@ -223,6 +225,7 @@ enum enetc_bdr_type {TX, RX};
 #define ENETC_SET_TX_MTU(val)	((val) << 16)
 #define ENETC_SET_MAXFRM(val)	((val) & 0xffff)
 #define ENETC_PM0_IF_MODE	0x8300
+#define ENETC_PM1_IF_MODE       0x9300
 #define ENETC_PMO_IFM_RG	BIT(2)
 #define ENETC_PM0_IFM_RLP	(BIT(5) | BIT(11))
 #define ENETC_PM0_IFM_RGAUTO	(BIT(15) | ENETC_PMO_IFM_RG | BIT(1))
@@ -276,6 +279,15 @@ enum enetc_bdr_type {TX, RX};
 #define ENETC_PM0_TSCOL		0x82E0
 #define ENETC_PM0_TLCOL		0x82E8
 #define ENETC_PM0_TECOL		0x82F0
+#define ENETC_PM1_RFRM		0x9120
+#define ENETC_PM1_RDRP		0x9158
+#define ENETC_PM1_RPKT		0x9160
+#define ENETC_PM1_RFRG		0x91B8
+#define ENETC_PM1_TFRM		0x9220
+#define ENETC_PM1_TERR		0x9238
+#define ENETC_PM1_TPKT		0x9260
+#define ENETC_MAC_MERGE_MMFCRXR	0x1f14
+#define ENETC_MAC_MERGE_MMFCTXR	0x1f18
 
 /* Port counters */
 #define ENETC_PICDR(n)		(0x0700 + (n) * 8) /* n = [0..3] */
@@ -615,3 +627,8 @@ struct enetc_cbd {
 /* Port time gating capability register */
 #define ENETC_QBV_PTGCAPR_OFFSET	0x11a08
 #define ENETC_QBV_MAX_GCL_LEN_MASK	GENMASK(15, 0)
+
+#define ENETC_QBU_TC_MASK	GENMASK(7, 0)
+
+#define ENETC_PTCFPR(n)         (0x1910 + (n) * 4)
+#define ENETC_FPE               BIT(31)
drivers/net/ethernet/freescale/enetc/enetc_pf.c +13 -2
index e7482d483b28..f1873c4da77f 100644
@@ -523,10 +523,15 @@ static void enetc_configure_port_mac(struct enetc_hw *hw)
 		      ENETC_PM0_CMD_TXP	| ENETC_PM0_PROMISC |
 		      ENETC_PM0_TX_EN | ENETC_PM0_RX_EN);
 	/* set auto-speed for RGMII */
-	if (enetc_port_rd(hw, ENETC_PM0_IF_MODE) & ENETC_PMO_IFM_RG)
+	if (enetc_port_rd(hw, ENETC_PM0_IF_MODE) & ENETC_PMO_IFM_RG) {
 		enetc_port_wr(hw, ENETC_PM0_IF_MODE, ENETC_PM0_IFM_RGAUTO);
-	if (enetc_global_rd(hw, ENETC_G_EPFBLPR(1)) == ENETC_G_EPFBLPR1_XGMII)
+		enetc_port_wr(hw, ENETC_PM1_IF_MODE, ENETC_PM0_IFM_RGAUTO);
+	}
+
+	if (enetc_global_rd(hw, ENETC_G_EPFBLPR(1)) == ENETC_G_EPFBLPR1_XGMII) {
 		enetc_port_wr(hw, ENETC_PM0_IF_MODE, ENETC_PM0_IFM_XGMII);
+		enetc_port_wr(hw, ENETC_PM1_IF_MODE, ENETC_PM0_IFM_XGMII);
+	}
 }
 
 static void enetc_configure_port_pmac(struct enetc_hw *hw)
@@ -745,6 +750,12 @@ static void enetc_pf_netdev_setup(struct enetc_si *si, struct net_device *ndev,
 	if (si->hw_features & ENETC_SI_F_QBV)
 		priv->active_offloads |= ENETC_F_QBV;
 
+	if (si->hw_features & ENETC_SI_F_QBU) {
+		ndev->hw_features |= NETIF_F_PREEMPTION;
+		ndev->features |= NETIF_F_PREEMPTION;
+		priv->active_offloads |= ENETC_F_QBU;
+	}
+
 	/* pick up primary MAC address from SI */
 	enetc_get_primary_mac_addr(&si->hw, ndev->dev_addr);
 }
drivers/net/ethernet/freescale/enetc/enetc_qos.c +4 -0
index 2e99438cb1bf..94dde847d052 100644
@@ -169,6 +169,10 @@ int enetc_setup_tc_taprio(struct net_device *ndev, void *type_data)
 					   priv->tx_ring[i]->index,
 					   taprio->enable ? 0 : i);
 
+	/* preemption off if TC priority is all 0 */
+	if ((err && taprio->enable) || !(err || taprio->enable))
+		enetc_preemption_set(ndev, 0);
+
 	return err;
 }
 
Signature · 1 line
2.17.1
Signature · 2 lines
Regards,
Ivan Khoronzhuk
#5 lore raw
In reply to Po Liu

On Wed, Nov 27, 2019 at 09:59:18AM +0000, Po Liu wrote:

Hi, Po Liu

Quoted 75 lines

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

---

include/linux/netdev_features.h | 5 ++++-
include/uapi/linux/ethtool.h    | 5 ++++-
net/core/ethtool.c              | 1 +
3 files changed, 9 insertions(+), 2 deletions(-)
include/linux/netdev_features.h +4 -1
index 4b19c544c59a..299750a8b414 100644
@@ -80,6 +80,7 @@ enum {
 
 	NETIF_F_GRO_HW_BIT,		/* Hardware Generic receive offload */
 	NETIF_F_HW_TLS_RECORD_BIT,	/* Offload TLS record */
+	NETIF_F_HW_PREEMPTION_BIT,	/* Hardware TC frame preemption */
 
 	/*
 	 * Add your fresh new feature above and remember to update
@@ -150,6 +151,7 @@ enum {
 #define NETIF_F_GSO_UDP_L4	__NETIF_F(GSO_UDP_L4)
 #define NETIF_F_HW_TLS_TX	__NETIF_F(HW_TLS_TX)
 #define NETIF_F_HW_TLS_RX	__NETIF_F(HW_TLS_RX)
+#define NETIF_F_PREEMPTION	__NETIF_F(HW_PREEMPTION)
 
 /* Finds the next feature with the highest number of the range of start till 0.
 */
@@ -175,7 +177,8 @@ static inline int find_next_netdev_feature(u64 feature, unsigned long start)
 /* Features valid for ethtool to change */
 /* = all defined minus driver/device-class-related */
 #define NETIF_F_NEVER_CHANGE	(NETIF_F_VLAN_CHALLENGED | \
-				 NETIF_F_LLTX | NETIF_F_NETNS_LOCAL)
+				 NETIF_F_LLTX | NETIF_F_NETNS_LOCAL | \
+				 NETIF_F_PREEMPTION)
 
 /* remember that ((t)1 << t_BITS) is undefined in C99 */
 #define NETIF_F_ETHTOOL_BITS	((__NETIF_F_BIT(NETDEV_FEATURE_COUNT - 1) | \
include/uapi/linux/ethtool.h +2 -0
index d4591792f0b4..12ffb34afbfa 100644
@@ -1776,6 +1776,8 @@ enum ethtool_reset_flags {
 };
 #define ETH_RESET_SHARED_SHIFT	16
 
+/* Disable preemtion. */
+#define PREEMPTION_DISABLE     0x0
 
 /**
 * struct ethtool_link_settings - link control and status

include/uapi/linux/ethtool.h struct ethtool_link_settings { · line 1891

Quoted 5 lines
include/uapi/linux/ethtool.h +1 -1
@@ -1886,7 +1888,8 @@ struct ethtool_link_settings {
 	__s8	link_mode_masks_nwords;
 	__u8	transceiver;
 	__u8	reserved1[3];
-	__u32	reserved[7];
+	__u32	preemption;

Why 32 when only 8 is needed?

net/core/ethtool.c static const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] · line 114

Quoted 13 lines
include/uapi/linux/ethtool.h +1 -0
+	__u32	reserved[6];
 	__u32	link_mode_masks[0];
 	/* layout of link_mode_masks fields:
 	 * __u32 map_supported[link_mode_masks_nwords];
net/core/ethtool.c +1 -0
index cd9bc67381b2..6ffcd8a602b8 100644
@@ -111,6 +111,7 @@ static const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN]
 	[NETIF_F_HW_TLS_RECORD_BIT] =	"tls-hw-record",
 	[NETIF_F_HW_TLS_TX_BIT] =	 "tls-hw-tx-offload",
 	[NETIF_F_HW_TLS_RX_BIT] =	 "tls-hw-rx-offload",
+	[NETIF_F_HW_PREEMPTION_BIT] =	 "tx-preemption",

What about tx-frame-preempt? or frame-preemption?

Quoted 6 lines
net/core/ethtool.c +0 -0
 };
 

static const char

Signature · 1 line
2.17.1
Signature · 2 lines
Regards,
Ivan Khoronzhuk
#6 lore raw
In reply to Po Liu

Hi Po,

Quoting Po Liu (2019-11-27 01:59:18)

Quoted 18 lines

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Quoted 5 lines

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

In an early RFC series [1], we proposed a way to support frame preemption. I'm not sure if you have considered it before implementing this other proposal based on ethtool interface so I thought it would be a good idea to bring that up to your attention, just in case.

In that initial proposal, Frame Preemption feature is configured via taprio qdisc. For example:

$ tc qdisc add dev IFACE parent root handle 100 taprio \
      num_tc 3 \
      map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
      queues 1@0 1@1 2@2 \
      preemption 0 1 1 1 \
      base-time 10000000 \
      sched-entry S 01 300000 \
      sched-entry S 02 300000 \
      sched-entry S 04 400000 \
      clockid CLOCK_TAI

It also aligns with the gate control operations Set-And-Hold-MAC and Set-And-Release-MAC that can be set via 'sched-entry' (see Table 8.7 from 802.1Q-2018 for further details.

Please share your thoughts on this.

Regards,

Andre

author
#7 lore raw

Hi Andre,

Br,
Po Liu

Quoted 44 lines

-----Original Message-----
From: Andre Guedes <[email protected]>
Sent: 2019年12月11日 10:53
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Po Liu <[email protected]>
Cc: [email protected]; [email protected]; Claudiu Manoil <[email protected]>; Vladimir Oltean <[email protected]>; Alexandru Marginean <[email protected]>; Xiaoliang Yang <[email protected]>; Roy Zang <[email protected]>; Mingkai Hu <[email protected]>; Jerry Huang <[email protected]>; Leo Li <[email protected]>; Po Liu <[email protected]>
Subject: [EXT] Re: [v1,net-next, 1/2] ethtool: add setting frame preemption of traffic classes

Caution: EXT Email

Hi Po,

Quoting Po Liu (2019-11-27 01:59:18)

Quoted 21 lines

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Quoted 7 lines
Quoted 2 lines

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

In an early RFC series [1], we proposed a way to support frame preemption. I'm not sure if you have considered it before implementing this other proposal based on ethtool interface so I thought it would be a good idea to bring that up to your attention, just in case.

Sorry, I didn't notice the RFC proposal. Using ethtool set the preemption just thinking about 8021Qbu as standalone. And not limit to the taprio if user won't set 802.1Qbv.

As some feedback also want to set the MAC merge minimal fragment size and get some more information of 802.3br.

Quoted 15 lines

In that initial proposal, Frame Preemption feature is configured via taprio qdisc. For example:

$ tc qdisc add dev IFACE parent root handle 100 taprio \
      num_tc 3 \
      map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
      queues 1@0 1@1 2@2 \
      preemption 0 1 1 1 \
      base-time 10000000 \
      sched-entry S 01 300000 \
      sched-entry S 02 300000 \
      sched-entry S 04 400000 \
      clockid CLOCK_TAI
Quoted 3 lines

It also aligns with the gate control operations Set-And-Hold-MAC and Set-And- Release-MAC that can be set via 'sched-entry' (see Table 8.7 from 802.1Q-2018 for further details.

I am curious about Set-And-Hold-Mac via 'sched-entry'. Actually, it could be understand as guardband by hardware preemption. MAC should auto calculate the nano seconds before express entry slot start to break to two fragments. Set-And-Hold-MAC should minimal larger than the fragment-size oct times.

Quoted 2 lines

Please share your thoughts on this.

I am good to see there is frame preemption proposal. Each way is ok for me but ethtool is more flexible. I've seen the RFC the code. The hardware offload is in the mainline, but preemption is not yet, I don't know why. Could you post it again?

Quoted 3 lines

Regards,

Andre

#8 lore raw

Hi Po,

Quoted 46 lines · Po Liu

Hi Andre,

Br,
Po Liu

Quoted 40 lines

-----Original Message-----
From: Andre Guedes <[email protected]>
Sent: 2019年12月11日 10:53
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Po Liu <[email protected]>
Cc: [email protected]; [email protected]; Claudiu Manoil <[email protected]>; Vladimir Oltean <[email protected]>; Alexandru Marginean <[email protected]>; Xiaoliang Yang <[email protected]>; Roy Zang <[email protected]>; Mingkai Hu <[email protected]>; Jerry Huang <[email protected]>; Leo Li <[email protected]>; Po Liu <[email protected]>
Subject: [EXT] Re: [v1,net-next, 1/2] ethtool: add setting frame preemption of traffic classes

Caution: EXT Email

Hi Po,

Quoting Po Liu (2019-11-27 01:59:18)

Quoted 17 lines

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Having some knobs in ethtool to enable when/how Frame Preemption is advertised on the wire makes sense. I also agree that it should be "on" by default.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

Quoted 8 lines
Quoted 4 lines

In an early RFC series [1], we proposed a way to support frame preemption. I'm not sure if you have considered it before implementing this other proposal based on ethtool interface so I thought it would be a good idea to bring that up to your attention, just in case.

Sorry, I didn't notice the RFC proposal. Using ethtool set the preemption just thinking about 8021Qbu as standalone. And not limit to the taprio if user won't set 802.1Qbv.

I see your point of using frame-preemption "standalone", I have two ideas:

1. add support in taprio to be configured without any schedule in the "full offload" mode. In practice, allowing taprio to work somewhat similar to (mqprio + frame-preemption), changes in the code should de fairly small;

2. extend mqprio to support frame-preemption;

Quoted 3 lines

As some feedback also want to set the MAC merge minimal fragment size and get some more information of 802.3br.

The minimal fragment size, I guess, also makes sense to be kept in ethtool. That is we have a sane default, and allow the user to change this setting for special cases.

Quoted 20 lines
Quoted 16 lines

In that initial proposal, Frame Preemption feature is configured via taprio qdisc. For example:

$ tc qdisc add dev IFACE parent root handle 100 taprio \
      num_tc 3 \
      map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
      queues 1@0 1@1 2@2 \
      preemption 0 1 1 1 \
      base-time 10000000 \
      sched-entry S 01 300000 \
      sched-entry S 02 300000 \
      sched-entry S 04 400000 \
      clockid CLOCK_TAI
Quoted 3 lines

It also aligns with the gate control operations Set-And-Hold-MAC and Set-And- Release-MAC that can be set via 'sched-entry' (see Table 8.7 from 802.1Q-2018 for further details.

Quoted 5 lines

I am curious about Set-And-Hold-Mac via 'sched-entry'. Actually, it could be understand as guardband by hardware preemption. MAC should auto calculate the nano seconds before express entry slot start to break to two fragments. Set-And-Hold-MAC should minimal larger than the fragment-size oct times.

Another interesting point. My first idea is that when the schedule is offloaded to the driver and the driver detects that the "entry" width is smaller than the fragment side, the driver could reject that schedule with a nice error message.

Quoted 8 lines
Quoted 3 lines

Please share your thoughts on this.

I am good to see there is frame preemption proposal. Each way is ok for me but ethtool is more flexible. I've seen the RFC the code. The hardware offload is in the mainline, but preemption is not yet, I don't know why. Could you post it again?

It's not mainline because this kind of stuff will not be accepted upstream without in-tree users. And you are the first one to propose such a thing :-)

It's just now that I have something that supports frame-preemption, the code I have is approaching RFC-like quality. I will send another RFC this week hopefully, and we can see how things look in practice.

Cheers,
--
Vinicius

#9 lore raw
Quoted 71 lines · Vinicius Costa Gomes

Hi Po,

Quoted 46 lines · Po Liu

Hi Andre,

Br,
Po Liu

Quoted 40 lines

-----Original Message-----
From: Andre Guedes <[email protected]>
Sent: 2019年12月11日 10:53
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Po Liu <[email protected]>
Cc: [email protected]; [email protected]; Claudiu Manoil <[email protected]>; Vladimir Oltean <[email protected]>; Alexandru Marginean <[email protected]>; Xiaoliang Yang <[email protected]>; Roy Zang <[email protected]>; Mingkai Hu <[email protected]>; Jerry Huang <[email protected]>; Leo Li <[email protected]>; Po Liu <[email protected]>
Subject: [EXT] Re: [v1,net-next, 1/2] ethtool: add setting frame preemption of traffic classes

Caution: EXT Email

Hi Po,

Quoting Po Liu (2019-11-27 01:59:18)

Quoted 17 lines

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Having some knobs in ethtool to enable when/how Frame Preemption is advertised on the wire makes sense. I also agree that it should be "on" by default.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

Quoted 8 lines
Quoted 4 lines

In an early RFC series [1], we proposed a way to support frame preemption. I'm not sure if you have considered it before implementing this other proposal based on ethtool interface so I thought it would be a good idea to bring that up to your attention, just in case.

Sorry, I didn't notice the RFC proposal. Using ethtool set the preemption just thinking about 8021Qbu as standalone. And not limit to the taprio if user won't set 802.1Qbv.

Quoted 7 lines

I see your point of using frame-preemption "standalone", I have two ideas:

1. add support in taprio to be configured without any schedule in the "full offload" mode. In practice, allowing taprio to work somewhat similar to (mqprio + frame-preemption), changes in the code should de fairly small;

+

And if follow mqprio settings logic then preemption also can be enabled immediately while configuring taprio first time, and similarly new ADMIN can't change it and can be set w/o preemption option afterwards.

So that following is correct:

OPER

$ tc qdisc add dev IFACE parent root handle 100 taprio \
     base-time 10000000 \
     num_tc 3 \
     map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
     queues 1@0 1@1 2@2 \
     preemption 0 1 1 1
     flags 1

then
ADMIN

$ tc qdisc add dev IFACE parent root handle 100 taprio \
     base-time 12000000 \
     num_tc 3 \
     map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
     queues 1@0 1@1 2@2 \
     preemption 0 1 1 1
     sched-entry S 01 300000 \
     sched-entry S 02 300000 \
     flags 1

then
ADMIN

$ tc qdisc add dev IFACE parent root handle 100 taprio \
     base-time 13000000 \
     sched-entry S 01 300000 \
     sched-entry S 02 300000 \
     flags 1

BUT:

1) The question is only should it be in this way? I mean preemption to be enabled immediately? Also should include other parameters like fragment size.

2) What if I want to use frame preemption with another "transmission selection algorithm"? Say another one "time sensitive" - CBS? How is it going to be stacked?

In this case ethtool looks better, allowing this "MAC level" feature, to be configured separately.

Quoted 37 lines

2. extend mqprio to support frame-preemption;

Quoted 3 lines

As some feedback also want to set the MAC merge minimal fragment size and get some more information of 802.3br.

The minimal fragment size, I guess, also makes sense to be kept in ethtool. That is we have a sane default, and allow the user to change this setting for special cases.

Quoted 20 lines
Quoted 16 lines

In that initial proposal, Frame Preemption feature is configured via taprio qdisc. For example:

$ tc qdisc add dev IFACE parent root handle 100 taprio \
      num_tc 3 \
      map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
      queues 1@0 1@1 2@2 \
      preemption 0 1 1 1 \
      base-time 10000000 \
      sched-entry S 01 300000 \
      sched-entry S 02 300000 \
      sched-entry S 04 400000 \
      clockid CLOCK_TAI
Quoted 3 lines

It also aligns with the gate control operations Set-And-Hold-MAC and Set-And- Release-MAC that can be set via 'sched-entry' (see Table 8.7 from 802.1Q-2018 for further details.

Quoted 5 lines

I am curious about Set-And-Hold-Mac via 'sched-entry'. Actually, it could be understand as guardband by hardware preemption. MAC should auto calculate the nano seconds before express entry slot start to break to two fragments. Set-And-Hold-MAC should minimal larger than the fragment-size oct times.

Quoted 4 lines

Another interesting point. My first idea is that when the schedule is offloaded to the driver and the driver detects that the "entry" width is smaller than the fragment side, the driver could reject that schedule with a nice error message.

Looks ok, if entry command is RELEASE or SET only, but not HOLD, and only if it contains express queues. And if for entry is expectable to have interval shorter, the entry has to be marked as HOLD then.

But not every offload is able to support mac/hold per sched (there is no HOLD/RELEASE commands in this case). For this case seems like here can be 2 cases:

1) there is no “gate close” event for the preemptible traffic
2) there is "gate close" event for the preemptable traffic

And both can have the following impact, if assume the main reason to this guard check is to guarantee the express queue cannot be blocked while this "close to short" interval opening ofc:

If a preemption fragment is started before "express" frame, then interval should allow to complete preemption fragment and has to have enough time to insert express frame. So here situation when maximum packet size per each queue can have place.

In case of TI am65 this queue MTU is configurable per queue (for similar reasons and couple more (packet fill feature for instance)) and can be used for guard check also, but not clear where it should be. Seems like it should be done using ethtool also, but can be needed for taprio interface....

Quoted 21 lines
Quoted 8 lines
Quoted 3 lines

Please share your thoughts on this.

I am good to see there is frame preemption proposal. Each way is ok for me but ethtool is more flexible. I've seen the RFC the code. The hardware offload is in the mainline, but preemption is not yet, I don't know why. Could you post it again?

It's not mainline because this kind of stuff will not be accepted upstream without in-tree users. And you are the first one to propose such a thing :-)

It's just now that I have something that supports frame-preemption, the code I have is approaching RFC-like quality. I will send another RFC this week hopefully, and we can see how things look in practice.

Cheers,
--
Vinicius

Signature · 2 lines
Regards,
Ivan Khoronzhuk
#10 lore raw

Hi Ivan,

Quoted 84 lines · Ivan Khoronzhuk
Quoted 39 lines
Quoted 13 lines
Quoted 13 lines

Quoting Po Liu (2019-11-27 01:59:18)

Quoted 12 lines

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Having some knobs in ethtool to enable when/how Frame Preemption is advertised on the wire makes sense. I also agree that it should be "on" by default.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

Quoted 8 lines
Quoted 4 lines

In an early RFC series [1], we proposed a way to support frame preemption. I'm not sure if you have considered it before implementing this other proposal based on ethtool interface so I thought it would be a good idea to bring that up to your attention, just in case.

Sorry, I didn't notice the RFC proposal. Using ethtool set the preemption just thinking about 8021Qbu as standalone. And not limit to the taprio if user won't set 802.1Qbv.

Quoted 7 lines

I see your point of using frame-preemption "standalone", I have two ideas:

1. add support in taprio to be configured without any schedule in the "full offload" mode. In practice, allowing taprio to work somewhat similar to (mqprio + frame-preemption), changes in the code should de fairly small;

+

And if follow mqprio settings logic then preemption also can be enabled immediately while configuring taprio first time, and similarly new ADMIN can't change it and can be set w/o preemption option afterwards.

So that following is correct:

OPER

$ tc qdisc add dev IFACE parent root handle 100 taprio \
      base-time 10000000 \
      num_tc 3 \
      map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
      queues 1@0 1@1 2@2 \
      preemption 0 1 1 1
      flags 1

then
ADMIN

$ tc qdisc add dev IFACE parent root handle 100 taprio \
      base-time 12000000 \
      num_tc 3 \
      map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
      queues 1@0 1@1 2@2 \
      preemption 0 1 1 1
      sched-entry S 01 300000 \
      sched-entry S 02 300000 \
      flags 1

then
ADMIN

$ tc qdisc add dev IFACE parent root handle 100 taprio \
      base-time 13000000 \
      sched-entry S 01 300000 \
      sched-entry S 02 300000 \
      flags 1
Quoted 5 lines

BUT:

1) The question is only should it be in this way? I mean preemption to be enabled immediately? Also should include other parameters like fragment size.

We can decide what things are allowed/useful here. For example, it might make sense to allow "preemption" to be changed. We can extend taprio to support changing the fragment size, if that makes sense.

Quoted 4 lines

2) What if I want to use frame preemption with another "transmission selection algorithm"? Say another one "time sensitive" - CBS? How is it going to be stacked?

I am not seeing any (conceptual*) problems when plugging a cbs (for example) qdisc into one of taprio children. Or, are you talking about a more general problem?

* here I am considering that support for taprio without an schedule is added.

Quoted 3 lines

In this case ethtool looks better, allowing this "MAC level" feature, to be configured separately.

My only issue with using ethtool is that then we would have two different interfaces for "complementary" features. And it would make things even harder to configure and debug. The fact that one talks about traffic classes and the other transmission queues doesn't make me more comfortable as well.

On the other hand, as there isn't a way to implement frame preemption in software, I agree that it makes it kind of awkward to have it in the tc subsystem.

At this point, I am slightly in favor of the taprio approach (yes, I am biased :-), but I can be convinced otherwise. I will be only a little sad if we choose to go with ethtool for now, and then add support up in the stack, something similar to "ethtool -N" and "tc-flower".

Quoted 43 lines
Quoted 38 lines

2. extend mqprio to support frame-preemption;

Quoted 3 lines

As some feedback also want to set the MAC merge minimal fragment size and get some more information of 802.3br.

The minimal fragment size, I guess, also makes sense to be kept in ethtool. That is we have a sane default, and allow the user to change this setting for special cases.

Quoted 20 lines
Quoted 16 lines

In that initial proposal, Frame Preemption feature is configured via taprio qdisc. For example:

$ tc qdisc add dev IFACE parent root handle 100 taprio \
      num_tc 3 \
      map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
      queues 1@0 1@1 2@2 \
      preemption 0 1 1 1 \
      base-time 10000000 \
      sched-entry S 01 300000 \
      sched-entry S 02 300000 \
      sched-entry S 04 400000 \
      clockid CLOCK_TAI
Quoted 3 lines

It also aligns with the gate control operations Set-And-Hold-MAC and Set-And- Release-MAC that can be set via 'sched-entry' (see Table 8.7 from 802.1Q-2018 for further details.

Quoted 5 lines

I am curious about Set-And-Hold-Mac via 'sched-entry'. Actually, it could be understand as guardband by hardware preemption. MAC should auto calculate the nano seconds before express entry slot start to break to two fragments. Set-And-Hold-MAC should minimal larger than the fragment-size oct times.

Quoted 4 lines

Another interesting point. My first idea is that when the schedule is offloaded to the driver and the driver detects that the "entry" width is smaller than the fragment side, the driver could reject that schedule with a nice error message.

Quoted 7 lines

Looks ok, if entry command is RELEASE or SET only, but not HOLD, and only if it contains express queues. And if for entry is expectable to have interval shorter, the entry has to be marked as HOLD then.

But not every offload is able to support mac/hold per sched (there is no HOLD/RELEASE commands in this case). For this case seems like here can be 2 cases:

Yeah, the hw I have in hand also doesn't support the HOLD/RELEASE commands.

Quoted 13 lines

1) there is no “gate close” event for the preemptible traffic
2) there is "gate close" event for the preemptable traffic

And both can have the following impact, if assume the main reason to this guard check is to guarantee the express queue cannot be blocked while this "close to short" interval opening ofc:

If a preemption fragment is started before "express" frame, then interval should allow to complete preemption fragment and has to have enough time to insert express frame. So here situation when maximum packet size per each queue can have place.

Quoted 5 lines

In case of TI am65 this queue MTU is configurable per queue (for similar reasons and couple more (packet fill feature for instance)) and can be used for guard check also, but not clear where it should be. Seems like it should be done using ethtool also, but can be needed for taprio interface....

For now, at least for the hardware I am working on, something like this is configurable, but I wasn't planning on exposing it, using the maximum ethernet frame size seemed a good default.

Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
Quoted 4 lines

Please share your thoughts on this.

I am good to see there is frame preemption proposal. Each way is ok for me but ethtool is more flexible. I've seen the RFC the code. The hardware offload is in the mainline, but preemption is not yet, I don't know why. Could you post it again?

It's not mainline because this kind of stuff will not be accepted upstream without in-tree users. And you are the first one to propose such a thing :-)

Quoted 12 lines
Quoted 8 lines

It's just now that I have something that supports frame-preemption, the code I have is approaching RFC-like quality. I will send another RFC this week hopefully, and we can see how things look in practice.

Cheers,
--
Vinicius

Signature · 2 lines
Regards,
Ivan Khoronzhuk

Cheers,
--
Vinicius

#11 lore raw

Hi Vinicius,

Quoted 90 lines · Vinicius Costa Gomes

Hi Ivan,

Quoted 86 lines · Ivan Khoronzhuk
Quoted 39 lines
Quoted 13 lines
Quoted 13 lines

Quoting Po Liu (2019-11-27 01:59:18)

Quoted 12 lines

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Having some knobs in ethtool to enable when/how Frame Preemption is advertised on the wire makes sense. I also agree that it should be "on" by default.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

Quoted 8 lines
Quoted 4 lines

In an early RFC series [1], we proposed a way to support frame preemption. I'm not sure if you have considered it before implementing this other proposal based on ethtool interface so I thought it would be a good idea to bring that up to your attention, just in case.

Sorry, I didn't notice the RFC proposal. Using ethtool set the preemption just thinking about 8021Qbu as standalone. And not limit to the taprio if user won't set 802.1Qbv.

Quoted 7 lines

I see your point of using frame-preemption "standalone", I have two ideas:

1. add support in taprio to be configured without any schedule in the "full offload" mode. In practice, allowing taprio to work somewhat similar to (mqprio + frame-preemption), changes in the code should de fairly small;

+

And if follow mqprio settings logic then preemption also can be enabled immediately while configuring taprio first time, and similarly new ADMIN can't change it and can be set w/o preemption option afterwards.

So that following is correct:

OPER

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       base-time 10000000 \
       num_tc 3 \
       map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
       queues 1@0 1@1 2@2 \
       preemption 0 1 1 1
       flags 1

then
ADMIN

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       base-time 12000000 \
       num_tc 3 \
       map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
       queues 1@0 1@1 2@2 \
       preemption 0 1 1 1
       sched-entry S 01 300000 \
       sched-entry S 02 300000 \
       flags 1

then
ADMIN

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       base-time 13000000 \
       sched-entry S 01 300000 \
       sched-entry S 02 300000 \
       flags 1

BUT:

Quoted 8 lines
Quoted 3 lines

1) The question is only should it be in this way? I mean preemption to be enabled immediately? Also should include other parameters like fragment size.

We can decide what things are allowed/useful here. For example, it might make sense to allow "preemption" to be changed. We can extend taprio to support changing the fragment size, if that makes sense.

The point is it make sense to configure pre-emption related parameters independently of taprio since they are not related. User may use just pre-emption to reduce latency in the communication path for their end application.. So there should be a way to configure it independently of taprio. Right??

Quoted 9 lines
Quoted 4 lines

2) What if I want to use frame preemption with another "transmission selection algorithm"? Say another one "time sensitive" - CBS? How is it going to be stacked?

I am not seeing any (conceptual*) problems when plugging a cbs (for example) qdisc into one of taprio children. Or, are you talking about a more general problem?

If I understand it correctly problem is not stacking taprio with cbs, but rather pre-emption with other qdiscs and allow configuring the parameters such as frag size. How do I use frame pre-emption as an independent feature and configure frag size? Ethool appears to be better from this point of view as Ivan has mentioned below.

Murali

Quoted 130 lines

* here I am considering that support for taprio without an schedule is added.

Quoted 3 lines

In this case ethtool looks better, allowing this "MAC level" feature, to be configured separately.

My only issue with using ethtool is that then we would have two different interfaces for "complementary" features. And it would make things even harder to configure and debug. The fact that one talks about traffic classes and the other transmission queues doesn't make me more comfortable as well.

On the other hand, as there isn't a way to implement frame preemption in software, I agree that it makes it kind of awkward to have it in the tc subsystem.

At this point, I am slightly in favor of the taprio approach (yes, I am biased :-), but I can be convinced otherwise. I will be only a little sad if we choose to go with ethtool for now, and then add support up in the stack, something similar to "ethtool -N" and "tc-flower".

Quoted 43 lines
Quoted 38 lines

2. extend mqprio to support frame-preemption;

Quoted 3 lines

As some feedback also want to set the MAC merge minimal fragment size and get some more information of 802.3br.

The minimal fragment size, I guess, also makes sense to be kept in ethtool. That is we have a sane default, and allow the user to change this setting for special cases.

Quoted 20 lines
Quoted 16 lines

In that initial proposal, Frame Preemption feature is configured via taprio qdisc. For example:

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       num_tc 3 \
       map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
       queues 1@0 1@1 2@2 \
       preemption 0 1 1 1 \
       base-time 10000000 \
       sched-entry S 01 300000 \
       sched-entry S 02 300000 \
       sched-entry S 04 400000 \
       clockid CLOCK_TAI
Quoted 3 lines

It also aligns with the gate control operations Set-And-Hold-MAC and Set-And- Release-MAC that can be set via 'sched-entry' (see Table 8.7 from 802.1Q-2018 for further details.

Quoted 5 lines

I am curious about Set-And-Hold-Mac via 'sched-entry'. Actually, it could be understand as guardband by hardware preemption. MAC should auto calculate the nano seconds before express entry slot start to break to two fragments. Set-And-Hold-MAC should minimal larger than the fragment-size oct times.

Quoted 4 lines

Another interesting point. My first idea is that when the schedule is offloaded to the driver and the driver detects that the "entry" width is smaller than the fragment side, the driver could reject that schedule with a nice error message.

Quoted 7 lines

Looks ok, if entry command is RELEASE or SET only, but not HOLD, and only if it contains express queues. And if for entry is expectable to have interval shorter, the entry has to be marked as HOLD then.

But not every offload is able to support mac/hold per sched (there is no HOLD/RELEASE commands in this case). For this case seems like here can be 2 cases:

Yeah, the hw I have in hand also doesn't support the HOLD/RELEASE commands.

Quoted 13 lines

1) there is no “gate close” event for the preemptible traffic
2) there is "gate close" event for the preemptable traffic

And both can have the following impact, if assume the main reason to this guard check is to guarantee the express queue cannot be blocked while this "close to short" interval opening ofc:

If a preemption fragment is started before "express" frame, then interval should allow to complete preemption fragment and has to have enough time to insert express frame. So here situation when maximum packet size per each queue can have place.

Quoted 5 lines

In case of TI am65 this queue MTU is configurable per queue (for similar reasons and couple more (packet fill feature for instance)) and can be used for guard check also, but not clear where it should be. Seems like it should be done using ethtool also, but can be needed for taprio interface....

For now, at least for the hardware I am working on, something like this is configurable, but I wasn't planning on exposing it, using the maximum ethernet frame size seemed a good default.

Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
Quoted 4 lines

Please share your thoughts on this.

I am good to see there is frame preemption proposal. Each way is ok for me but ethtool is more flexible. I've seen the RFC the code. The hardware offload is in the mainline, but preemption is not yet, I don't know why. Could you post it again?

It's not mainline because this kind of stuff will not be accepted upstream without in-tree users. And you are the first one to propose such a thing :-)

Quoted 12 lines
Quoted 8 lines

It's just now that I have something that supports frame-preemption, the code I have is approaching RFC-like quality. I will send another RFC this week hopefully, and we can see how things look in practice.

Cheers,
--
Vinicius

Signature · 2 lines
Regards,
Ivan Khoronzhuk

Cheers,
--
Vinicius

Signature · 2 lines
Murali Karicheri
Texas Instruments
#12 lore raw

Hi,

Murali Karicheri <[email protected]> writes:

[...]

Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

2) What if I want to use frame preemption with another "transmission selection algorithm"? Say another one "time sensitive" - CBS? How is it going to be stacked?

I am not seeing any (conceptual*) problems when plugging a cbs (for example) qdisc into one of taprio children. Or, are you talking about a more general problem?

Quoted 6 lines

If I understand it correctly problem is not stacking taprio with cbs, but rather pre-emption with other qdiscs and allow configuring the parameters such as frag size. How do I use frame pre-emption as an independent feature and configure frag size? Ethool appears to be better from this point of view as Ivan has mentioned below.

Yeah, after all the discussion here, I think we came to the rough consensus that ethtool is the best place we have to enable/disable frame preemption and its configuration knobs.

I just have a few comments to add to the series (sorry for the long delay).

Cheers,
--
Vinicius

#13 lore raw
In reply to Vinicius Costa Gomes

Hi Vinicius,

Quoted 120 lines · Vinicius Costa Gomes

Hi Ivan,

Quoted 84 lines · Ivan Khoronzhuk
Quoted 39 lines
Quoted 13 lines
Quoted 13 lines

Quoting Po Liu (2019-11-27 01:59:18)

Quoted 12 lines

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Having some knobs in ethtool to enable when/how Frame Preemption is advertised on the wire makes sense. I also agree that it should be "on" by default.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

Quoted 8 lines
Quoted 4 lines

In an early RFC series [1], we proposed a way to support frame preemption. I'm not sure if you have considered it before implementing this other proposal based on ethtool interface so I thought it would be a good idea to bring that up to your attention, just in case.

Sorry, I didn't notice the RFC proposal. Using ethtool set the preemption just thinking about 8021Qbu as standalone. And not limit to the taprio if user won't set 802.1Qbv.

Quoted 7 lines

I see your point of using frame-preemption "standalone", I have two ideas:

1. add support in taprio to be configured without any schedule in the "full offload" mode. In practice, allowing taprio to work somewhat similar to (mqprio + frame-preemption), changes in the code should de fairly small;

+

And if follow mqprio settings logic then preemption also can be enabled immediately while configuring taprio first time, and similarly new ADMIN can't change it and can be set w/o preemption option afterwards.

So that following is correct:

OPER

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       base-time 10000000 \
       num_tc 3 \
       map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
       queues 1@0 1@1 2@2 \
       preemption 0 1 1 1
       flags 1

then
ADMIN

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       base-time 12000000 \
       num_tc 3 \
       map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
       queues 1@0 1@1 2@2 \
       preemption 0 1 1 1
       sched-entry S 01 300000 \
       sched-entry S 02 300000 \
       flags 1

then
ADMIN

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       base-time 13000000 \
       sched-entry S 01 300000 \
       sched-entry S 02 300000 \
       flags 1
Quoted 5 lines

BUT:

1) The question is only should it be in this way? I mean preemption to be enabled immediately? Also should include other parameters like fragment size.

We can decide what things are allowed/useful here. For example, it might make sense to allow "preemption" to be changed. We can extend taprio to support changing the fragment size, if that makes sense.

Quoted 4 lines

2) What if I want to use frame preemption with another "transmission selection algorithm"? Say another one "time sensitive" - CBS? How is it going to be stacked?

I am not seeing any (conceptual*) problems when plugging a cbs (for example) qdisc into one of taprio children. Or, are you talking about a more general problem?

* here I am considering that support for taprio without an schedule is added.

Quoted 3 lines

In this case ethtool looks better, allowing this "MAC level" feature, to be configured separately.

My only issue with using ethtool is that then we would have two different interfaces for "complementary" features. And it would make things even harder to configure and debug. The fact that one talks about traffic classes and the other transmission queues doesn't make me more comfortable as well.

Quoted 3 lines

On the other hand, as there isn't a way to implement frame preemption in software, I agree that it makes it kind of awkward to have it in the tc subsystem.

Absolutely. I think frame pre-emption feature flag, per queue express/ pre-empt state, frag size, timers (hold/release) to be configured independently (perhaps through ethtool) and then taprio should check this with the lower device and then allow supporting additional Gate operations such as Hold/release if supported by underlying device.

What do you think? Why to abuse tc for this?

Thanks and regards,

Murali

Quoted 115 lines

At this point, I am slightly in favor of the taprio approach (yes, I am biased :-), but I can be convinced otherwise. I will be only a little sad if we choose to go with ethtool for now, and then add support up in the stack, something similar to "ethtool -N" and "tc-flower".

Quoted 43 lines
Quoted 38 lines

2. extend mqprio to support frame-preemption;

Quoted 3 lines

As some feedback also want to set the MAC merge minimal fragment size and get some more information of 802.3br.

The minimal fragment size, I guess, also makes sense to be kept in ethtool. That is we have a sane default, and allow the user to change this setting for special cases.

Quoted 20 lines
Quoted 16 lines

In that initial proposal, Frame Preemption feature is configured via taprio qdisc. For example:

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       num_tc 3 \
       map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
       queues 1@0 1@1 2@2 \
       preemption 0 1 1 1 \
       base-time 10000000 \
       sched-entry S 01 300000 \
       sched-entry S 02 300000 \
       sched-entry S 04 400000 \
       clockid CLOCK_TAI
Quoted 3 lines

It also aligns with the gate control operations Set-And-Hold-MAC and Set-And- Release-MAC that can be set via 'sched-entry' (see Table 8.7 from 802.1Q-2018 for further details.

Quoted 5 lines

I am curious about Set-And-Hold-Mac via 'sched-entry'. Actually, it could be understand as guardband by hardware preemption. MAC should auto calculate the nano seconds before express entry slot start to break to two fragments. Set-And-Hold-MAC should minimal larger than the fragment-size oct times.

Quoted 4 lines

Another interesting point. My first idea is that when the schedule is offloaded to the driver and the driver detects that the "entry" width is smaller than the fragment side, the driver could reject that schedule with a nice error message.

Quoted 7 lines

Looks ok, if entry command is RELEASE or SET only, but not HOLD, and only if it contains express queues. And if for entry is expectable to have interval shorter, the entry has to be marked as HOLD then.

But not every offload is able to support mac/hold per sched (there is no HOLD/RELEASE commands in this case). For this case seems like here can be 2 cases:

Yeah, the hw I have in hand also doesn't support the HOLD/RELEASE commands.

Quoted 13 lines

1) there is no “gate close” event for the preemptible traffic
2) there is "gate close" event for the preemptable traffic

And both can have the following impact, if assume the main reason to this guard check is to guarantee the express queue cannot be blocked while this "close to short" interval opening ofc:

If a preemption fragment is started before "express" frame, then interval should allow to complete preemption fragment and has to have enough time to insert express frame. So here situation when maximum packet size per each queue can have place.

Quoted 5 lines

In case of TI am65 this queue MTU is configurable per queue (for similar reasons and couple more (packet fill feature for instance)) and can be used for guard check also, but not clear where it should be. Seems like it should be done using ethtool also, but can be needed for taprio interface....

For now, at least for the hardware I am working on, something like this is configurable, but I wasn't planning on exposing it, using the maximum ethernet frame size seemed a good default.

Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
Quoted 4 lines

Please share your thoughts on this.

I am good to see there is frame preemption proposal. Each way is ok for me but ethtool is more flexible. I've seen the RFC the code. The hardware offload is in the mainline, but preemption is not yet, I don't know why. Could you post it again?

It's not mainline because this kind of stuff will not be accepted upstream without in-tree users. And you are the first one to propose such a thing :-)

Quoted 12 lines
Quoted 8 lines

It's just now that I have something that supports frame-preemption, the code I have is approaching RFC-like quality. I will send another RFC this week hopefully, and we can see how things look in practice.

Cheers,
--
Vinicius

Signature · 2 lines
Regards,
Ivan Khoronzhuk

Cheers,
--
Vinicius

Signature · 2 lines
Murali Karicheri
Texas Instruments
#14 lore raw

Hi,

Quoted 77 lines
Quoted 74 lines
Quoted 42 lines
Quoted 4 lines

1. add support in taprio to be configured without any schedule in the "full offload" mode. In practice, allowing taprio to work somewhat similar to (mqprio + frame-preemption), changes in the code should de fairly small;

+

And if follow mqprio settings logic then preemption also can be enabled immediately while configuring taprio first time, and similarly new ADMIN can't change it and can be set w/o preemption option afterwards.

So that following is correct:

OPER

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       base-time 10000000 \
       num_tc 3 \
       map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
       queues 1@0 1@1 2@2 \
       preemption 0 1 1 1
       flags 1

then
ADMIN

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       base-time 12000000 \
       num_tc 3 \
       map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
       queues 1@0 1@1 2@2 \
       preemption 0 1 1 1
       sched-entry S 01 300000 \
       sched-entry S 02 300000 \
       flags 1

then
ADMIN

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       base-time 13000000 \
       sched-entry S 01 300000 \
       sched-entry S 02 300000 \
       flags 1
Quoted 5 lines

BUT:

1) The question is only should it be in this way? I mean preemption to be enabled immediately? Also should include other parameters like fragment size.

We can decide what things are allowed/useful here. For example, it might make sense to allow "preemption" to be changed. We can extend taprio to support changing the fragment size, if that makes sense.

Quoted 4 lines

2) What if I want to use frame preemption with another "transmission selection algorithm"? Say another one "time sensitive" - CBS? How is it going to be stacked?

I am not seeing any (conceptual*) problems when plugging a cbs (for example) qdisc into one of taprio children. Or, are you talking about a more general problem?

* here I am considering that support for taprio without an schedule is added.

Quoted 3 lines

In this case ethtool looks better, allowing this "MAC level" feature, to be configured separately.

My only issue with using ethtool is that then we would have two different interfaces for "complementary" features. And it would make things even harder to configure and debug. The fact that one talks about traffic classes and the other transmission queues doesn't make me more comfortable as well.

Quoted 3 lines

On the other hand, as there isn't a way to implement frame preemption in software, I agree that it makes it kind of awkward to have it in the tc subsystem.

Quoted 8 lines

Absolutely. I think frame pre-emption feature flag, per queue express/ pre-empt state, frag size, timers (hold/release) to be configured independently (perhaps through ethtool) and then taprio should check this with the lower device and then allow supporting additional Gate operations such as Hold/release if supported by underlying device.

What do you think? Why to abuse tc for this?

After reading all this great discussion and revisiting the 802.1Q and 802.3br specs, I'm now leaning towards to not coupling Frame Preemption support under taprio qdisc. Besides what have been discussed, Annex S.2 from 802.1Q-2018 foresees FP without EST so it makes me feel like we should keep them separate.

Regarding the FP configuration knobs, the following seems reasonable to me:
* Enable/disable FP feature
* Preemptable queue mapping
* Fragment size multiplier

I'm not sure about the knob 'timers (hold/release)' described in the quotes above. I couldn't find a match in the specs. If it refers to 'holdAdvance' and 'releaseAdvance' parameters described in 802.1Q-2018, I believe they are not configurable. Do we know any hardware where they are configurable?

Regards,

Andre

#15 lore raw

From: Andre Guedes <[email protected]>
Date: Jan/09/2020, 01:07:37 (UTC+00:00)

Quoted 4 lines

After reading all this great discussion and revisiting the 802.1Q and 802.3br specs, I'm now leaning towards to not coupling Frame Preemption support under taprio qdisc. Besides what have been discussed, Annex S.2 from 802.1Q-2018 foresees FP without EST so it makes me feel like we should keep them separate.

I agree that EST and FP can be used individually. But how can you
specify the hold and release commands for gates without changing taprio qdisc user space API ?

Quoted 9 lines

Regarding the FP configuration knobs, the following seems reasonable to me:
* Enable/disable FP feature
* Preemptable queue mapping
* Fragment size multiplier

I'm not sure about the knob 'timers (hold/release)' described in the quotes above. I couldn't find a match in the specs. If it refers to 'holdAdvance' and 'releaseAdvance' parameters described in 802.1Q-2018, I believe they are not configurable. Do we know any hardware where they are configurable?

Synopsys' HW supports reconfiguring these parameters. They are, however, fixed independently of Queues. i.e. all queues will have same holdAdvance / releaseAdvance.

---
Thanks,
Jose Miguel Abreu

#16 lore raw

Hi Jose,

Quoting Jose Abreu (2020-01-09 00:59:24)

Quoted 10 lines

From: Andre Guedes <[email protected]>
Date: Jan/09/2020, 01:07:37 (UTC+00:00)

Quoted 4 lines

After reading all this great discussion and revisiting the 802.1Q and 802.3br specs, I'm now leaning towards to not coupling Frame Preemption support under taprio qdisc. Besides what have been discussed, Annex S.2 from 802.1Q-2018 foresees FP without EST so it makes me feel like we should keep them separate.

I agree that EST and FP can be used individually. But how can you
specify the hold and release commands for gates without changing taprio qdisc user space API ?

The 'hold' and 'release' are operations from the GCL, which is part of EST. So they should still be specified via taprio. No changing in the user space API is required since these operations are already supported in taprio API. What is missing today is just the 'tc' side of it, which you already have a patch for it.

Quoted 5 lines
Quoted 5 lines

Regarding the FP configuration knobs, the following seems reasonable to me:
* Enable/disable FP feature
* Preemptable queue mapping
* Fragment size multiplier

Quoted 7 lines
Quoted 4 lines

I'm not sure about the knob 'timers (hold/release)' described in the quotes above. I couldn't find a match in the specs. If it refers to 'holdAdvance' and 'releaseAdvance' parameters described in 802.1Q-2018, I believe they are not configurable. Do we know any hardware where they are configurable?

Synopsys' HW supports reconfiguring these parameters. They are, however, fixed independently of Queues. i.e. all queues will have same holdAdvance / releaseAdvance.

Good to know. Is the datasheet publicly available? If so, could you please point me to it? I'd like to learn more about the FP knobs provided by different HW.

Regards,

Andre

#17 lore raw

From: Andre Guedes <[email protected]>
Date: Jan/09/2020, 18:04:55 (UTC+00:00)

Quoted 9 lines

Quoting Jose Abreu (2020-01-09 00:59:24)

Quoted 8 lines

From: Andre Guedes <[email protected]>
Date: Jan/09/2020, 01:07:37 (UTC+00:00)

Quoted 4 lines

After reading all this great discussion and revisiting the 802.1Q and 802.3br specs, I'm now leaning towards to not coupling Frame Preemption support under taprio qdisc. Besides what have been discussed, Annex S.2 from 802.1Q-2018 foresees FP without EST so it makes me feel like we should keep them separate.

Quoted 8 lines
Quoted 2 lines

I agree that EST and FP can be used individually. But how can you
specify the hold and release commands for gates without changing taprio qdisc user space API ?

The 'hold' and 'release' are operations from the GCL, which is part of EST. So they should still be specified via taprio. No changing in the user space API is required since these operations are already supported in taprio API. What is missing today is just the 'tc' side of it, which you already have a patch for it.

OK. Should we ask to merge it as-is then ?

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

Regarding the FP configuration knobs, the following seems reasonable to me:
* Enable/disable FP feature
* Preemptable queue mapping
* Fragment size multiplier

I'm not sure about the knob 'timers (hold/release)' described in the quotes above. I couldn't find a match in the specs. If it refers to 'holdAdvance' and 'releaseAdvance' parameters described in 802.1Q-2018, I believe they are not configurable. Do we know any hardware where they are configurable?

Quoted 6 lines
Quoted 2 lines

Synopsys' HW supports reconfiguring these parameters. They are, however, fixed independently of Queues. i.e. all queues will have same holdAdvance / releaseAdvance.

Good to know. Is the datasheet publicly available? If so, could you please point me to it? I'd like to learn more about the FP knobs provided by different HW.

I'm afraid its not available unless you are a Synopsys customer. You should however be able to figure out the behavior by reading my patch that adds support for FPE in XGMAC and QoS cores. I can clarify any doubts.

---
Thanks,
Jose Miguel Abreu

#18 lore raw
In reply to Andre Guedes

Hi Andre,

Quoted 99 lines · Andre Guedes

Hi,

Quoted 77 lines
Quoted 74 lines
Quoted 42 lines
Quoted 4 lines

1. add support in taprio to be configured without any schedule in the "full offload" mode. In practice, allowing taprio to work somewhat similar to (mqprio + frame-preemption), changes in the code should de fairly small;

+

And if follow mqprio settings logic then preemption also can be enabled immediately while configuring taprio first time, and similarly new ADMIN can't change it and can be set w/o preemption option afterwards.

So that following is correct:

OPER

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       base-time 10000000 \
       num_tc 3 \
       map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
       queues 1@0 1@1 2@2 \
       preemption 0 1 1 1
       flags 1

then
ADMIN

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       base-time 12000000 \
       num_tc 3 \
       map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
       queues 1@0 1@1 2@2 \
       preemption 0 1 1 1
       sched-entry S 01 300000 \
       sched-entry S 02 300000 \
       flags 1

then
ADMIN

$ tc qdisc add dev IFACE parent root handle 100 taprio \
       base-time 13000000 \
       sched-entry S 01 300000 \
       sched-entry S 02 300000 \
       flags 1
Quoted 5 lines

BUT:

1) The question is only should it be in this way? I mean preemption to be enabled immediately? Also should include other parameters like fragment size.

We can decide what things are allowed/useful here. For example, it might make sense to allow "preemption" to be changed. We can extend taprio to support changing the fragment size, if that makes sense.

Quoted 4 lines

2) What if I want to use frame preemption with another "transmission selection algorithm"? Say another one "time sensitive" - CBS? How is it going to be stacked?

I am not seeing any (conceptual*) problems when plugging a cbs (for example) qdisc into one of taprio children. Or, are you talking about a more general problem?

* here I am considering that support for taprio without an schedule is added.

Quoted 3 lines

In this case ethtool looks better, allowing this "MAC level" feature, to be configured separately.

My only issue with using ethtool is that then we would have two different interfaces for "complementary" features. And it would make things even harder to configure and debug. The fact that one talks about traffic classes and the other transmission queues doesn't make me more comfortable as well.

Quoted 3 lines

On the other hand, as there isn't a way to implement frame preemption in software, I agree that it makes it kind of awkward to have it in the tc subsystem.

Quoted 8 lines

Absolutely. I think frame pre-emption feature flag, per queue express/ pre-empt state, frag size, timers (hold/release) to be configured independently (perhaps through ethtool) and then taprio should check this with the lower device and then allow supporting additional Gate operations such as Hold/release if supported by underlying device.

What do you think? Why to abuse tc for this?

After reading all this great discussion and revisiting the 802.1Q and 802.3br specs, I'm now leaning towards to not coupling Frame Preemption support under taprio qdisc. Besides what have been discussed, Annex S.2 from 802.1Q-2018 foresees FP without EST so it makes me feel like we should keep them separate.

Regarding the FP configuration knobs, the following seems reasonable to me:
* Enable/disable FP feature
* Preemptable queue mapping
* Fragment size multiplier

Quoted 5 lines

I'm not sure about the knob 'timers (hold/release)' described in the quotes above. I couldn't find a match in the specs. If it refers to 'holdAdvance' and 'releaseAdvance' parameters described in 802.1Q-2018, I believe they are not configurable. Do we know any hardware where they are configurable?

On NXP LS1028A, HOLD_ADVANCE is configurable on both ENETC and the Felix switch (its default value is 127 bytes). Same as Synopsys, it is a global setting and not per queue or per GCL entry. RELEASE_ADVANCE is not configurable.
Regardless, I am not sure if there is any value in configuring this knob. Remember that the minimum guard band size still needs to be twice as large as the minimum Ethernet frame size.

As for the main topic of tc-taprio vs ethtool for configuring frame preemption, I think ethtool is the more natural place for configuring the traffic class to pMAC/eMAC mapping, global enable bit, and fragment size, while tc-taprio is the more natural place for configuring hold/release on individual GCL entries. The tc-taprio offload can check the netdev support of the ethtool feature, just as it checks right now the support for PTP clock for the full offload feature.

Introducing tc-taprio with a null schedule is not really natural, and frame preemption is a hardware-only feature that cannot be emulated, so it is odd to enable it through tc.

Quoted 3 lines

Regards,

Andre

Regards,
-Vladimir

#19 lore raw

Hi Vladimir,

Quoting Vladimir Oltean (2020-01-10 08:02:45)

Quoted 6 lines
Quoted 5 lines

I'm not sure about the knob 'timers (hold/release)' described in the quotes above. I couldn't find a match in the specs. If it refers to 'holdAdvance' and 'releaseAdvance' parameters described in 802.1Q-2018, I believe they are not configurable. Do we know any hardware where they are configurable?

Quoted 7 lines

On NXP LS1028A, HOLD_ADVANCE is configurable on both ENETC and the Felix switch (its default value is 127 bytes). Same as Synopsys, it is a global setting and not per queue or per GCL entry. RELEASE_ADVANCE is not configurable.
Regardless, I am not sure if there is any value in configuring this knob. Remember that the minimum guard band size still needs to be twice as large as the minimum Ethernet frame size.

Not adding this knob now sounds reasonable to me too, but let's consider it in the design so we can easily add it later, in case we need it.

Regards,

Andre

#20 lore raw
In reply to Vinicius Costa Gomes

Hi,

Quoting Vinicius Costa Gomes (2019-12-16 13:44:13)

Quoted 10 lines
Quoted 6 lines
Quoted 6 lines

Quoting Po Liu (2019-11-27 01:59:18)

Quoted 5 lines

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Having some knobs in ethtool to enable when/how Frame Preemption is advertised on the wire makes sense. I also agree that it should be "on" by default.

Agreed. If Frame Preemption is supported by hardware and it can be enabled/disabled, I think we should allow the user to configure it, instead of having it fixed in 'on'.

Quoted 12 lines
Quoted 8 lines
Quoted 4 lines

In an early RFC series [1], we proposed a way to support frame preemption. I'm not sure if you have considered it before implementing this other proposal based on ethtool interface so I thought it would be a good idea to bring that up to your attention, just in case.

Sorry, I didn't notice the RFC proposal. Using ethtool set the preemption just thinking about 8021Qbu as standalone. And not limit to the taprio if user won't set 802.1Qbv.

I see your point of using frame-preemption "standalone", I have two ideas:

Quoted 6 lines

1. add support in taprio to be configured without any schedule in the "full offload" mode. In practice, allowing taprio to work somewhat similar to (mqprio + frame-preemption), changes in the code should de fairly small;

2. extend mqprio to support frame-preemption;

I'm not sure 2) is a good way to support frame preemption "standalone" functionality since mpqrio already looks overloaded. Besides its original goal (map traffic flows to hardware queues), it also supports different modes and traffic shaping. I rather not add another functionality to it.

Quoted 6 lines
Quoted 2 lines

As some feedback also want to set the MAC merge minimal fragment size and get some more information of 802.3br.

The minimal fragment size, I guess, also makes sense to be kept in ethtool. That is we have a sane default, and allow the user to change this setting for special cases.

Yes, the mim fragment size is another configuration knob we should have, and it should probably live at the same place we land the enable/disable knob.

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

It also aligns with the gate control operations Set-And-Hold-MAC and Set-And- Release-MAC that can be set via 'sched-entry' (see Table 8.7 from 802.1Q-2018 for further details.

I am curious about Set-And-Hold-Mac via 'sched-entry'. Actually, it could be understand as guardband by hardware preemption. MAC should auto calculate the nano seconds before express entry slot start to break to two fragments. Set-And-Hold-MAC should minimal larger than the fragment-size oct times.

Quoted 4 lines

Another interesting point. My first idea is that when the schedule is offloaded to the driver and the driver detects that the "entry" width is smaller than the fragment side, the driver could reject that schedule with a nice error message.

My understanding is that, if HOLD operation is supported, the hardware issues an MM_CTL.request(HOLD) at 'holdAdvance' nsecs in advance to the point where the 'sched-entry' with Set-And-Hold-MAC starts. This comes from the description in Table 8-7 from 802.1Q-2018.

Best regards,

Andre

#21 lore raw
In reply to Po Liu

Hi,

Quoted 7 lines · Po Liu

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

I think that we should keep the concept of 'traffic classes' outside of ethtool. Ethtool should only care about queues (or similar concepts). And unless I am missing something here, what you mean by 'traffic class' here is really a hardware queue.

Quoted 91 lines

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

---
 include/linux/netdev_features.h | 5 ++++-
 include/uapi/linux/ethtool.h    | 5 ++++-
 net/core/ethtool.c              | 1 +
 3 files changed, 9 insertions(+), 2 deletions(-)
include/linux/netdev_features.h +4 -1
index 4b19c544c59a..299750a8b414 100644
@@ -80,6 +80,7 @@ enum {
 
 	NETIF_F_GRO_HW_BIT,		/* Hardware Generic receive offload */
 	NETIF_F_HW_TLS_RECORD_BIT,	/* Offload TLS record */
+	NETIF_F_HW_PREEMPTION_BIT,	/* Hardware TC frame preemption */
 
 	/*
 	 * Add your fresh new feature above and remember to update
@@ -150,6 +151,7 @@ enum {
 #define NETIF_F_GSO_UDP_L4	__NETIF_F(GSO_UDP_L4)
 #define NETIF_F_HW_TLS_TX	__NETIF_F(HW_TLS_TX)
 #define NETIF_F_HW_TLS_RX	__NETIF_F(HW_TLS_RX)
+#define NETIF_F_PREEMPTION	__NETIF_F(HW_PREEMPTION)
 
 /* Finds the next feature with the highest number of the range of start till 0.
  */
@@ -175,7 +177,8 @@ static inline int find_next_netdev_feature(u64 feature, unsigned long start)
 /* Features valid for ethtool to change */
 /* = all defined minus driver/device-class-related */
 #define NETIF_F_NEVER_CHANGE	(NETIF_F_VLAN_CHALLENGED | \
-				 NETIF_F_LLTX | NETIF_F_NETNS_LOCAL)
+				 NETIF_F_LLTX | NETIF_F_NETNS_LOCAL | \
+				 NETIF_F_PREEMPTION)
 
 /* remember that ((t)1 << t_BITS) is undefined in C99 */
 #define NETIF_F_ETHTOOL_BITS	((__NETIF_F_BIT(NETDEV_FEATURE_COUNT - 1) | \
include/uapi/linux/ethtool.h +4 -1
index d4591792f0b4..12ffb34afbfa 100644
@@ -1776,6 +1776,8 @@ enum ethtool_reset_flags {
 };
 #define ETH_RESET_SHARED_SHIFT	16
 
+/* Disable preemtion. */
+#define PREEMPTION_DISABLE     0x0
 
 /**
  * struct ethtool_link_settings - link control and status
@@ -1886,7 +1888,8 @@ struct ethtool_link_settings {
 	__s8	link_mode_masks_nwords;
 	__u8	transceiver;
 	__u8	reserved1[3];
-	__u32	reserved[7];
+	__u32	preemption;
+	__u32	reserved[6];
 	__u32	link_mode_masks[0];
 	/* layout of link_mode_masks fields:
 	 * __u32 map_supported[link_mode_masks_nwords];
net/core/ethtool.c +1 -0
index cd9bc67381b2..6ffcd8a602b8 100644
@@ -111,6 +111,7 @@ static const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN]
 	[NETIF_F_HW_TLS_RECORD_BIT] =	"tls-hw-record",
 	[NETIF_F_HW_TLS_TX_BIT] =	 "tls-hw-tx-offload",
 	[NETIF_F_HW_TLS_RX_BIT] =	 "tls-hw-rx-offload",
+	[NETIF_F_HW_PREEMPTION_BIT] =	 "tx-preemption",
 };
 
 static const char
Signature · 1 line
2.17.1
#22 lore raw

Hi, Vinicius,

Quoted 12 lines · Vinicius Costa Gomes

Hi,

Quoted 7 lines · Po Liu

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

Quoted 5 lines

I think that we should keep the concept of 'traffic classes' outside of ethtool. Ethtool should only care about queues (or similar concepts). And unless I am missing something here, what you mean by 'traffic class' here is really a hardware queue.

Agree.

So this is my understanding..

Once driver support FPE, then drive accepts tc gate operation, Set-And-Hold MAC & Set-And-Release-MAC. Otherwise it can only offload SetGateStates.

So candidates for ethtool command.

Set

- FPE enable/disable
- framePreemptionStatusTable of Table 12-30—Frame Preemption Parameter
table of 802.1Q 2018 which are defined as Read/Write. This is an array of Hw queues and if they are preemptible or express.
- Additionally 802.3br defines Min fragment size that is a candidate
for ethtool command.

Show the following:-

 -  holdAdvance
 -  releaseAdvance
 -  preemptionActive
 -  holdRequest

There is also Table 12-29—The Gate Parameter Table for taprio. These related to taprio schedule is already part of tc command. However there are below parameters that requires configuration by user as well. Can we use ethtool for the same as this
- queueMaxSDUTable

Below is what I read about this which is also for hw queues AFAIK.

==== from 802.1Q====================================================
A per-traffic class queue queueMaxSDU parameter defines the maximum
service data unit size for each queue; frames that exceed queueMaxSDU are discarded. The value of queueMaxSDU for each queue is configurable by management its default value is the maximum SDU size supported by the
MAC procedures employed on the LAN to which the frame is to be relayed
=======================================================================

I have question about the below parameters in The Gate Parameter Table that are not currently supported by tc command. Looks like they need to be shown to user for management.

- ConfigChange - Looks like this needs to be controlled by user. After sending admin command, user send this trigger to start copying admin schedule to operation schedule. Is this getting added to tc command?
- ConfigChangeTime - The time at which the administrative variables that determine the cycle are to be copied across to the corresponding operational variables, expressed as a PTP timescale
- TickGranularity - the management parameters specified in Gate Parameter Table allow a management station to discover the characteristics of an implementation’s cycle timer clock (TickGranularity) and to set the parameters for the gating cycle accordingly.
- ConfigPending - A Boolean variable, set TRUE to indicate that there is a new cycle configuration awaiting installation.
- ConfigChangeError - Error in configuration (AdminBaseTime < CurrentTime)
- SupportedListMax - Maximum supported Admin/Open shed list.

Is there a plan to export these from driver through tc show or such command? The reason being, there would be applications developed to manage configuration/schedule of TSN nodes that would requires these information from the node. So would need a support either in tc or some other means to retrieve them from hardware or driver. That is my understanding...

Regards,

Murali

Quoted 92 lines
Quoted 91 lines

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

---
  include/linux/netdev_features.h | 5 ++++-
  include/uapi/linux/ethtool.h    | 5 ++++-
  net/core/ethtool.c              | 1 +
  3 files changed, 9 insertions(+), 2 deletions(-)
include/linux/netdev_features.h +4 -1
index 4b19c544c59a..299750a8b414 100644
@@ -80,6 +80,7 @@ enum {
 
  	NETIF_F_GRO_HW_BIT,		/* Hardware Generic receive offload */
  	NETIF_F_HW_TLS_RECORD_BIT,	/* Offload TLS record */
+	NETIF_F_HW_PREEMPTION_BIT,	/* Hardware TC frame preemption */
 
  	/*
  	 * Add your fresh new feature above and remember to update
@@ -150,6 +151,7 @@ enum {
  #define NETIF_F_GSO_UDP_L4	__NETIF_F(GSO_UDP_L4)
  #define NETIF_F_HW_TLS_TX	__NETIF_F(HW_TLS_TX)
  #define NETIF_F_HW_TLS_RX	__NETIF_F(HW_TLS_RX)
+#define NETIF_F_PREEMPTION	__NETIF_F(HW_PREEMPTION)
 
  /* Finds the next feature with the highest number of the range of start till 0.
   */
@@ -175,7 +177,8 @@ static inline int find_next_netdev_feature(u64 feature, unsigned long start)
  /* Features valid for ethtool to change */
  /* = all defined minus driver/device-class-related */
  #define NETIF_F_NEVER_CHANGE	(NETIF_F_VLAN_CHALLENGED | \
-				 NETIF_F_LLTX | NETIF_F_NETNS_LOCAL)
+				 NETIF_F_LLTX | NETIF_F_NETNS_LOCAL | \
+				 NETIF_F_PREEMPTION)
 
  /* remember that ((t)1 << t_BITS) is undefined in C99 */
  #define NETIF_F_ETHTOOL_BITS	((__NETIF_F_BIT(NETDEV_FEATURE_COUNT - 1) | \
include/uapi/linux/ethtool.h +4 -1
index d4591792f0b4..12ffb34afbfa 100644
@@ -1776,6 +1776,8 @@ enum ethtool_reset_flags {
  };
  #define ETH_RESET_SHARED_SHIFT	16
 
+/* Disable preemtion. */
+#define PREEMPTION_DISABLE     0x0
 
  /**
   * struct ethtool_link_settings - link control and status
@@ -1886,7 +1888,8 @@ struct ethtool_link_settings {
  	__s8	link_mode_masks_nwords;
  	__u8	transceiver;
  	__u8	reserved1[3];
-	__u32	reserved[7];
+	__u32	preemption;
+	__u32	reserved[6];
  	__u32	link_mode_masks[0];
  	/* layout of link_mode_masks fields:
  	 * __u32 map_supported[link_mode_masks_nwords];
net/core/ethtool.c +1 -0
index cd9bc67381b2..6ffcd8a602b8 100644
@@ -111,6 +111,7 @@ static const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN]
  	[NETIF_F_HW_TLS_RECORD_BIT] =	"tls-hw-record",
  	[NETIF_F_HW_TLS_TX_BIT] =	 "tls-hw-tx-offload",
  	[NETIF_F_HW_TLS_RX_BIT] =	 "tls-hw-rx-offload",
+	[NETIF_F_HW_PREEMPTION_BIT] =	 "tx-preemption",
  };
 
  static const char
Signature · 1 line
2.17.1
Signature · 2 lines
Murali Karicheri
Texas Instruments
#23 lore raw

Hi Murali,

Quoted 9 lines · Murali Karicheri

I have question about the below parameters in The Gate Parameter Table that are not currently supported by tc command. Looks like they need to be shown to user for management.

- ConfigChange - Looks like this needs to be controlled by user. After sending admin command, user send this trigger to start copying admin schedule to operation schedule. Is this getting added to tc command?

"The ConfigChange parameter signals the start of a configuration change for the gate
when it is set to TRUE. This should only be done when the various administrative parameters are all set to appropriate values."

As far as my understanding goes, all tc-taprio commands currently behave as though this boolean is implicitly set to TRUE after the structures have been set up. I'm not sure there is any value in doing otherwise.

Quoted 3 lines

- ConfigChangeTime - The time at which the administrative variables that determine the cycle are to be copied across to the corresponding operational variables, expressed as a PTP timescale

This is the base-time of the admin schedule, no?

"The PTPtime at which the next config change is scheduled to occur. The value is a representation of a PTPtime value, consisting of a 48-bit integer
number of seconds and a 32-bit integer number of nanoseconds."

Quoted 5 lines

- TickGranularity - the management parameters specified in Gate Parameter Table allow a management station to discover the characteristics of an implementation’s cycle timer clock (TickGranularity) and to set the parameters for the gating cycle accordingly.

Not sure who is going to use this and for what purpose, but ok.

Quoted 2 lines

- ConfigPending - A Boolean variable, set TRUE to indicate that there is a new cycle configuration awaiting installation.

I had tried to export something like this (driver calls back into sch_taprio.c when hw has applied the config, this would result in ConfigPending = FALSE), but ultimately didn't finish the idea, and it caused some problems too, due to incorrect RCU usage.

Quoted 2 lines

- ConfigChangeError - Error in configuration (AdminBaseTime < CurrentTime)

This can be exported similarly.

Quoted 9 lines

- SupportedListMax - Maximum supported Admin/Open shed list.

Is there a plan to export these from driver through tc show or such command? The reason being, there would be applications developed to manage configuration/schedule of TSN nodes that would requires these information from the node. So would need a support either in tc or some other means to retrieve them from hardware or driver. That is my understanding...

Not sure what answer you expect to receive for "is there any plan". You can go ahead and propose something, as long as it is reasonably useful to have.

Quoted 7 lines

Regards,

Murali

--
Murali Karicheri
Texas Instruments

Thanks,
-Vladimir

#24 lore raw

Hi,

Quoted 43 lines · Vladimir Oltean

Hi Murali,

Quoted 9 lines · Murali Karicheri

I have question about the below parameters in The Gate Parameter Table that are not currently supported by tc command. Looks like they need to be shown to user for management.

- ConfigChange - Looks like this needs to be controlled by user. After sending admin command, user send this trigger to start copying admin schedule to operation schedule. Is this getting added to tc command?

"The ConfigChange parameter signals the start of a configuration change for the gate
when it is set to TRUE. This should only be done when the various administrative parameters are all set to appropriate values."

As far as my understanding goes, all tc-taprio commands currently behave as though this boolean is implicitly set to TRUE after the structures have been set up. I'm not sure there is any value in doing otherwise.

Quoted 3 lines

- ConfigChangeTime - The time at which the administrative variables that determine the cycle are to be copied across to the corresponding operational variables, expressed as a PTP timescale

This is the base-time of the admin schedule, no?

"The PTPtime at which the next config change is scheduled to occur. The value is a representation of a PTPtime value, consisting of a 48-bit integer
number of seconds and a 32-bit integer number of nanoseconds."

Quoted 5 lines

- TickGranularity - the management parameters specified in Gate Parameter Table allow a management station to discover the characteristics of an implementation’s cycle timer clock (TickGranularity) and to set the parameters for the gating cycle accordingly.

Not sure who is going to use this and for what purpose, but ok.

Quoted 8 lines
Quoted 2 lines

- ConfigPending - A Boolean variable, set TRUE to indicate that there is a new cycle configuration awaiting installation.

I had tried to export something like this (driver calls back into sch_taprio.c when hw has applied the config, this would result in ConfigPending = FALSE), but ultimately didn't finish the idea, and it caused some problems too, due to incorrect RCU usage.

If this should be exported, this should be done from taprio, perhaps adding a new field to what is exported via the dump() callback, which should be quite easy.

Quoted 4 lines
Quoted 2 lines

- ConfigChangeError - Error in configuration (AdminBaseTime < CurrentTime)

This can be exported similarly.

In my view, having this as a "runtime" error is not useful, as we can verify this at configuration time.

Quoted 10 lines
Quoted 10 lines

- SupportedListMax - Maximum supported Admin/Open shed list.

Is there a plan to export these from driver through tc show or such command? The reason being, there would be applications developed to manage configuration/schedule of TSN nodes that would requires these information from the node. So would need a support either in tc or some other means to retrieve them from hardware or driver. That is my understanding...

Hm, now I understamd what you meant here...

Quoted 4 lines

Not sure what answer you expect to receive for "is there any plan". You can go ahead and propose something, as long as it is reasonably useful to have.

... if this is indeed useful, perhaps one way to do is to add a subcommand to TC_SETUP_QDISC_TAPRIO, so we can retrieve the stats/information we want from the driver. Similar to what cls_flower does.

Quoted 11 lines
Quoted 8 lines

Regards,

Murali

--
Murali Karicheri
Texas Instruments

Thanks,
-Vladimir

Cheers,
--
Vinicius

#25 lore raw

Hi,

Quoted 56 lines · Vinicius Costa Gomes

Hi,

Quoted 43 lines · Vladimir Oltean

Hi Murali,

Quoted 9 lines · Murali Karicheri

I have question about the below parameters in The Gate Parameter Table that are not currently supported by tc command. Looks like they need to be shown to user for management.

- ConfigChange - Looks like this needs to be controlled by user. After sending admin command, user send this trigger to start copying admin schedule to operation schedule. Is this getting added to tc command?

"The ConfigChange parameter signals the start of a configuration change for the gate
when it is set to TRUE. This should only be done when the various administrative parameters are all set to appropriate values."

As far as my understanding goes, all tc-taprio commands currently behave as though this boolean is implicitly set to TRUE after the structures have been set up. I'm not sure there is any value in doing otherwise.

Quoted 3 lines

- ConfigChangeTime - The time at which the administrative variables that determine the cycle are to be copied across to the corresponding operational variables, expressed as a PTP timescale

This is the base-time of the admin schedule, no?

"The PTPtime at which the next config change is scheduled to occur. The value is a representation of a PTPtime value, consisting of a 48-bit integer
number of seconds and a 32-bit integer number of nanoseconds."

Quoted 5 lines

- TickGranularity - the management parameters specified in Gate Parameter Table allow a management station to discover the characteristics of an implementation’s cycle timer clock (TickGranularity) and to set the parameters for the gating cycle accordingly.

Not sure who is going to use this and for what purpose, but ok.

Quoted 8 lines
Quoted 2 lines

- ConfigPending - A Boolean variable, set TRUE to indicate that there is a new cycle configuration awaiting installation.

I had tried to export something like this (driver calls back into sch_taprio.c when hw has applied the config, this would result in ConfigPending = FALSE), but ultimately didn't finish the idea, and it caused some problems too, due to incorrect RCU usage.

Quoted 4 lines

If this should be exported, this should be done from taprio, perhaps adding a new field to what is exported via the dump() callback, which should be quite easy.

We are still working to send a patch for taprio offload on our hardware and it may take a while to get to this. So if someone can help to add the required kernel/driver interface for this, that will be great!

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

- ConfigChangeError - Error in configuration (AdminBaseTime < CurrentTime)

This can be exported similarly.

In my view, having this as a "runtime" error is not useful, as we can verify this at configuration time.

Looks like this is not an error per 802.1Q standard if I understood it correctly.

This is what I see.
=======================================================================
From 802.1Q 2018, 8.6.9.1.1 SetCycleStartTime()

If AdminBaseTime is set to the same time in the past in all bridges and end stations, OperBaseTime is always in the past, and all cycles start synchronized. Using AdminBaseTime in the past is appropriate when you can start schedules prior to starting the application that uses the schedules. Use of AdminBaseTime in the future is intended to change a currently running schedule in all bridges and end stations to a new schedule at a future time. Using AdminBaseTime in the future is appropriate when schedules must be changed without stopping the application
========================================================================

Quoted 15 lines
Quoted 11 lines
Quoted 11 lines

- SupportedListMax - Maximum supported Admin/Open shed list.

Is there a plan to export these from driver through tc show or such command? The reason being, there would be applications developed to manage configuration/schedule of TSN nodes that would requires these information from the node. So would need a support either in tc or some other means to retrieve them from hardware or driver. That is my understanding...

Hm, now I understamd what you meant here...

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Not sure what answer you expect to receive for "is there any plan". You can go ahead and propose something, as long as it is reasonably useful to have.

... if this is indeed useful, perhaps one way to do is to add a subcommand to TC_SETUP_QDISC_TAPRIO, so we can retrieve the stats/information we want from the driver. Similar to what cls_flower does.

What I understand is that there will be some work done to allow auto configuration of TSN nodes from user space and that would need access to all or some of the above parameters along with tc command to configure the same. May be a open source project for this or some custom application? Any such projects existing??

Regards,

Murali

Quoted 16 lines
Quoted 11 lines
Quoted 8 lines

Regards,

Murali

--
Murali Karicheri
Texas Instruments

Thanks,
-Vladimir

Cheers,
--
Vinicius

Signature · 2 lines
Murali Karicheri
Texas Instruments
#26 lore raw
Quoted 3 lines · Murali Karicheri

We are still working to send a patch for taprio offload on our hardware and it may take a while to get to this. So if someone can help to add the required kernel/driver interface for this, that will be great!

Will add this to my todo list. But if anyone else has the spare cycles feel free to have a go at it.

Quoted 16 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

- ConfigChangeError - Error in configuration (AdminBaseTime < CurrentTime)

This can be exported similarly.

In my view, having this as a "runtime" error is not useful, as we can verify this at configuration time.

Looks like this is not an error per 802.1Q standard if I understood it correctly.

This is what I see.
=======================================================================
From 802.1Q 2018, 8.6.9.1.1 SetCycleStartTime()

Quoted 11 lines

If AdminBaseTime is set to the same time in the past in all bridges and end stations, OperBaseTime is always in the past, and all cycles start synchronized. Using AdminBaseTime in the past is appropriate when you can start schedules prior to starting the application that uses the schedules. Use of AdminBaseTime in the future is intended to change a currently running schedule in all bridges and end stations to a new schedule at a future time. Using AdminBaseTime in the future is appropriate when schedules must be changed without stopping the application
========================================================================

What I meant here is the case that I already have an "oper" schedule running, so my "oper->base_time" is in the past, and I try to add an "admin" schedule with a "base_time" also in the past. What's the expected behavior in this case? The text about stopping/starting applications doesn't seem to apply to the way the tc subsystem interacts with the applications.

Quoted 24 lines
Quoted 15 lines
Quoted 11 lines
Quoted 11 lines

- SupportedListMax - Maximum supported Admin/Open shed list.

Is there a plan to export these from driver through tc show or such command? The reason being, there would be applications developed to manage configuration/schedule of TSN nodes that would requires these information from the node. So would need a support either in tc or some other means to retrieve them from hardware or driver. That is my understanding...

Hm, now I understamd what you meant here...

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Not sure what answer you expect to receive for "is there any plan". You can go ahead and propose something, as long as it is reasonably useful to have.

... if this is indeed useful, perhaps one way to do is to add a subcommand to TC_SETUP_QDISC_TAPRIO, so we can retrieve the stats/information we want from the driver. Similar to what cls_flower does.

Quoted 5 lines

What I understand is that there will be some work done to allow auto configuration of TSN nodes from user space and that would need access to all or some of the above parameters along with tc command to configure the same. May be a open source project for this or some custom application? Any such projects existing??

Yeah, this is a big missing piece for TSN. I've heard 'netopeer2' and 'sysrepo' mentioned when similar questions were asked, but I have still to take a look at them and see what's missing. (Or if they are the right tool for the job)

Signature · 1 line
Vinicius
#27 lore raw

Hi Vinicius,

Quoted 9 lines · Vinicius Costa Gomes
Quoted 3 lines · Murali Karicheri

We are still working to send a patch for taprio offload on our hardware and it may take a while to get to this. So if someone can help to add the required kernel/driver interface for this, that will be great!

Will add this to my todo list. But if anyone else has the spare cycles feel free to have a go at it.

Thanks! We have made some progress in sending the base driver to netdev list now https://lkml.org/lkml/2020/2/22/157

This device is taprio offload capable. Next step is to add taprio offload to this driver. Then other features will follow.

Quoted 35 lines
Quoted 16 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

- ConfigChangeError - Error in configuration (AdminBaseTime < CurrentTime)

This can be exported similarly.

In my view, having this as a "runtime" error is not useful, as we can verify this at configuration time.

Looks like this is not an error per 802.1Q standard if I understood it correctly.

This is what I see.
=======================================================================
From 802.1Q 2018, 8.6.9.1.1 SetCycleStartTime()

Quoted 11 lines

If AdminBaseTime is set to the same time in the past in all bridges and end stations, OperBaseTime is always in the past, and all cycles start synchronized. Using AdminBaseTime in the past is appropriate when you can start schedules prior to starting the application that uses the schedules. Use of AdminBaseTime in the future is intended to change a currently running schedule in all bridges and end stations to a new schedule at a future time. Using AdminBaseTime in the future is appropriate when schedules must be changed without stopping the application
========================================================================

What I meant here is the case that I already have an "oper" schedule running, so my "oper->base_time" is in the past, and I try to add an "admin" schedule with a "base_time" also in the past. What's the expected behavior in this case? The text about stopping/starting applications doesn't seem to apply to the way the tc subsystem interacts with the applications.

Quoted 2 lines

I try to add an "admin" schedule with a "base_time" also in the past. What's the expected behavior in this case?

Ok got it. I don't think this behavior is explained in the spec. I would
assume a sane thing to do is to switch to admin schedule if admin->base_time is newer than oper->base_time and flag the ConfigChangeError to be compliant to the spec, but frankly speaking
I don't know how application is going to use this. It is a low priority
item IMO and can be added as needed.

Regards,

Murali

Quoted 36 lines
Quoted 24 lines
Quoted 15 lines
Quoted 11 lines
Quoted 11 lines

- SupportedListMax - Maximum supported Admin/Open shed list.

Is there a plan to export these from driver through tc show or such command? The reason being, there would be applications developed to manage configuration/schedule of TSN nodes that would requires these information from the node. So would need a support either in tc or some other means to retrieve them from hardware or driver. That is my understanding...

Hm, now I understamd what you meant here...

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Not sure what answer you expect to receive for "is there any plan". You can go ahead and propose something, as long as it is reasonably useful to have.

... if this is indeed useful, perhaps one way to do is to add a subcommand to TC_SETUP_QDISC_TAPRIO, so we can retrieve the stats/information we want from the driver. Similar to what cls_flower does.

Quoted 5 lines

What I understand is that there will be some work done to allow auto configuration of TSN nodes from user space and that would need access to all or some of the above parameters along with tc command to configure the same. May be a open source project for this or some custom application? Any such projects existing??

Yeah, this is a big missing piece for TSN. I've heard 'netopeer2' and 'sysrepo' mentioned when similar questions were asked, but I have still to take a look at them and see what's missing. (Or if they are the right tool for the job)

Signature · 2 lines
Murali Karicheri
Texas Instruments
#28 lore raw
In reply to Vladimir Oltean

Hi Vladimir,

Quoted 19 lines · Vladimir Oltean

Hi Murali,

Quoted 9 lines · Murali Karicheri

I have question about the below parameters in The Gate Parameter Table that are not currently supported by tc command. Looks like they need to be shown to user for management.

- ConfigChange - Looks like this needs to be controlled by user. After sending admin command, user send this trigger to start copying admin schedule to operation schedule. Is this getting added to tc command?

"The ConfigChange parameter signals the start of a configuration change for the gate
when it is set to TRUE. This should only be done when the various administrative parameters are all set to appropriate values."

Quoted 5 lines

As far as my understanding goes, all tc-taprio commands currently behave as though this boolean is implicitly set to TRUE after the structures have been set up. I'm not sure there is any value in doing otherwise.

That is my understanding as well. However I found this in the 802.1Q and want to see if someone has insight into this parameter. So perhaps we can ignore this for now and re-visit when there is a real need for the same.

Quoted 5 lines
Quoted 3 lines

- ConfigChangeTime - The time at which the administrative variables that determine the cycle are to be copied across to the corresponding operational variables, expressed as a PTP timescale

This is the base-time of the admin schedule, no?

I think there is cycle time extension possible and in that case, ConfigChangeTime may be different from Admin base-time. So this gives the actual time when the cycle configuration is actually applied.

Quoted 6 lines

"The PTPtime at which the next config change is scheduled to occur. The value is a representation of a PTPtime value, consisting of a 48-bit integer
number of seconds and a 32-bit integer number of nanoseconds."

Quoted 7 lines
Quoted 5 lines

- TickGranularity - the management parameters specified in Gate Parameter Table allow a management station to discover the characteristics of an implementation’s cycle timer clock (TickGranularity) and to set the parameters for the gating cycle accordingly.

Not sure who is going to use this and for what purpose, but ok.

Same here. Not sure how this will be used by application.

Quoted 9 lines
Quoted 3 lines

- ConfigPending - A Boolean variable, set TRUE to indicate that there is a new cycle configuration awaiting installation.

I had tried to export something like this (driver calls back into sch_taprio.c when hw has applied the config, this would result in ConfigPending = FALSE), but ultimately didn't finish the idea, and it caused some problems too, due to incorrect RCU usage.

Ok. Hope you plan to send a patch for this in the future.

Quoted 4 lines
Quoted 2 lines

- ConfigChangeError - Error in configuration (AdminBaseTime < CurrentTime)

This can be exported similarly.

Ok.

Quoted 11 lines
Quoted 10 lines

- SupportedListMax - Maximum supported Admin/Open shed list.

Is there a plan to export these from driver through tc show or such command? The reason being, there would be applications developed to manage configuration/schedule of TSN nodes that would requires these information from the node. So would need a support either in tc or some other means to retrieve them from hardware or driver. That is my understanding...

Quoted 1 line

Not sure what answer you expect to receive for "is there any plan".

To avoid duplicating the work if someone is already working on this.

Quoted 2 lines

You can go ahead and propose something, as long as it is reasonably useful to have.

Ok. Will keep in mind.

Quoted 12 lines
Quoted 8 lines

Regards,

Murali

--
Murali Karicheri
Texas Instruments

Thanks,
-Vladimir

Signature · 2 lines
Murali Karicheri
Texas Instruments
#29 lore raw
In reply to Po Liu

Hi,

Quoted 18 lines · Po Liu

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Quoted 6 lines

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

Any updates on this series? If you think that there's something that I could help, just tell.

Cheers,

Signature · 1 line
Vinicius
author
#30 lore raw

Hi Vinicius,

Br,
Po Liu

Quoted 45 lines

-----Original Message-----
From: Vinicius Costa Gomes <[email protected]>
Sent: 2020年2月22日 5:44
To: Po Liu <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]
Cc: [email protected]; Claudiu Manoil <[email protected]>; Vladimir Oltean <[email protected]>; Alexandru Marginean <[email protected]>; Xiaoliang Yang <[email protected]>; Roy Zang <[email protected]>; Mingkai Hu <[email protected]>; Jerry Huang <[email protected]>; Leo Li <[email protected]>; Po Liu <[email protected]>
Subject: [EXT] Re: [v1,net-next, 1/2] ethtool: add setting frame preemption of traffic classes

Caution: EXT Email

Hi,

Quoted 21 lines · Po Liu

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Quoted 6 lines
Quoted 3 lines

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

Any updates on this series? If you think that there's something that I could help, just tell.

Sorry for the long time not involve the discussion. I am focus on other tsn code for tc flower. If you can take more about this preemption serial, that would be good.

I summary some suggestions from Marali Karicheri and Ivan Khornonzhuk and by you and also others:
- Add config the fragment size, hold advance, release advance and flags;
My comments about the fragment size is in the Qbu spec limit the fragment size " the minimum non-final fragment size is 64,
128, 192, or 256 octets " this setting would affect the guardband setting for Qbv. But the ethtool setting could not involve this issues but by the taprio side.
- " Furthermore, this setting could be extend for a serial setting for mac and traffic class." "Better not to using the traffic class concept."
Could adding a serial setting by "ethtool --preemption xxx" or other name. I don' t think it is good to involve in the queue control since queues number may bigger than the TC number.
- The ethtool is the better choice to configure the preemption
I agree.

Thanks!

Quoted 5 lines

Cheers,
--
Vinicius

author
#32 lore raw

Br,
Po Liu

Quoted 101 lines

-----Original Message-----
From: Murali Karicheri <[email protected]>
Sent: 2020年2月26日 1:59
To: Po Liu <[email protected]>; Vinicius Costa Gomes <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Claudiu Manoil <[email protected]>; Vladimir Oltean <[email protected]>; Alexandru Marginean <[email protected]>; Xiaoliang Yang <[email protected]>; Roy Zang <[email protected]>; Mingkai Hu <[email protected]>; Jerry Huang <[email protected]>; Leo Li <[email protected]>; Ivan Khoronzhuk <[email protected]>
Subject: Re: [EXT] Re: [v1,net-next, 1/2] ethtool: add setting frame preemption of traffic classes

Caution: EXT Email

Hi Po,

Quoted 14 lines · Po Liu

Hi Vinicius,

Br,
Po Liu

Quoted 8 lines

-----Original Message-----
From: Vinicius Costa Gomes <[email protected]>
Sent: 2020年2月22日 5:44
To: Po Liu <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 4 lines
Quoted 4 lines

[email protected]; linux- [email protected]; [email protected]
Cc: [email protected]; Claudiu Manoil <[email protected]>; Vladimir Oltean

<[email protected]>;

Quoted 1 line
Quoted 1 line

Alexandru Marginean <[email protected]>; Xiaoliang

Yang

Quoted 1 line
Quoted 1 line

<[email protected]>; Roy Zang <[email protected]>; Mingkai

Hu

Quoted 32 lines
Quoted 32 lines

<[email protected]>; Jerry Huang <[email protected]>; Leo Li <[email protected]>; Po Liu <[email protected]>
Subject: [EXT] Re: [v1,net-next, 1/2] ethtool: add setting frame preemption of traffic classes

Caution: EXT Email

Hi,

Quoted 21 lines · Po Liu

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

Any updates on this series? If you think that there's something that I could help, just tell.

Sorry for the long time not involve the discussion. I am focus on other

tsn code for tc flower.

Quoted 3 lines

If you can take more about this preemption serial, that would be good.

I summary some suggestions from Marali Karicheri and Ivan

Khornonzhuk and by you and also others:

It is Murali :)

Quoted 2 lines

- Add config the fragment size, hold advance, release advance and flags;

- I believe hold advance, release advance values are read only

Quoted 8 lines
Quoted 2 lines

My comments about the fragment size is in the Qbu spec limit the
fragment size " the minimum non-final fragment size is 64, 128, 192, or

256 octets " this setting would affect the guardband setting for Qbv. But the ethtool setting could not involve this issues but by the taprio side.

Quoted 1 line

- " Furthermore, this setting could be extend for a serial setting for mac

and traffic class." "Better not to using the traffic class concept."
- Couldn't understand the above? Could you please explain a bit more on what you meant here?

Oh, I copy the original suggestion by you and other here. Just for more evaluation for next version.

Quoted 18 lines

Murali

Quoted 1 line

Could adding a serial setting by "ethtool --preemption xxx" or other

name. I don' t think it is good to involve in the queue control since queues number may bigger than the TC number.

Quoted 9 lines

- The ethtool is the better choice to configure the preemption I agree.

Thanks!

Quoted 5 lines

Cheers,
--
Vinicius

--
Murali Karicheri
Texas Instruments

#33 lore raw
In reply to Po Liu

Hi,

Quoted 61 lines · Po Liu

Hi Vinicius,

Br,
Po Liu

Quoted 45 lines

-----Original Message-----
From: Vinicius Costa Gomes <[email protected]>
Sent: 2020年2月22日 5:44
To: Po Liu <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]
Cc: [email protected]; Claudiu Manoil <[email protected]>; Vladimir Oltean <[email protected]>; Alexandru Marginean <[email protected]>; Xiaoliang Yang <[email protected]>; Roy Zang <[email protected]>; Mingkai Hu <[email protected]>; Jerry Huang <[email protected]>; Leo Li <[email protected]>; Po Liu <[email protected]>
Subject: [EXT] Re: [v1,net-next, 1/2] ethtool: add setting frame preemption of traffic classes

Caution: EXT Email

Hi,

Quoted 21 lines · Po Liu

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Quoted 6 lines
Quoted 3 lines

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

Any updates on this series? If you think that there's something that I could help, just tell.

Sorry for the long time not involve the discussion. I am focus on other tsn code for tc flower. If you can take more about this preemption serial, that would be good.

Quoted 8 lines

I summary some suggestions from Marali Karicheri and Ivan Khornonzhuk and by you and also others:
- Add config the fragment size, hold advance, release advance and flags;
My comments about the fragment size is in the Qbu spec limit the fragment size " the minimum non-final fragment size is 64,
128, 192, or 256 octets " this setting would affect the guardband setting for Qbv. But the ethtool setting could not involve this issues but by the taprio side.
- " Furthermore, this setting could be extend for a serial setting for mac and traffic class." "Better not to using the traffic class concept."
Could adding a serial setting by "ethtool --preemption xxx" or other name. I don' t think it is good to involve in the queue control since queues number may bigger than the TC number.
- The ethtool is the better choice to configure the preemption
I agree.

Just a quick update. I was able to dedicate some time to this, and have something aproaching RFC-quality, but it needs more testing.

So, question, what were you using for testing this? Anything special?

And btw, thanks for the summary of the discussion.

Quoted 7 lines

Thanks!

Quoted 5 lines

Cheers,
--
Vinicius

Signature · 1 line
Vinicius
author
#34 lore raw

Br,
Po Liu

Quoted 100 lines

-----Original Message-----
From: Vinicius Costa Gomes <[email protected]>
Sent: 2020年3月13日 7:35
To: Po Liu <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Claudiu Manoil <[email protected]>; Vladimir Oltean <[email protected]>; Alexandru Marginean <[email protected]>; Xiaoliang Yang <[email protected]>; Roy Zang <[email protected]>; Mingkai Hu <[email protected]>; Jerry Huang <[email protected]>; Leo Li <[email protected]>; Murali Karicheri <[email protected]>; Ivan Khoronzhuk <[email protected]>
Subject: RE: [EXT] Re: [v1,net-next, 1/2] ethtool: add setting frame preemption of traffic classes

Caution: EXT Email

Hi,

Quoted 52 lines · Po Liu

Hi Vinicius,

Br,
Po Liu

Quoted 46 lines

-----Original Message-----
From: Vinicius Costa Gomes <[email protected]>
Sent: 2020年2月22日 5:44
To: Po Liu <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]
Cc: [email protected]; Claudiu Manoil <[email protected]>; Vladimir Oltean <[email protected]>; Alexandru Marginean <[email protected]>; Xiaoliang Yang <[email protected]>; Roy Zang <[email protected]>; Mingkai Hu <[email protected]>; Jerry Huang <[email protected]>; Leo Li <[email protected]>; Po Liu <[email protected]>
Subject: [EXT] Re: [v1,net-next, 1/2] ethtool: add setting frame preemption of traffic classes

Caution: EXT Email

Hi,

Quoted 21 lines · Po Liu

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

Any updates on this series? If you think that there's something that I could help, just tell.

Sorry for the long time not involve the discussion. I am focus on other tsn

code for tc flower.

Quoted 3 lines

If you can take more about this preemption serial, that would be good.

I summary some suggestions from Marali Karicheri and Ivan Khornonzhuk

and by you and also others:

Quoted 3 lines

- Add config the fragment size, hold advance, release advance and flags; My comments about the fragment size is in the Qbu spec limit the
fragment size " the minimum non-final fragment size is 64, 128, 192, or 256

octets " this setting would affect the guardband setting for Qbv. But the ethtool setting could not involve this issues but by the taprio side.

Quoted 1 line

- " Furthermore, this setting could be extend for a serial setting for mac and

traffic class." "Better not to using the traffic class concept."

Quoted 1 line

Could adding a serial setting by "ethtool --preemption xxx" or other name.

I don' t think it is good to involve in the queue control since queues number may bigger than the TC number.

Quoted 7 lines
Quoted 2 lines

- The ethtool is the better choice to configure the preemption I agree.

Just a quick update. I was able to dedicate some time to this, and have something aproaching RFC-quality, but it needs more testing.

So, question, what were you using for testing this? Anything special?

I tested on www.nxp.com/LS1028A. There is nothing special for preemption.

Quoted 16 lines

And btw, thanks for the summary of the discussion.

Quoted 7 lines

Thanks!

Quoted 5 lines

Cheers,
--
Vinicius

--
Vinicius

#35 lore raw
In reply to Vinicius Costa Gomes

Hi Vinicius,

Quoted 74 lines · Vinicius Costa Gomes

Hi,

Quoted 61 lines · Po Liu

Hi Vinicius,

Br,
Po Liu

Quoted 45 lines

-----Original Message-----
From: Vinicius Costa Gomes <[email protected]>
Sent: 2020年2月22日 5:44
To: Po Liu <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]
Cc: [email protected]; Claudiu Manoil <[email protected]>; Vladimir Oltean <[email protected]>; Alexandru Marginean <[email protected]>; Xiaoliang Yang <[email protected]>; Roy Zang <[email protected]>; Mingkai Hu <[email protected]>; Jerry Huang <[email protected]>; Leo Li <[email protected]>; Po Liu <[email protected]>
Subject: [EXT] Re: [v1,net-next, 1/2] ethtool: add setting frame preemption of traffic classes

Caution: EXT Email

Hi,

Quoted 21 lines · Po Liu

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class. MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Quoted 6 lines
Quoted 3 lines

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

Any updates on this series? If you think that there's something that I could help, just tell.

Sorry for the long time not involve the discussion. I am focus on other tsn code for tc flower. If you can take more about this preemption serial, that would be good.

Quoted 8 lines

I summary some suggestions from Marali Karicheri and Ivan Khornonzhuk and by you and also others:
- Add config the fragment size, hold advance, release advance and flags;
My comments about the fragment size is in the Qbu spec limit the fragment size " the minimum non-final fragment size is 64,
128, 192, or 256 octets " this setting would affect the guardband setting for Qbv. But the ethtool setting could not involve this issues but by the taprio side.
- " Furthermore, this setting could be extend for a serial setting for mac and traffic class." "Better not to using the traffic class concept."
Could adding a serial setting by "ethtool --preemption xxx" or other name. I don' t think it is good to involve in the queue control since queues number may bigger than the TC number.
- The ethtool is the better choice to configure the preemption
I agree.

Quoted 3 lines

Just a quick update. I was able to dedicate some time to this, and have something aproaching RFC-quality, but it needs more testing.

Great! I have got my frame preemption working on my SoC. Currently I am using some defaults. I test it by using statistics provided by the SoC. I will be able to integrate and test your patch using my internal version and will include it in my patch to upstream once I am ready.

Regards,

Murali

Quoted 13 lines

So, question, what were you using for testing this? Anything special?

And btw, thanks for the summary of the discussion.

Quoted 7 lines

Thanks!

Quoted 5 lines

Cheers,
--
Vinicius

Signature · 2 lines
Murali Karicheri
Texas Instruments
#36 lore raw

Hi Vinicius,

Quoted 78 lines · Murali Karicheri

Hi Vinicius,

Quoted 75 lines · Vinicius Costa Gomes

Hi,

Quoted 70 lines · Po Liu

Hi Vinicius,

Br,
Po Liu

Quoted 45 lines

-----Original Message-----
From: Vinicius Costa Gomes <[email protected]>
Sent: 2020年2月22日 5:44
To: Po Liu <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]
Cc: [email protected]; Claudiu Manoil <[email protected]>; Vladimir Oltean <[email protected]>; Alexandru Marginean <[email protected]>; Xiaoliang Yang <[email protected]>; Roy Zang <[email protected]>; Mingkai Hu <[email protected]>; Jerry Huang <[email protected]>; Leo Li <[email protected]>; Po Liu <[email protected]>
Subject: [EXT] Re: [v1,net-next, 1/2] ethtool: add setting frame preemption of traffic classes

Caution: EXT Email

Hi,

Quoted 21 lines · Po Liu

IEEE Std 802.1Qbu standard defined the frame preemption of port traffic classes. This patch introduce a method to set traffic classes preemption. Add a parameter 'preemption' in struct ethtool_link_settings. The value will be translated to a binary, each bit represent a traffic class. Bit "1" means preemptable traffic class. Bit "0" means express traffic class.  MSB represent high number traffic class.

If hardware support the frame preemption, driver could set the ethernet device with hw_features and features with NETIF_F_PREEMPTION when initializing the port driver.

User can check the feature 'tx-preemption' by command 'ethtool -k devname'. If hareware set preemption feature. The property would be a fixed value 'on' if hardware support the frame preemption. Feature would show a fixed value 'off' if hardware don't support the frame preemption.

Command 'ethtool devname' and 'ethtool -s devname preemption N' would show/set which traffic classes are frame preemptable.

Quoted 6 lines
Quoted 3 lines

Port driver would implement the frame preemption in the function get_link_ksettings() and set_link_ksettings() in the struct ethtool_ops.

Any updates on this series? If you think that there's something that I could help, just tell.

Sorry for the long time not involve the discussion. I am focus on other tsn code for tc flower. If you can take more about this preemption serial, that would be good.

I summary some suggestions from Marali Karicheri and Ivan Khornonzhuk and by you and also others:
- Add config the fragment size, hold advance, release advance and flags;
     My comments about the fragment size is in the Qbu spec limit the fragment size " the minimum non-final fragment size is 64,
128, 192, or 256 octets " this setting would affect the guardband setting for Qbv. But the ethtool setting could not involve this issues but by the taprio side.
- " Furthermore, this setting could be extend for a serial setting
for mac and traffic class."  "Better not to using the traffic class concept."
    Could adding a serial setting by "ethtool --preemption xxx" or other name. I don' t think it is good to involve in the queue control since queues number may bigger than the TC number.
- The ethtool is the better choice to configure the preemption
   I agree.

Quoted 8 lines
Quoted 3 lines

Just a quick update. I was able to dedicate some time to this, and have something aproaching RFC-quality, but it needs more testing.

Great! I have got my frame preemption working on my SoC. Currently I am using some defaults. I test it by using statistics provided by the SoC. I will be able to integrate and test your patch using my internal version and will include it in my patch to upstream once I am ready.

Any progress on your side for a patch for the support?

I have posted my EST offload series for AM65x CPSW to netdev list today at

https://marc.info/?l=linux-netdev&m=158937640015582&w=2
https://marc.info/?l=linux-netdev&m=158937639515579&w=2
https://marc.info/?l=linux-netdev&m=158937638315573&w=2

Next on my list of things to do is the IET FPE support for which I need to have ethtool interface to allow configuring the express/preemptible queues and feature enable/disable. Currently I am using a ethtool priv-flags and some defaults. If you can post a patch, I will be able to integrate and test it on AM65x CPSW driver and provide my comments/ Tested-by:

Regards,

Murali

Quoted 17 lines

Regards,

Murali

Quoted 13 lines

So, question, what were you using for testing this? Anything special?

And btw, thanks for the summary of the discussion.

Quoted 7 lines

Thanks!

Quoted 5 lines

Cheers,

Signature · 1 line
Vinicius
Signature · 2 lines
Murali Karicheri
Texas Instruments
#37 lore raw

Hi Murali,

Quoted 2 lines · Murali Karicheri

Any progress on your side for a patch for the support?

Sorry for the delay, things got a bit crazy here for some time.

I have a RFC-quality series that I am finishing testing, I'll try to post it this week.

Quoted 6 lines

I have posted my EST offload series for AM65x CPSW to netdev list today at

https://marc.info/?l=linux-netdev&m=158937640015582&w=2
https://marc.info/?l=linux-netdev&m=158937639515579&w=2
https://marc.info/?l=linux-netdev&m=158937638315573&w=2

I'll try to have a look.

Quoted 7 lines

Next on my list of things to do is the IET FPE support for which I need to have ethtool interface to allow configuring the express/preemptible queues and feature enable/disable. Currently I am using a ethtool priv-flags and some defaults. If you can post a patch, I will be able to integrate and test it on AM65x CPSW driver and provide my comments/ Tested-by:

Understood. Thanks.

Signature · 1 line
Vinicius