mlxsw: Handle MTU values

Amit Cohen writes:

The driver uses two values for maximum MTU, but neither is accurate. In addition, the value which is configured to hardware is not calculated correctly. Handle these issues and expose accurate values for minimum and maximum MTU per netdevice.

Add test cases to check that the exposed values are really supported.

Patch set overview:
Patches #1-#3 set the driver to use accurate values for MTU Patch #4 aligns the driver to always use the same value for maximum MTU Patch #5 adds a test

Amit Cohen (5):
mlxsw: port: Edit maximum MTU value
mlxsw: Adjust MTU value to hardware check
mlxsw: spectrum: Set more accurate values for netdevice min/max MTU
mlxsw: Use the same maximum MTU value throughout the driver
selftests: forwarding: Add test for minimum and maximum MTU

 drivers/net/ethernet/mellanox/mlxsw/port.h    |   3 +-
 .../net/ethernet/mellanox/mlxsw/spectrum.c    |  31 +-
 .../net/ethernet/mellanox/mlxsw/spectrum.h    |   1 -
 .../mellanox/mlxsw/spectrum_buffers.c         |   8 +-
 .../testing/selftests/net/forwarding/Makefile |   1 +
 .../selftests/net/forwarding/min_max_mtu.sh   | 283 ++++++++++++++++++
 6 files changed, 295 insertions(+), 32 deletions(-)
 create mode 100755 tools/testing/selftests/net/forwarding/min_max_mtu.sh

2.45.0

From: Amit Cohen <[email protected]>

Currently mlxsw driver supports up to 10000 bytes for maximum MTU, this value is not accurate, we can support up to 10K bytes. Change the value to the maximum supported MTU by firmware.

---
 drivers/net/ethernet/mellanox/mlxsw/port.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index ac4d4ea51597..aa309615eff3 100644
@@ -6,7 +6,7 @@
 
 #include <linux/types.h>
 
-#define MLXSW_PORT_MAX_MTU		10000
+#define MLXSW_PORT_MAX_MTU		(10 * 1024)
 
 #define MLXSW_PORT_DEFAULT_VID		1
 

2.45.0

From: Amit Cohen <[email protected]>

Ethernet frame consists of - Ethernet header, payload, FCS. The MTU value which is used by user is the size of the payload, which means that when user sets MTU to X, the total frame size will be larger due to the addition of the Ethernet header and FCS.

Spectrum ASICs take into account Ethernet header and FCS as part of packet size for MTU check. Adjust MTU value when user sets MTU, to configure the MTU size which is required by hardware. The Tx header length which was used by the driver is not relevant for such calculation, take into account Ethernet header (with VLAN extension) and FCS.

---
 drivers/net/ethernet/mellanox/mlxsw/port.h     | 1 +
 drivers/net/ethernet/mellanox/mlxsw/spectrum.c | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

index aa309615eff3..0a73b1a4526e 100644
@@ -7,6 +7,7 @@
 #include <linux/types.h>
 
 #define MLXSW_PORT_MAX_MTU		(10 * 1024)
+#define MLXSW_PORT_ETH_FRAME_HDR	(ETH_HLEN + VLAN_HLEN + ETH_FCS_LEN)
 
 #define MLXSW_PORT_DEFAULT_VID		1
 

index 030ed71f945d..879daa18ccca 100644
@@ -425,7 +425,7 @@ static int mlxsw_sp_port_mtu_set(struct mlxsw_sp_port *mlxsw_sp_port, u16 mtu)
 	struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;
 	char pmtu_pl[MLXSW_REG_PMTU_LEN];
 
-	mtu += MLXSW_TXHDR_LEN + ETH_HLEN;
+	mtu += MLXSW_PORT_ETH_FRAME_HDR;
 	if (mtu > mlxsw_sp_port->max_mtu)
 		return -EINVAL;
 

2.45.0

From: Amit Cohen <[email protected]>

Currently, the driver uses ETH_MAX_MTU as maximum MTU of netdevices, instead, use the accurate value which is supported by the driver. Subtract Ethernet headers which are taken into account by hardware for MTU checking, as described in the previous patch.

Set minimum MTU to ETH_MIN_MTU, as zero MTU is not really supported.

With this change:
a. The stack will do the MTU checking, so we can remove it from the driver. b. User space will be able to query the actual MTU limits.
Before this patch:

   $ ip -j -d link show dev swp1 | jq | grep mtu
    "mtu": 1500,
    "min_mtu": 0,
    "max_mtu": 65535,

With this patch:

   $ ip -j -d link show dev swp1 | jq | grep mtu
    "mtu": 1500,
    "min_mtu": 68,
    "max_mtu": 10218,

---
 drivers/net/ethernet/mellanox/mlxsw/spectrum.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

index 879daa18ccca..c5856f4d6b8b 100644
@@ -426,8 +426,6 @@ static int mlxsw_sp_port_mtu_set(struct mlxsw_sp_port *mlxsw_sp_port, u16 mtu)
 	char pmtu_pl[MLXSW_REG_PMTU_LEN];
 
 	mtu += MLXSW_PORT_ETH_FRAME_HDR;
-	if (mtu > mlxsw_sp_port->max_mtu)
-		return -EINVAL;
 
 	mlxsw_reg_pmtu_pack(pmtu_pl, mlxsw_sp_port->local_port, mtu);
 	return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(pmtu), pmtu_pl);

@@ -1697,8 +1695,8 @@ static int mlxsw_sp_port_create(struct mlxsw_sp *mlxsw_sp, u16 local_port,
 			 NETIF_F_HW_VLAN_CTAG_FILTER | NETIF_F_HW_TC;
 	dev->hw_features |= NETIF_F_HW_TC | NETIF_F_LOOPBACK;
 
-	dev->min_mtu = 0;
-	dev->max_mtu = ETH_MAX_MTU;
+	dev->min_mtu = ETH_MIN_MTU;
+	dev->max_mtu = MLXSW_PORT_MAX_MTU - MLXSW_PORT_ETH_FRAME_HDR;
 
 	/* Each packet needs to have a Tx header (metadata) on top all other
 	 * headers.

2.45.0

From: Amit Cohen <[email protected]>

Currently, the driver uses two different values for maximum MTU, one is stored in mlxsw_port->dev->max_mtu and the second is stored in mlxsw_port->max_mtu. The second one is set to value which is queried from firmware. This value was never tested, and unfortunately is not really supported. That means that with the existing code, user can set MTU to X, which is not really supported by firmware and which is bigger than buffer size which is allocated in pci.

To make the driver consistent, use only mlxsw_port->dev->max_mtu for maximum MTU value, for buffers headroom add Ethernet frame headers, which are not included in mlxsw_port->dev->max_mtu. Remove mlxsw_port->max_mtu.

---
 .../net/ethernet/mellanox/mlxsw/spectrum.c    | 23 -------------------
 .../net/ethernet/mellanox/mlxsw/spectrum.h    |  1 -
 .../mellanox/mlxsw/spectrum_buffers.c         |  8 +++++--
 3 files changed, 6 insertions(+), 26 deletions(-)

index c5856f4d6b8b..f064789f3240 100644
@@ -405,21 +405,6 @@ static int mlxsw_sp_port_dev_addr_init(struct mlxsw_sp_port *mlxsw_sp_port)
 					  mlxsw_sp_port->dev->dev_addr);
 }
 
-static int mlxsw_sp_port_max_mtu_get(struct mlxsw_sp_port *mlxsw_sp_port, int *p_max_mtu)
-{
-	struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;
-	char pmtu_pl[MLXSW_REG_PMTU_LEN];
-	int err;
-
-	mlxsw_reg_pmtu_pack(pmtu_pl, mlxsw_sp_port->local_port, 0);
-	err = mlxsw_reg_query(mlxsw_sp->core, MLXSW_REG(pmtu), pmtu_pl);
-	if (err)
-		return err;
-
-	*p_max_mtu = mlxsw_reg_pmtu_max_mtu_get(pmtu_pl);
-	return 0;
-}
-
 static int mlxsw_sp_port_mtu_set(struct mlxsw_sp_port *mlxsw_sp_port, u16 mtu)
 {
 	struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;

@@ -1725,13 +1710,6 @@ static int mlxsw_sp_port_create(struct mlxsw_sp *mlxsw_sp, u16 local_port,
 		goto err_max_speed_get;
 	}
 
-	err = mlxsw_sp_port_max_mtu_get(mlxsw_sp_port, &mlxsw_sp_port->max_mtu);
-	if (err) {
-		dev_err(mlxsw_sp->bus_info->dev, "Port %d: Failed to get maximum MTU\n",
-			mlxsw_sp_port->local_port);
-		goto err_port_max_mtu_get;
-	}
-
 	err = mlxsw_sp_port_mtu_set(mlxsw_sp_port, ETH_DATA_LEN);
 	if (err) {
 		dev_err(mlxsw_sp->bus_info->dev, "Port %d: Failed to set MTU\n",

@@ -1875,7 +1853,6 @@ static int mlxsw_sp_port_create(struct mlxsw_sp *mlxsw_sp, u16 local_port,
 err_port_buffers_init:
 err_port_admin_status_set:
 err_port_mtu_set:
-err_port_max_mtu_get:
 err_max_speed_get:
 err_port_speed_by_width_set:
 err_port_system_port_mapping_set:

index 3beb5d0847ab..bb0586b45c8d 100644
@@ -359,7 +359,6 @@ struct mlxsw_sp_port {
 		u16 egr_types;
 		struct mlxsw_sp_ptp_port_stats stats;
 	} ptp;
-	int max_mtu;
 	u32 max_speed;
 	struct mlxsw_sp_hdroom *hdroom;
 	u64 module_overheat_initial_val;

index ba090262e27e..2c0cfa79d138 100644
@@ -399,11 +399,13 @@ void mlxsw_sp_hdroom_bufs_reset_sizes(struct mlxsw_sp_port *mlxsw_sp_port,
 				      struct mlxsw_sp_hdroom *hdroom)
 {
 	struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;
+	unsigned int max_mtu = mlxsw_sp_port->dev->max_mtu;
 	u16 reserve_cells;
 	int i;
 
+	max_mtu += MLXSW_PORT_ETH_FRAME_HDR;
 	/* Internal buffer. */
-	reserve_cells = mlxsw_sp_hdroom_int_buf_size_get(mlxsw_sp, mlxsw_sp_port->max_mtu,
+	reserve_cells = mlxsw_sp_hdroom_int_buf_size_get(mlxsw_sp, max_mtu,
 							 mlxsw_sp_port->max_speed);
 	reserve_cells = mlxsw_sp_port_headroom_8x_adjust(mlxsw_sp_port, reserve_cells);
 	hdroom->int_buf.reserve_cells = reserve_cells;

@@ -613,7 +615,9 @@ static int mlxsw_sp_port_headroom_init(struct mlxsw_sp_port *mlxsw_sp_port)
 	mlxsw_sp_hdroom_bufs_reset_sizes(mlxsw_sp_port, &hdroom);
 
 	/* Buffer 9 is used for control traffic. */
-	size9 = mlxsw_sp_port_headroom_8x_adjust(mlxsw_sp_port, mlxsw_sp_port->max_mtu);
+	size9 = mlxsw_sp_port_headroom_8x_adjust(mlxsw_sp_port,
+						 mlxsw_sp_port->dev->max_mtu +
+						 MLXSW_PORT_ETH_FRAME_HDR);
 	hdroom.bufs.buf[9].size_cells = mlxsw_sp_bytes_cells(mlxsw_sp, size9);
 
 	return __mlxsw_sp_hdroom_configure(mlxsw_sp_port, &hdroom, true);

2.45.0

From: Amit Cohen <[email protected]>

Add cases to check minimum and maximum MTU which are exposed via "ip -d link show". Test configuration and traffic. Use VLAN devices as usually VLAN header (4 bytes) is not included in the MTU, and drivers should configure hardware correctly to send maximum MTU payload size in VLAN tagged packets.

$ ./min_max_mtu.sh
TEST: ping						[ OK ]
TEST: ping6						[ OK ]
TEST: Test maximum MTU configuration			[ OK ]
TEST: Test traffic, packet size is maximum MTU		[ OK ]
TEST: Test minimum MTU configuration			[ OK ]
TEST: Test traffic, packet size is minimum MTU		[ OK ]

---

 .../testing/selftests/net/forwarding/Makefile |   1 +
 .../selftests/net/forwarding/min_max_mtu.sh   | 283 ++++++++++++++++++
 2 files changed, 284 insertions(+)
 create mode 100755 tools/testing/selftests/net/forwarding/min_max_mtu.sh

index 99576d7ecbf6..224346426ef2 100644
@@ -39,6 +39,7 @@ TEST_PROGS = bridge_fdb_learning_limit.sh \
 	ipip_hier_gre.sh \
 	lib_sh_test.sh \
 	local_termination.sh \
+	min_max_mtu.sh \
 	mirror_gre_bound.sh \
 	mirror_gre_bridge_1d.sh \
 	mirror_gre_bridge_1d_vlan.sh \

2.45.0