Skip to contentSkip to conversation

net/mlx5: E-Switch, Size FDB peer groups for asymmetric peers

PATCH New Tariq Toukan 3 participants 3 messages Nov 16 – Nov 18

Subsystems (5)
lore
Share & tools
lore

Tariq Toukan

author maintainer
#0 lore raw

PATCH 1/1 net/mlx5: E-Switch, Size FDB peer groups for asymmetric peers

From: Uri Dar <[email protected]>

The slow path FDB send-to-vport and peer-miss groups are sized from the local eswitch total_vports, assuming every eswitch that may pair has the same number of vports. Per-PF VF configuration (e.g. PF_NUM_OF_VF set differently per port) breaks that assumption: the larger peer installs rules sized by its own total_vports into the smaller peer's groups, which cannot hold them, so pairing fails.

In SW and HW steering the flow table is unbounded (max_fte is INT_MAX) and flow groups are backed by dynamically grown matchers, so size the peer dependent groups to the architectural vport maximum. FW steering, whose table is bounded by log_max_ft_size, keeps the current sizing.

---
 .../mellanox/mlx5/core/eswitch_offloads.c       | 17 +++++++++++++++--
 include/linux/mlx5/eswitch.h                    |  2 ++
 2 files changed, 17 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +15 -2
index 474784131091..20eca8b89b8f 100644
@@ -1744,6 +1744,17 @@ esw_chains_destroy(struct mlx5_eswitch *esw, struct mlx5_fs_chains *chains)
 
 #endif
 
+/* SW and HW steering tables are unbounded, so they can be sized for any peer.
+ * FW steering is bounded by log_max_ft_size, so it uses local total_vports.
+ */
+static int esw_max_vports(const struct mlx5_eswitch *esw)
+{
+	if (esw->dev->priv.steering->mode == MLX5_FLOW_STEERING_MODE_DMFS)
+		return esw->total_vports;
+
+	return MLX5_ESW_MAX_VPORT_ID;
+}
+
 static int
 esw_create_send_to_vport_group(struct mlx5_eswitch *esw,
 			       struct mlx5_flow_table *fdb,
@@ -1771,7 +1782,9 @@ esw_create_send_to_vport_group(struct mlx5_eswitch *esw,
 	}
 
 	/* See comment at table_size calculation */
-	count = MLX5_MAX_PORTS * (esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ);
+	count = MLX5_MAX_PORTS *
+		(esw_max_vports(esw) * MAX_SQ_NVPORTS + MAX_PF_SQ);
+
 	MLX5_SET(create_flow_group_in, flow_group_in, start_flow_index, 0);
 	MLX5_SET(create_flow_group_in, flow_group_in, end_flow_index, *ix + count - 1);
 	*ix += count;
@@ -1841,7 +1854,7 @@ esw_create_peer_esw_miss_group(struct mlx5_eswitch *esw,
 			       u32 *flow_group_in,
 			       int *ix)
 {
-	int max_peer_ports = (esw->total_vports - 1) * (MLX5_MAX_PORTS - 1);
+	int max_peer_ports = (esw_max_vports(esw) - 1) * (MLX5_MAX_PORTS - 1);
 	int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
 	struct mlx5_flow_group *g;
 	void *match_criteria;
include/linux/mlx5/eswitch.h +2 -0
index 03d3620141c8..cfc52d0697e4 100644
@@ -103,6 +103,8 @@ bool mlx5_eswitch_vport_match_metadata_enabled(const struct mlx5_eswitch *esw);
 #define ESW_REG_C0_USER_DATA_METADATA_BITS (32 - ESW_SOURCE_PORT_METADATA_BITS)
 #define ESW_REG_C0_USER_DATA_METADATA_MASK GENMASK(ESW_REG_C0_USER_DATA_METADATA_BITS - 1, 0)
 
+#define MLX5_ESW_MAX_VPORT_ID ((1 << ESW_VPORT_BITS) - 1)
+
 static inline u32 mlx5_eswitch_get_vport_metadata_mask(void)
 {
 	return GENMASK(31, 32 - ESW_SOURCE_PORT_METADATA_BITS);

base-commit: 7ccc5e05043b1bb84326b19a2b05938d8f7a2ae6

Signature · 1 line
2.44.0
#1 lore raw
Sashiko has reviewed this patch and found no issues.
Quoted 14 lines

From: Uri Dar <[email protected]>

The slow path FDB send-to-vport and peer-miss groups are sized from the local eswitch total_vports, assuming every eswitch that may pair has the same number of vports. Per-PF VF configuration (e.g. PF_NUM_OF_VF set differently per port) breaks that assumption: the larger peer installs rules sized by its own total_vports into the smaller peer's groups, which cannot hold them, so pairing fails.

In SW and HW steering the flow table is unbounded (max_fte is INT_MAX) and flow groups are backed by dynamically grown matchers, so size the peer dependent groups to the architectural vport maximum. FW steering, whose table is bounded by log_max_ft_size, keeps the current sizing.

Quoted 4 lines

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Tariq Toukan
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider.

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider.

Critical: 0 · High: 0 · Medium: 1 · Low: 1

If you believe none of the issues are legitimate, please reply to each issue of Medium severity or higher and explain why the AI is wrong. If any of them are legitimate and you plan to respin, please let patchwork know by including "pw-bot: cr" as a separate line at the end of your reply (one such reply per series is enough).

- [Medium] In SMFS/HMFS mode, esw_max_vports() returns the fixed MLX5_ESW_MAX_VPORT_ID (4095) and ignores esw->total_vports.
- [Low] The patch changed how the send-to-vport and peer-miss groups are sized, but left the comment and the table_size formula in…

--

net/mlx5: E-Switch, Size FDB peer groups for asymmetric peers

The patch changes how the slow path FDB send-to-vport and peer-miss groups are sized in SW and HW steering modes. They now use a fixed architectural vport maximum, MLX5_ESW_MAX_VPORT_ID, instead of the local total_vports. This lets asymmetric peers pair. FW steering keeps the existing sizing.

Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +0 -0
index 4747841310914..20eca8b89b8f4 100644
@@ -1744,6 +1744,17 @@ esw_chains_destroy(struct mlx5_eswitch *esw, struct mlx5_fs_chains *chains)
 
 #endif
 

drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c esw_chains_destroy(struct mlx5_eswitch *esw, struct mlx5_fs_chains *chains) · line 1756

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +10 -0
+/* SW and HW steering tables are unbounded, so they can be sized for any peer.
+ * FW steering is bounded by log_max_ft_size, so it uses local total_vports.
+ */
+static int esw_max_vports(const struct mlx5_eswitch *esw)
+{
+	if (esw->dev->priv.steering->mode == MLX5_FLOW_STEERING_MODE_DMFS)
+		return esw->total_vports;
+
+	return MLX5_ESW_MAX_VPORT_ID;
+}

[Severity: Medium]
In SMFS/HMFS mode, can esw_max_vports() return less than the old sizing?

MLX5_ESW_MAX_VPORT_ID is 4095, which is the width of the ESW_VPORT_BITS field in the reg_c_0 source-port metadata. It limits the number of vports only when vport-match metadata is enabled. In that case esw_offloads_vport_metadata_setup() fails with -ENOSPC if there are more than 4095 vports:

	return vport->metadata ? 0 : -ENOSPC;

Metadata matching can be turned off, though. One way is the esw_port_metadata devlink parameter. Another is a device without metadata support. The FDB then matches on the 16-bit source_port/source_sqn space, and nothing limits total_vports to 4095. total_vports comes from firmware (VFs plus mlx5_sf_max_functions() and the host-PF/SPF SFs and EC VFs).

If total_vports goes above 4095 in that configuration, both groups end up smaller than before:

send-to-vport: 8 * (4095 * 32 + 256) = 1,050,368 entries,

                 versus 8 * (32 * N + 256) before

peer-miss: 4094 * 7 = 28,658 entries,
versus (N - 1) * 7 before

Once the number of rules goes past the new capacity, would insert_fte() fail here?

	index = ida_alloc_max(&fg->fte_allocator, fg->max_ftes - 1, GFP_KERNEL);

Peer-miss or send-to-vport rule installation would then fail, where the old sizing worked.

Would max(esw->total_vports, MLX5_ESW_MAX_VPORT_ID) keep the old sizing as a lower bound?

A smaller point: the new public macro is named MLX5_ESW_MAX_VPORT_ID, but real vport numbers are 16-bit (MLX5_VPORT_UPLINK is 0xffff and MLX5_VPORT_ECPF is 0xfffe). Here it is used as a vport count, not an ID. Would a name tied to the metadata field width be clearer?

drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c esw_create_send_to_vport_group(struct mlx5_eswitch *esw, · line 1787

Quoted 12 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +4 -1
+
 static int
 esw_create_send_to_vport_group(struct mlx5_eswitch *esw,
 			       struct mlx5_flow_table *fdb,
@@ -1771,7 +1782,9 @@ esw_create_send_to_vport_group(struct mlx5_eswitch *esw,
 	}
 
 	/* See comment at table_size calculation */
-	count = MLX5_MAX_PORTS * (esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ);
+	count = MLX5_MAX_PORTS *
+		(esw_max_vports(esw) * MAX_SQ_NVPORTS + MAX_PF_SQ);
+

[Severity: Low]
This isn't a bug, but should the comment and the table_size calculation in esw_create_offloads_fdb_tables() be updated as well? The cross-reference above now points at text that no longer matches this group:

	/* To be strictly correct:
	 *	MLX5_MAX_PORTS * (esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ)
	 [ ... ]
	 * the peer miss group should also be created based on the number of
	 * total vports of the peer (currently is also uses esw->total_vports).
	 */
	table_size = MLX5_MAX_PORTS * (esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ) +
		     esw->total_vports * MLX5_MAX_PORTS + MLX5_ESW_MISS_FLOWS;

In SMFS/HMFS, both the send-to-vport group and the peer-miss group now use esw_max_vports(). table_size, and so ft_attr.max_fte, is still computed from total_vports. The group index layout therefore goes well past the max_fte the table asks for.

This works today only because mlx5_cmd_dr_create_flow_table() and the HWS create path overwrite the value:

	ft->max_fte = INT_MAX;

Nothing at the table_size site records that dependency. Could the comment mention it?

Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +0 -0
 	MLX5_SET(create_flow_group_in, flow_group_in, start_flow_index, 0);
 	MLX5_SET(create_flow_group_in, flow_group_in, end_flow_index, *ix + count - 1);
 	*ix += count;

[ ... ]

Signature · 1 line
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261116210917.1297111-1-tariqt%40nvidia.com