net/mlx5: E-Switch, Size FDB peer groups for asymmetric peers
Subsystems (5)
Tariq Toukan
maintainerPATCH 1/1 net/mlx5: E-Switch, Size FDB peer groups for asymmetric peers
From: Uri Dar <[email protected]>
The slow path FDB send-to-vport and peer-miss groups are sized from the local eswitch total_vports, assuming every eswitch that may pair has the same number of vports. Per-PF VF configuration (e.g. PF_NUM_OF_VF set differently per port) breaks that assumption: the larger peer installs rules sized by its own total_vports into the smaller peer's groups, which cannot hold them, so pairing fails.
In SW and HW steering the flow table is unbounded (max_fte is INT_MAX) and flow groups are backed by dynamically grown matchers, so size the peer dependent groups to the architectural vport maximum. FW steering, whose table is bounded by log_max_ft_size, keeps the current sizing.
--- .../mellanox/mlx5/core/eswitch_offloads.c | 17 +++++++++++++++-- include/linux/mlx5/eswitch.h | 2 ++ 2 files changed, 17 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +15 -2
@@ -1744,6 +1744,17 @@ esw_chains_destroy(struct mlx5_eswitch *esw, struct mlx5_fs_chains *chains)#endif +/* SW and HW steering tables are unbounded, so they can be sized for any peer. +* FW steering is bounded by log_max_ft_size, so it uses local total_vports. +*/ +static int esw_max_vports (const struct mlx5_eswitch *esw) +{ +if (esw->dev->priv.steering->mode ==MLX5_FLOW_STEERING_MODE_DMFS ) +return esw->total_vports; + +return MLX5_ESW_MAX_VPORT_ID ; +} + static int esw_create_send_to_vport_group(struct mlx5_eswitch *esw, struct mlx5_flow_table *fdb,
@@ -1771,7 +1782,9 @@ esw_create_send_to_vport_group(struct mlx5_eswitch *esw, }/* See comment at table_size calculation */ - count =MLX5_MAX_PORTS * (esw->total_vports *MAX_SQ_NVPORTS +MAX_PF_SQ ); + count =MLX5_MAX_PORTS * + (esw_max_vports (esw) *MAX_SQ_NVPORTS +MAX_PF_SQ ); +MLX5_SET (create_flow_group_in, flow_group_in, start_flow_index,0 );MLX5_SET (create_flow_group_in, flow_group_in, end_flow_index, *ix + count -1 ); *ix += count;
@@ -1841,7 +1854,7 @@ esw_create_peer_esw_miss_group(struct mlx5_eswitch *esw, u32 *flow_group_in, int *ix) { -int max_peer_ports = (esw->total_vports -1 ) * (MLX5_MAX_PORTS -1 ); +int max_peer_ports = (esw_max_vports (esw) -1 ) * (MLX5_MAX_PORTS -1 );int inlen =MLX5_ST_SZ_BYTES (create_flow_group_in);struct mlx5_flow_group *g;void *match_criteria;
include/linux/mlx5/eswitch.h +2 -0
@@ -103,6 +103,8 @@ bool mlx5_eswitch_vport_match_metadata_enabled(const struct mlx5_eswitch *esw);#define ESW_REG_C0_USER_DATA_METADATA_BITS (32 - ESW_SOURCE_PORT_METADATA_BITS)#define ESW_REG_C0_USER_DATA_METADATA_MASK GENMASK(ESW_REG_C0_USER_DATA_METADATA_BITS - 1, 0) +#define MLX5_ESW_MAX_VPORT_ID ((1 << ESW_VPORT_BITS) - 1) +static inline u32 mlx5_eswitch_get_vport_metadata_mask (void ) {return GENMASK (31 ,32 -ESW_SOURCE_PORT_METADATA_BITS );
base-commit: 7ccc5e05043b1bb84326b19a2b05938d8f7a2ae6
Signature · 1 line
2.44.0
[email protected]
AI reviewSashiko has reviewed this patch and found no issues.
Quoted 14 lines
From: Uri Dar <[email protected]>
The slow path FDB send-to-vport and peer-miss groups are sized from the local eswitch
total_vports, assuming every eswitch that may pair has the same number of vports. Per-PF VF configuration (e.g.PF_NUM_OF_VFset differently per port) breaks that assumption: the larger peer installs rules sized by its owntotal_vportsinto the smaller peer's groups, which cannot hold them, so pairing fails.In SW and HW steering the flow table is unbounded (
max_fteisINT_MAX) and flow groups are backed by dynamically grown matchers, so size the peer dependent groups to the architectural vport maximum. FW steering, whose table is bounded bylog_max_ft_size, keeps the current sizing.
Quoted 4 lines
Signed-off-by Uri Dar <[email protected]>Reviewed-by Moshe Shemesh <[email protected]>Reviewed-by Shay Drori <[email protected]>Signed-off-by Tariq Toukan <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider.
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 · Low: 1
If you believe none of the issues are legitimate, please reply to each issue of Medium severity or higher and explain why the AI is wrong. If any of them are legitimate and you plan to respin, please let patchwork know by including "pw-bot: cr" as a separate line at the end of your reply (one such reply per series is enough).
- [Medium] In SMFS/HMFS mode, esw_max_vports() returns the fixed MLX5_ESW_MAX_VPORT_ID (4095) and ignores esw->total_vports.
- [Low] The patch changed how the send-to-vport and peer-miss groups are sized, but left the comment and the table_size formula in…
--
net/mlx5: E-Switch, Size FDB peer groups for asymmetric peers
The patch changes how the slow path FDB send-to-vport and peer-miss groups are sized in SW and HW steering modes. They now use a fixed architectural vport maximum, MLX5_ESW_MAX_VPORT_ID, instead of the local total_vports. This lets asymmetric peers pair. FW steering keeps the existing sizing.
Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +0
-0@@ -1744,6 +1744,17 @@ esw_chains_destroy(struct mlx5_eswitch *esw, struct mlx5_fs_chains *chains)#endif
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c esw_chains_destroy(struct mlx5_eswitch *esw, struct mlx5_fs_chains *chains) · line 1756
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +10
-0+/* SW and HW steering tables are unbounded, so they can be sized for any peer. +* FW steering is bounded by log_max_ft_size, so it uses local total_vports. +*/ +static int esw_max_vports (const struct mlx5_eswitch *esw) +{ +if (esw->dev->priv.steering->mode ==MLX5_FLOW_STEERING_MODE_DMFS ) +return esw->total_vports; + +return MLX5_ESW_MAX_VPORT_ID ; +}
[Severity: Medium]
In SMFS/HMFS mode, can esw_max_vports() return less than the old sizing?
MLX5_ESW_MAX_VPORT_ID is 4095, which is the width of the ESW_VPORT_BITS field in the reg_c_0 source-port metadata. It limits the number of vports only when vport-match metadata is enabled. In that case esw_offloads_vport_metadata_setup() fails with -ENOSPC if there are more than 4095 vports:
return vport->metadata ? 0 : -ENOSPC;
Metadata matching can be turned off, though. One way is the esw_port_metadata devlink parameter. Another is a device without metadata support. The FDB then matches on the 16-bit source_port/source_sqn space, and nothing limits total_vports to 4095. total_vports comes from firmware (VFs plus mlx5_sf_max_functions() and the host-PF/SPF SFs and EC VFs).
If total_vports goes above 4095 in that configuration, both groups end up smaller than before:
send-to-vport: 8 * (4095 * 32 + 256) = 1,050,368 entries,
versus 8 * (32 * N + 256) before
peer-miss: 4094 * 7 = 28,658 entries,
versus (N - 1) * 7 before
Once the number of rules goes past the new capacity, would insert_fte() fail here?
index = ida_alloc_max(&fg->fte_allocator, fg->max_ftes - 1, GFP_KERNEL);
Peer-miss or send-to-vport rule installation would then fail, where the old sizing worked.
Would max(esw->total_vports, MLX5_ESW_MAX_VPORT_ID) keep the old sizing as a lower bound?
A smaller point: the new public macro is named MLX5_ESW_MAX_VPORT_ID, but real vport numbers are 16-bit (MLX5_VPORT_UPLINK is 0xffff and MLX5_VPORT_ECPF is 0xfffe). Here it is used as a vport count, not an ID. Would a name tied to the metadata field width be clearer?
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c esw_create_send_to_vport_group(struct mlx5_eswitch *esw, · line 1787
Quoted 12 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +4
-1+ static int esw_create_send_to_vport_group(struct mlx5_eswitch *esw, struct mlx5_flow_table *fdb,@@ -1771,7 +1782,9 @@ esw_create_send_to_vport_group(struct mlx5_eswitch *esw, }/* See comment at table_size calculation */ - count =MLX5_MAX_PORTS * (esw->total_vports *MAX_SQ_NVPORTS +MAX_PF_SQ ); + count =MLX5_MAX_PORTS * + (esw_max_vports (esw) *MAX_SQ_NVPORTS +MAX_PF_SQ ); +
[Severity: Low]
This isn't a bug, but should the comment and the table_size calculation in esw_create_offloads_fdb_tables() be updated as well? The cross-reference above now points at text that no longer matches this group:
/* To be strictly correct: * MLX5_MAX_PORTS * (esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ) [ ... ] * the peer miss group should also be created based on the number of * total vports of the peer (currently is also uses esw->total_vports). */ table_size = MLX5_MAX_PORTS * (esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ) + esw->total_vports * MLX5_MAX_PORTS + MLX5_ESW_MISS_FLOWS;
In SMFS/HMFS, both the send-to-vport group and the peer-miss group now use esw_max_vports(). table_size, and so ft_attr.max_fte, is still computed from total_vports. The group index layout therefore goes well past the max_fte the table asks for.
This works today only because mlx5_cmd_dr_create_flow_table() and the HWS create path overwrite the value:
ft->max_fte = INT_MAX;
Nothing at the table_size site records that dependency. Could the comment mention it?
Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +0
-0MLX5_SET (create_flow_group_in, flow_group_in, start_flow_index,0 );MLX5_SET (create_flow_group_in, flow_group_in, end_flow_index, *ix + count -1 ); *ix += count;
[ ... ]
Signature · 1 line
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261116210917.1297111-1-tariqt%40nvidia.com