net/mlx5: Preserve speed and state across vport modify commands
Subsystems (6)
Tariq Toukan
maintainercover net/mlx5: Preserve speed and state across vport modify commands
Hi,
The firmware vport modify command bundles both admin state and max tx speed in a single operation, which requires each side to preserve the other field when it only intends to change one.
When modifying max tx speed, the driver already queries the current admin state and passes it back to avoid overwriting it. However, this query and the subsequent modify were not atomic, a state change between the two could cause the modify to overwrite the new state with a stale value. The fix holds esw->state_lock across the query-modify sequence.
When support for setting max tx speed via the vport modify command was introduced, the existing admin state modify path was not updated to preserve the current speed. As a result, the firmware interprets the zero speed field as an intentional reset. The fix adds a speed query before the state modify and passes the result back in the command.
To support that, mlx5_query_vport_max_tx_speed() had to be fixed first:
it was returning zero whenever the vport was DOWN, which was correct for the query_port_speed verb but would defeat the purpose of querying before a state modify. The DOWN-to-zero logic is moved to the verb-layer caller so the function returns the raw firmware value.
Patch #1 holds esw->state_lock across the state query and modify in
the speed modify path
Patch #2 moves the vport DOWN zero mapping to the verb-layer caller
so the query returns the raw firmware value
Patch #3 queries current max tx speed before modifying vport state to
preserve it
Regards,
Tariq
Mark Bloch (1):
net/mlx5: E-Switch, use state lock for vport state changes
Or Har-Toov (2):
net/mlx5: Move vport DOWN state check out of
mlx5_query_vport_max_tx_speed()
net/mlx5: E-Switch, preserve max tx speed on vport state modification
drivers/infiniband/hw/mlx5/main.c | 5 ++- .../net/ethernet/mellanox/mlx5/core/en_rep.c | 20 ++++++--- .../mellanox/mlx5/core/esw/adj_vport.c | 22 ++++++++++ .../mellanox/mlx5/core/eswitch_offloads.c | 4 ++ .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 2 + .../net/ethernet/mellanox/mlx5/core/vport.c | 44 +++++++++++++++---- include/linux/mlx5/vport.h | 3 +- 7 files changed, 84 insertions(+), 16 deletions(-)
base-commit: 24ef02f934eeb48830cff6b739abc3c62b1d107b
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH 1/3 net/mlx5: E-Switch, use state lock for vport state changes
From: Mark Bloch <[email protected]>
Protect vport admin state modifications and vport iteration with the eswitch state_lock mutex to ensure proper serialization of concurrent vport state changes.
Currently, calls to mlx5_modify_vport_admin_state() and loops iterating over eswitch vports can race with each other, potentially leading to inconsistent vport state. Fix this by acquiring esw->state_lock
--- .../net/ethernet/mellanox/mlx5/core/en_rep.c | 20 ++++++++++++++----- .../mellanox/mlx5/core/esw/adj_vport.c | 2 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 4 ++++ .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 2 ++ .../net/ethernet/mellanox/mlx5/core/vport.c | 8 ++++++++ 5 files changed, 31 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_rep.c +15 -5
@@ -689,11 +689,13 @@ static int mlx5e_rep_open(struct net_device *dev)if (err) goto unlock; +mutex_lock (&rep->esw->state_lock);if (!mlx5_modify_vport_admin_state (priv->mdev,MLX5_VPORT_STATE_OP_MOD_ESW_VPORT , rep->vport,1 ,MLX5_VPORT_ADMIN_STATE_UP ))netif_carrier_on (dev); +mutex_unlock (&rep->esw->state_lock); unlock:mutex_unlock (&priv->state_lock);
@@ -708,10 +710,12 @@ static int mlx5e_rep_close(struct net_device *dev)int ret;mutex_lock (&priv->state_lock); +mutex_lock (&rep->esw->state_lock);mlx5_modify_vport_admin_state (priv->mdev,MLX5_VPORT_STATE_OP_MOD_ESW_VPORT , rep->vport,1 ,MLX5_VPORT_ADMIN_STATE_DOWN ); +mutex_unlock (&rep->esw->state_lock); ret =mlx5e_close_locked (dev);mutex_unlock (&priv->state_lock);return ret;
@@ -783,22 +787,25 @@ static int mlx5e_rep_change_carrier(struct net_device *dev, bool new_carrier)struct mlx5e_priv *priv =netdev_priv (dev);struct mlx5e_rep_priv *rpriv = priv->ppriv;struct mlx5_eswitch_rep *rep = rpriv->rep; -int err; +int err =0 ; +mutex_lock (&rep->esw->state_lock);if (new_carrier) { err =mlx5_modify_vport_admin_state (priv->mdev,MLX5_VPORT_STATE_OP_MOD_ESW_VPORT , rep->vport,1 ,MLX5_VPORT_ADMIN_STATE_UP );if (err) -return err; + goto unlock;netif_carrier_on (dev); }else { err =mlx5_modify_vport_admin_state (priv->mdev,MLX5_VPORT_STATE_OP_MOD_ESW_VPORT , rep->vport,1 ,MLX5_VPORT_ADMIN_STATE_DOWN );if (err) -return err; + goto unlock;netif_carrier_off (dev); } -return 0 ; +unlock: +mutex_unlock (&rep->esw->state_lock); +return err; }static const struct net_device_ops mlx5e_netdev_ops_rep = {
@@ -1337,9 +1344,12 @@ static void mlx5e_uplink_rep_enable(struct mlx5e_priv *priv)mlx5e_rep_tc_enable (priv); -if (MLX5_CAP_GEN (mdev, uplink_follow)) +if (MLX5_CAP_GEN (mdev, uplink_follow)) { +mutex_lock (&mdev->priv.eswitch->state_lock);mlx5_modify_vport_admin_state (mdev,MLX5_VPORT_STATE_OP_MOD_UPLINK ,0 ,0 ,MLX5_VPORT_ADMIN_STATE_AUTO ); +mutex_unlock (&mdev->priv.eswitch->state_lock); + }mlx5_lag_add_netdev (mdev, netdev); priv->events_nb.notifier_call = uplink_rep_async_event;mlx5_notifier_register (mdev, &priv->events_nb);
drivers/net/ethernet/mellanox/mlx5/core/esw/adj_vport.c +2 -0
@@ -9,6 +9,8 @@ int mlx5_esw_adj_vport_modify(struct mlx5_core_dev *dev, u16 vport, {u32 in[MLX5_ST_SZ_DW (modify_vport_state_in)] = {}; +lockdep_assert_held (&dev->priv.eswitch->state_lock); +MLX5_SET (modify_vport_state_in, in, opcode,MLX5_CMD_OP_MODIFY_VPORT_STATE ); MLX5_SET(modify_vport_state_in, in, op_mod,
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +4 -0
@@ -2567,6 +2567,7 @@ static void mlx5_esw_fdb_active(struct mlx5_eswitch *esw)mlx5_esw_fdb_drop_destroy (esw);mlx5_mpfs_enable (esw->dev); +mutex_lock (&esw->state_lock);mlx5_esw_for_each_vf_vport (esw, i, vport,U16_MAX ) {if (!vport->adjacent)continue ;
@@ -2574,6 +2575,7 @@ static void mlx5_esw_fdb_active(struct mlx5_eswitch *esw) vport->vport);mlx5_esw_adj_vport_modify (esw->dev, vport->vport, true); } +mutex_unlock (&esw->state_lock); esw->offloads_inactive = false;esw_warn (esw->dev,"MPFS/FDB active\n" );
@@ -2587,6 +2589,7 @@ static void mlx5_esw_fdb_inactive(struct mlx5_eswitch *esw)mlx5_mpfs_disable (esw->dev);mlx5_esw_fdb_drop_create (esw); +mutex_lock (&esw->state_lock);mlx5_esw_for_each_vf_vport (esw, i, vport,U16_MAX ) {if (!vport->adjacent)continue ;
@@ -2595,6 +2598,7 @@ static void mlx5_esw_fdb_inactive(struct mlx5_eswitch *esw)mlx5_esw_adj_vport_modify (esw->dev, vport->vport, false); } +mutex_unlock (&esw->state_lock); esw->offloads_inactive = true;esw_warn (esw->dev,"MPFS/FDB inactive\n" );
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c +2 -0
@@ -1471,6 +1471,7 @@ static void mlx5_lag_modify_device_vports_speed(struct mlx5_core_dev *mdev,if (!MLX5_CAP_ESW (mdev, esw_vport_state_max_tx_speed))return ; +mutex_lock (&esw->state_lock);mlx5_esw_for_each_vport (esw, i, vport) {if (!vport)continue ;
@@ -1490,6 +1491,7 @@ static void mlx5_lag_modify_device_vports_speed(struct mlx5_core_dev *mdev,"Failed to set vport %d speed %d, err=%d\n" , vport->vport, speed, ret); } +mutex_unlock (&esw->state_lock); }void mlx5_lag_set_vports_agg_speed (struct mlx5_lag *ldev)
drivers/net/ethernet/mellanox/mlx5/core/vport.c +8 -0
@@ -89,6 +89,10 @@ int mlx5_modify_vport_admin_state(struct mlx5_core_dev *mdev, u8 opmod, {u32 in[MLX5_ST_SZ_DW (modify_vport_state_in)] = {}; +#ifdef CONFIG_MLX5_ESWITCH +lockdep_assert_held (&mdev->priv.eswitch->state_lock); +#endif +MLX5_SET (modify_vport_state_in, in, opcode,MLX5_CMD_OP_MODIFY_VPORT_STATE );MLX5_SET (modify_vport_state_in, in, op_mod, opmod);
@@ -106,6 +110,10 @@ int mlx5_modify_vport_max_tx_speed(struct mlx5_core_dev *mdev, u8 opmod,u8 admin_state;int err; +#ifdef CONFIG_MLX5_ESWITCH +lockdep_assert_held (&mdev->priv.eswitch->state_lock); +#endif + err =mlx5_query_vport_admin_state (mdev, opmod, vport, other_vport, &admin_state);if (err)
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH 2/3 net/mlx5: Move vport DOWN state check out of mlx5_query_vport_max_tx_speed()
In reply to Tariq ToukanFrom: Or Har-Toov <[email protected]>
mlx5_query_vport_max_tx_speed() was introduced to serve the query_port_speed path, which uses max_tx_speed == 0 when port is down.
This is incorrect for callers that need the actual configured speed regardless of vport state, such as modify-vport-state helpers that must preserve the speed across state transitions.
Move this logic to the caller function in the verb flow and let mlx5_query_vport_max_tx_speed() return the raw firmware value unconditionally.
--- drivers/infiniband/hw/mlx5/main.c | 5 +++-- drivers/net/ethernet/mellanox/mlx5/core/vport.c | 12 ++++-------- include/linux/mlx5/vport.h | 3 ++- 3 files changed, 9 insertions(+), 11 deletions(-)
drivers/infiniband/hw/mlx5/main.c +3 -2
@@ -1631,14 +1631,15 @@ static int mlx5_ib_query_port_speed_from_vport(struct mlx5_core_dev *mdev, u32 port_num) {u32 max_tx_speed; +u8 vport_state;int err; err =mlx5_query_vport_max_tx_speed (mdev, op_mod, vport, other_vport, - &max_tx_speed); + &max_tx_speed, &vport_state);if (err)return err; -if (max_tx_speed ==0 ) +if (vport_state ==VPORT_STATE_DOWN || max_tx_speed ==0 )/* Value 0 indicates field not supported, fallback */ return mlx5_ib_query_port_speed_from_port (dev, port_num, speed);
drivers/net/ethernet/mellanox/mlx5/core/vport.c +4 -8
@@ -131,11 +131,11 @@ int mlx5_modify_vport_max_tx_speed(struct mlx5_core_dev *mdev, u8 opmod, }int mlx5_query_vport_max_tx_speed (struct mlx5_core_dev *mdev,u8 op_mod, -u16 vport,u8 other_vport,u32 *max_tx_speed) +u16 vport,u8 other_vport, +u32 *max_tx_speed,u8 *state) {u32 out[MLX5_ST_SZ_DW (query_vport_state_out)] = {};u32 in[MLX5_ST_SZ_DW (query_vport_state_in)] = {}; -u32 state;int err; MLX5_SET(query_vport_state_in, in, opcode,
@@ -148,13 +148,9 @@ int mlx5_query_vport_max_tx_speed(struct mlx5_core_dev *mdev, u8 op_mod,if (err)return err; - state =MLX5_GET (query_vport_state_out, out, state); -if (state ==VPORT_STATE_DOWN ) { - *max_tx_speed =0 ; -return 0 ; - } - *max_tx_speed =MLX5_GET (query_vport_state_out, out, max_tx_speed); +if (state) + *state =MLX5_GET (query_vport_state_out, out, state);return 0 ; }EXPORT_SYMBOL_GPL (mlx5_query_vport_max_tx_speed);
include/linux/mlx5/vport.h +2 -1
@@ -61,7 +61,8 @@ u8 mlx5_query_vport_state(struct mlx5_core_dev *mdev, u8 opmod, u16 vport);int mlx5_modify_vport_admin_state (struct mlx5_core_dev *mdev,u8 opmod,u16 vport,u8 other_vport,u8 state);int mlx5_query_vport_max_tx_speed (struct mlx5_core_dev *mdev,u8 op_mod, -u16 vport,u8 other_vport,u32 *max_tx_speed); +u16 vport,u8 other_vport, +u32 *max_tx_speed,u8 *state);int mlx5_modify_vport_max_tx_speed (struct mlx5_core_dev *mdev,u8 opmod,u16 vport,u8 other_vport,u16 max_tx_speed); int mlx5_query_nic_vport_mac_address(struct mlx5_core_dev *mdev,
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH 3/3 net/mlx5: E-Switch, preserve max tx speed on vport state modification
In reply to Tariq ToukanFrom: Or Har-Toov <[email protected]>
When modifying vport state, the firmware interprets a zero in the max tx speed field as an intentional reset, which can overwrite previously set values. This patch attempts to fix this by querying the current max tx speed from firmware before modifying the vport state and passing it back in the modification command. If the query fails, fall back to the cached agg_max_tx_speed value to avoid inadvertently resetting the speed.
--- .../mellanox/mlx5/core/esw/adj_vport.c | 20 ++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/vport.c | 24 +++++++++++++++++++ 2 files changed, 44 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/esw/adj_vport.c +20 -0
@@ -11,6 +11,26 @@ int mlx5_esw_adj_vport_modify(struct mlx5_core_dev *dev, u16 vport,lockdep_assert_held (&dev->priv.eswitch->state_lock); +if (MLX5_CAP_ESW (dev, esw_vport_state_max_tx_speed)) { +u8 op_mod =MLX5_VPORT_STATE_OP_MOD_ESW_VPORT ; +struct mlx5_vport *esw_vport; +u32 speed =0 ; +int err; + + err =mlx5_query_vport_max_tx_speed (dev, op_mod, vport, + true, &speed,NULL ); +if (err) { + esw_vport =mlx5_eswitch_get_vport (dev->priv.eswitch, + vport); + speed =IS_ERR (esw_vport) ?0 : + esw_vport->agg_max_tx_speed; +mlx5_core_dbg (dev, +"Failed to query vport %d max tx speed, err=%d, using cached %u\n" , + vport, err, speed); + } +MLX5_SET (modify_vport_state_in, in, max_tx_speed, speed); + } +MLX5_SET (modify_vport_state_in, in, opcode,MLX5_CMD_OP_MODIFY_VPORT_STATE ); MLX5_SET(modify_vport_state_in, in, op_mod,
drivers/net/ethernet/mellanox/mlx5/core/vport.c +24 -0
@@ -93,6 +93,30 @@ int mlx5_modify_vport_admin_state(struct mlx5_core_dev *mdev, u8 opmod,lockdep_assert_held (&mdev->priv.eswitch->state_lock);#endif +if (MLX5_CAP_ESW (mdev, esw_vport_state_max_tx_speed) && + opmod ==MLX5_VPORT_STATE_OP_MOD_ESW_VPORT && + vport !=MLX5_VPORT_UPLINK ) { +u32 speed =0 ; +int err; + + err =mlx5_query_vport_max_tx_speed (mdev, opmod, vport, + other_vport, &speed,NULL ); +if (err) { +#ifdef CONFIG_MLX5_ESWITCH +struct mlx5_vport *esw_vport; + + esw_vport =mlx5_eswitch_get_vport (mdev->priv.eswitch, + vport); + speed =IS_ERR (esw_vport) ?0 : + esw_vport->agg_max_tx_speed; +#endif +mlx5_core_dbg (mdev, +"Failed to query vport %d max tx speed, err=%d, using cached %u\n" , + vport, err, speed); + } +MLX5_SET (modify_vport_state_in, in, max_tx_speed, speed); + } +MLX5_SET (modify_vport_state_in, in, opcode,MLX5_CMD_OP_MODIFY_VPORT_STATE );MLX5_SET (modify_vport_state_in, in, op_mod, opmod);
Signature · 1 line
2.44.0
Quoted 25 lines · Tariq Toukan
Hi,
The firmware vport modify command bundles both admin state and max tx speed in a single operation, which requires each side to preserve the other field when it only intends to change one.
When modifying max tx speed, the driver already queries the current admin state and passes it back to avoid overwriting it. However, this query and the subsequent modify were not atomic, a state change between the two could cause the modify to overwrite the new state with a stale value. The fix holds
esw->state_lockacross the query-modify sequence.When support for setting max tx speed via the vport modify command was introduced, the existing admin state modify path was not updated to preserve the current speed. As a result, the firmware interprets the zero speed field as an intentional reset. The fix adds a speed query before the state modify and passes the result back in the command.
To support that,
mlx5_query_vport_max_tx_speed()had to be fixed first:
it was returning zero whenever the vport was DOWN, which was correct for thequery_port_speedverb but would defeat the purpose of querying before a state modify. The DOWN-to-zero logic is moved to the verb-layer caller so the function returns the raw firmware value.
Quoted 6 lines
Patch #1 holds
esw->state_lockacross the state query and modify in
the speed modify path
Patch #2 moves the vport DOWN zero mapping to the verb-layer caller
so the query returns the raw firmware value
Patch #3 queries current max tx speed before modifying vport state to
preserve it
For the series:
This series was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:
Quoted 6 lines · On Sun, 16 Aug 2026 09:50:12 +0300 you wrote:
Hi,
The firmware vport modify command bundles both admin state and max tx speed in a single operation, which requires each side to preserve the other field when it only intends to change one.
Quoted 8 lines
When modifying max tx speed, the driver already queries the current admin state and passes it back to avoid overwriting it. However, this query and the subsequent modify were not atomic, a state change between the two could cause the modify to overwrite the new state with a stale value. The fix holds
esw->state_lockacross the query-modify sequence.[...]
Here is the summary with links:
- [net,1/3] net/mlx5: E-Switch, use state lock for vport state changes
https://git.kernel.org/netdev/net/c/ff0f9b7aa1c7 - [net,2/3] net/mlx5: Move vport DOWN state check out of mlx5_query_vport_max_tx_speed() https://git.kernel.org/netdev/net/c/20f11b5cfa42
- [net,3/3] net/mlx5: E-Switch, preserve max tx speed on vport state modification
https://git.kernel.org/netdev/net/c/ad0ae7aefa7a
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html