Skip to contentSkip to conversation

mlx5 fixes

PATCH Applied Tariq Toukan 2 participants 8 messages Apr 11, 2024 – Apr 13, 2024

Subsystems (4)
lore
Share & tools
lore

Tariq Toukan

author maintainer
#0 lore raw

cover mlx5 fixes

Hi,

This patchset provides bug fixes to mlx5 core and Eth drivers.

Series generated against:
commit fe87922cee61 ("net/mlx5: fix possible stack overflows")

Thanks,
Tariq.

Carolina Jubran (2):
net/mlx5e: Acquire RTNL lock before RQs/SQs activation/deactivation net/mlx5e: Prevent deadlock while disabling aRFS

Rahul Rameshbabu (1):
net/mlx5e: Use channel mdev reference instead of global mdev instance
for coalescing

Shay Drory (2):
net/mlx5: Lag, restore buckets number to default after hash LAG
deactivation
net/mlx5: Restore mistakenly dropped parts in register devlink flow

Tariq Toukan (1):
net/mlx5: SD, Handle possible devcom ERR_PTR

 .../mellanox/mlx5/core/en/reporter_tx.c       |  7 +++++
 .../net/ethernet/mellanox/mlx5/core/en_arfs.c | 27 +++++++++++--------
 .../ethernet/mellanox/mlx5/core/en_ethtool.c  |  4 +--
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  4 +--
 .../mellanox/mlx5/core/eswitch_offloads.c     |  2 +-
 .../net/ethernet/mellanox/mlx5/core/lag/lag.c |  4 ++-
 .../ethernet/mellanox/mlx5/core/lib/devcom.c  |  2 +-
 .../net/ethernet/mellanox/mlx5/core/lib/sd.c  |  4 +--
 .../net/ethernet/mellanox/mlx5/core/main.c    |  7 +++--
 .../mellanox/mlx5/core/sf/dev/driver.c        |  1 -
 10 files changed, 39 insertions(+), 23 deletions(-)
Signature · 1 line
2.44.0
author maintainer
#1 lore raw

PATCH 1/6 net/mlx5: Lag, restore buckets number to default after hash LAG deactivation

From: Shay Drory <[email protected]>

The cited patch introduces the concept of buckets in LAG in hash mode. However, the patch doesn't clear the number of buckets in the LAG deactivation. This results in using the wrong number of buckets in case user create a hash mode LAG and afterwards create a non-hash mode LAG.

Hence, restore buckets number to default after hash mode LAG deactivation.

Fixes 352899f384d4 ("net/mlx5: Lag, use buckets in hash mode")
---
 drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c +3 -1
index d14459e5c04f..69d482f7c5a2 100644
@@ -703,8 +703,10 @@ int mlx5_deactivate_lag(struct mlx5_lag *ldev)
 		return err;
 	}
 
-	if (test_bit(MLX5_LAG_MODE_FLAG_HASH_BASED, &flags))
+	if (test_bit(MLX5_LAG_MODE_FLAG_HASH_BASED, &flags)) {
 		mlx5_lag_port_sel_destroy(ldev);
+		ldev->buckets = 1;
+	}
 	if (mlx5_lag_has_drop_rule(ldev))
 		mlx5_lag_drop_rule_cleanup(ldev);
 
Signature · 1 line
2.44.0
author maintainer
#2 lore raw

PATCH 2/6 net/mlx5: SD, Handle possible devcom ERR_PTR

In reply to Tariq Toukan

Check if devcom holds an error pointer and return immediately.

This fixes Smatch static checker warning:
drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c:221 sd_register()
error: 'devcom' dereferencing possible ERR_PTR()

Enhance mlx5_devcom_register_component() so it stops returning NULL, making it easier for its callers.

Fixes d3d057666090 ("net/mlx5: SD, Implement devcom communication and primary election")
---
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c          | 4 ++--
 drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c | 2 +-
 drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.c       | 2 +-
 drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c           | 4 ++--
 drivers/net/ethernet/mellanox/mlx5/core/main.c             | 2 +-
 5 files changed, 7 insertions(+), 7 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +2 -2
index b375ef268671..319930c04093 100644
@@ -209,8 +209,8 @@ static int mlx5e_devcom_init_mpv(struct mlx5e_priv *priv, u64 *data)
 						      *data,
 						      mlx5e_devcom_event_mpv,
 						      priv);
-	if (IS_ERR_OR_NULL(priv->devcom))
-		return -EOPNOTSUPP;
+	if (IS_ERR(priv->devcom))
+		return PTR_ERR(priv->devcom);
 
 	if (mlx5_core_is_mp_master(priv->mdev)) {
 		mlx5_devcom_send_event(priv->devcom, MPV_DEVCOM_MASTER_UP,
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +1 -1
index 1f60954c12f7..844d3e3a65dd 100644
@@ -3060,7 +3060,7 @@ void mlx5_esw_offloads_devcom_init(struct mlx5_eswitch *esw, u64 key)
 						     key,
 						     mlx5_esw_offloads_devcom_event,
 						     esw);
-	if (IS_ERR_OR_NULL(esw->devcom))
+	if (IS_ERR(esw->devcom))
 		return;
 
 	mlx5_devcom_send_event(esw->devcom,
drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.c +1 -1
index e7d59cfa8708..7b0766c89f4c 100644
@@ -220,7 +220,7 @@ mlx5_devcom_register_component(struct mlx5_devcom_dev *devc,
 	struct mlx5_devcom_comp *comp;
 
 	if (IS_ERR_OR_NULL(devc))
-		return NULL;
+		return ERR_PTR(-EINVAL);
 
 	mutex_lock(&comp_list_lock);
 	comp = devcom_component_get(devc, id, key, handler);
drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c +2 -2
index 5b28084e8a03..dd5d186dc614 100644
@@ -213,8 +213,8 @@ static int sd_register(struct mlx5_core_dev *dev)
 	sd = mlx5_get_sd(dev);
 	devcom = mlx5_devcom_register_component(dev->priv.devc, MLX5_DEVCOM_SD_GROUP,
 						sd->group_id, NULL, dev);
-	if (!devcom)
-		return -ENOMEM;
+	if (IS_ERR(devcom))
+		return PTR_ERR(devcom);
 
 	sd->devcom = devcom;
 
drivers/net/ethernet/mellanox/mlx5/core/main.c +1 -1
index 59806553889e..a39c4b25ba28 100644
@@ -956,7 +956,7 @@ static void mlx5_register_hca_devcom_comp(struct mlx5_core_dev *dev)
 		mlx5_devcom_register_component(dev->priv.devc, MLX5_DEVCOM_HCA_PORTS,
 					       mlx5_query_nic_system_image_guid(dev),
 					       NULL, dev);
-	if (IS_ERR_OR_NULL(dev->priv.hca_devcom_comp))
+	if (IS_ERR(dev->priv.hca_devcom_comp))
 		mlx5_core_err(dev, "Failed to register devcom HCA component\n");
 }
 
Signature · 1 line
2.44.0
author maintainer
#3 lore raw

PATCH 3/6 net/mlx5: Restore mistakenly dropped parts in register devlink flow

In reply to Tariq Toukan

From: Shay Drory <[email protected]>

Code parts from cited commit were mistakenly dropped while rebasing before submission. Add them here.

Fixes c6e77aa9dd82 ("net/mlx5: Register devlink first under devlink lock")
---
 drivers/net/ethernet/mellanox/mlx5/core/main.c          | 5 ++++-
 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c | 1 -
 2 files changed, 4 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +4 -1
index a39c4b25ba28..331ce47f51a1 100644
@@ -1699,12 +1699,15 @@ int mlx5_init_one_light(struct mlx5_core_dev *dev)
 	err = mlx5_devlink_params_register(priv_to_devlink(dev));
 	if (err) {
 		mlx5_core_warn(dev, "mlx5_devlink_param_reg err = %d\n", err);
-		goto query_hca_caps_err;
+		goto params_reg_err;
 	}
 
 	devl_unlock(devlink);
 	return 0;
 
+params_reg_err:
+	devl_unregister(devlink);
+	devl_unlock(devlink);
 query_hca_caps_err:
 	devl_unregister(devlink);
 	devl_unlock(devlink);
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c +0 -1
index e3bf8c7e4baa..7ebe71280827 100644
@@ -75,7 +75,6 @@ static int mlx5_sf_dev_probe(struct auxiliary_device *adev, const struct auxilia
 		goto peer_devlink_set_err;
 	}
 
-	devlink_register(devlink);
 	return 0;
 
 peer_devlink_set_err:
Signature · 1 line
2.44.0
author maintainer
#4 lore raw

PATCH 4/6 net/mlx5e: Use channel mdev reference instead of global mdev instance for coalescing

In reply to Tariq Toukan

From: Rahul Rameshbabu <[email protected]>

Channels can potentially have independent mdev instances. Do not refer to the global mdev instance in the mlx5e_priv instance for channel FW operations related to coalescing. CQ numbers that would be valid on the channel's mdev instance may not be correctly referenced if using the mlx5e_priv instance.

Fixes 67936e138586 ("net/mlx5e: Let channels be SD-aware")
---
 drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c +2 -2
index 8f101181648c..67a29826bb57 100644
@@ -589,12 +589,12 @@ static int mlx5e_get_coalesce(struct net_device *netdev,
 static void
 mlx5e_set_priv_channels_tx_coalesce(struct mlx5e_priv *priv, struct ethtool_coalesce *coal)
 {
-	struct mlx5_core_dev *mdev = priv->mdev;
 	int tc;
 	int i;
 
 	for (i = 0; i < priv->channels.num; ++i) {
 		struct mlx5e_channel *c = priv->channels.c[i];
+		struct mlx5_core_dev *mdev = c->mdev;
 
 		for (tc = 0; tc < c->num_tc; tc++) {
 			mlx5_core_modify_cq_moderation(mdev,
@@ -608,11 +608,11 @@ mlx5e_set_priv_channels_tx_coalesce(struct mlx5e_priv *priv, struct ethtool_coal
 static void
 mlx5e_set_priv_channels_rx_coalesce(struct mlx5e_priv *priv, struct ethtool_coalesce *coal)
 {
-	struct mlx5_core_dev *mdev = priv->mdev;
 	int i;
 
 	for (i = 0; i < priv->channels.num; ++i) {
 		struct mlx5e_channel *c = priv->channels.c[i];
+		struct mlx5_core_dev *mdev = c->mdev;
 
 		mlx5_core_modify_cq_moderation(mdev, &c->rq.cq.mcq,
 					       coal->rx_coalesce_usecs,
Signature · 1 line
2.44.0
author maintainer
#5 lore raw

PATCH 5/6 net/mlx5e: Acquire RTNL lock before RQs/SQs activation/deactivation

In reply to Tariq Toukan

From: Carolina Jubran <[email protected]>

netif_queue_set_napi asserts whether RTNL lock is held if the netdev is initialized.

Acquire the RTNL lock before activating or deactivating RQs/SQs if the lock has not been held before in the flow.

Fixes f25e7b82635f ("net/mlx5e: link NAPI instances to queues and IRQs")
Cc Joe Damato <[email protected]>
Cc Jakub Kicinski <[email protected]>
---
 drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c +7 -0
index 0ab9db319530..22918b2ef7f1 100644
@@ -108,7 +108,10 @@ static int mlx5e_tx_reporter_err_cqe_recover(void *ctx)
 	mlx5e_reset_txqsq_cc_pc(sq);
 	sq->stats->recover++;
 	clear_bit(MLX5E_SQ_STATE_RECOVERING, &sq->state);
+	rtnl_lock();
 	mlx5e_activate_txqsq(sq);
+	rtnl_unlock();
+
 	if (sq->channel)
 		mlx5e_trigger_napi_icosq(sq->channel);
 	else
@@ -179,12 +182,16 @@ static int mlx5e_tx_reporter_ptpsq_unhealthy_recover(void *ctx)
 	carrier_ok = netif_carrier_ok(netdev);
 	netif_carrier_off(netdev);
 
+	rtnl_lock();
 	mlx5e_deactivate_priv_channels(priv);
+	rtnl_unlock();
 
 	mlx5e_ptp_close(chs->ptp);
 	err = mlx5e_ptp_open(priv, &chs->params, chs->c[0]->lag_port, &chs->ptp);
 
+	rtnl_lock();
 	mlx5e_activate_priv_channels(priv);
+	rtnl_unlock();
 
 	/* return carrier back if needed */
 	if (carrier_ok)
Signature · 1 line
2.44.0
author maintainer
#6 lore raw

PATCH 6/6 net/mlx5e: Prevent deadlock while disabling aRFS

In reply to Tariq Toukan

From: Carolina Jubran <[email protected]>

When disabling aRFS under the priv->state_lock, any scheduled aRFS works are canceled using the cancel_work_sync function, which waits for the work to end if it has already started. However, while waiting for the work handler, the handler will try to acquire the state_lock which is already acquired.

The worker acquires the lock to delete the rules if the state is down, which is not the worker's responsibility since disabling aRFS deletes the rules.

Add an aRFS state variable, which indicates whether the aRFS is enabled and prevent adding rules when the aRFS is disabled.

Kernel log:

======================================================
WARNING: possible circular locking dependency detected
6.7.0-rc4_net_next_mlx5_5483eb2 #1 Tainted: G I
------------------------------------------------------
ethtool/386089 is trying to acquire lock:
ffff88810f21ce68 ((work_completion)(&rule->arfs_work)){+.+.}-{0:0}, at: __flush_work+0x74/0x4e0

but task is already holding lock:
ffff8884a1808cc0 (&priv->state_lock){+.+.}-{3:3}, at: mlx5e_ethtool_set_channels+0x53/0x200 [mlx5_core]

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #1 (&priv->state_lock){+.+.}-{3:3}:

       __mutex_lock+0x80/0xc90
       arfs_handle_work+0x4b/0x3b0 [mlx5_core]
       process_one_work+0x1dc/0x4a0
       worker_thread+0x1bf/0x3c0
       kthread+0xd7/0x100
       ret_from_fork+0x2d/0x50
       ret_from_fork_asm+0x11/0x20

-> #0 ((work_completion)(&rule->arfs_work)){+.+.}-{0:0}:

       __lock_acquire+0x17b4/0x2c80
       lock_acquire+0xd0/0x2b0
       __flush_work+0x7a/0x4e0
       __cancel_work_timer+0x131/0x1c0
       arfs_del_rules+0x143/0x1e0 [mlx5_core]
       mlx5e_arfs_disable+0x1b/0x30 [mlx5_core]
       mlx5e_ethtool_set_channels+0xcb/0x200 [mlx5_core]
       ethnl_set_channels+0x28f/0x3b0
       ethnl_default_set_doit+0xec/0x240
       genl_family_rcv_msg_doit+0xd0/0x120
       genl_rcv_msg+0x188/0x2c0
       netlink_rcv_skb+0x54/0x100
       genl_rcv+0x24/0x40
       netlink_unicast+0x1a1/0x270
       netlink_sendmsg+0x214/0x460
       __sock_sendmsg+0x38/0x60
       __sys_sendto+0x113/0x170
       __x64_sys_sendto+0x20/0x30
       do_syscall_64+0x40/0xe0
       entry_SYSCALL_64_after_hwframe+0x46/0x4e

other info that might help us debug this:

 Possible unsafe locking scenario:
       CPU0                    CPU1
       ----                    ----
  lock(&priv->state_lock);
                               lock((work_completion)(&rule->arfs_work));
                               lock(&priv->state_lock);

lock((work_completion)(&rule->arfs_work));

*** DEADLOCK ***

3 locks held by ethtool/386089:
#0: ffffffff82ea7210 (cb_lock){++++}-{3:3}, at: genl_rcv+0x15/0x40
#1: ffffffff82e94c88 (rtnl_mutex){+.+.}-{3:3}, at: ethnl_default_set_doit+0xd3/0x240
#2: ffff8884a1808cc0 (&priv->state_lock){+.+.}-{3:3}, at: mlx5e_ethtool_set_channels+0x53/0x200 [mlx5_core]

stack backtrace:
CPU: 15 PID: 386089 Comm: ethtool Tainted: G          I        6.7.0-rc4_net_next_mlx5_5483eb2 #1
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
Call Trace:
 <TASK>
 dump_stack_lvl+0x60/0xa0
 check_noncircular+0x144/0x160
 __lock_acquire+0x17b4/0x2c80
 lock_acquire+0xd0/0x2b0
 ? __flush_work+0x74/0x4e0
 ? save_trace+0x3e/0x360
 ? __flush_work+0x74/0x4e0
 __flush_work+0x7a/0x4e0
 ? __flush_work+0x74/0x4e0
 ? __lock_acquire+0xa78/0x2c80
 ? lock_acquire+0xd0/0x2b0
 ? mark_held_locks+0x49/0x70
 __cancel_work_timer+0x131/0x1c0
 ? mark_held_locks+0x49/0x70
 arfs_del_rules+0x143/0x1e0 [mlx5_core]
 mlx5e_arfs_disable+0x1b/0x30 [mlx5_core]
 mlx5e_ethtool_set_channels+0xcb/0x200 [mlx5_core]
 ethnl_set_channels+0x28f/0x3b0
 ethnl_default_set_doit+0xec/0x240
 genl_family_rcv_msg_doit+0xd0/0x120
 genl_rcv_msg+0x188/0x2c0
 ? ethnl_ops_begin+0xb0/0xb0
 ? genl_family_rcv_msg_dumpit+0xf0/0xf0
 netlink_rcv_skb+0x54/0x100
 genl_rcv+0x24/0x40
 netlink_unicast+0x1a1/0x270
 netlink_sendmsg+0x214/0x460
 __sock_sendmsg+0x38/0x60
 __sys_sendto+0x113/0x170
 ? do_user_addr_fault+0x53f/0x8f0
 __x64_sys_sendto+0x20/0x30
 do_syscall_64+0x40/0xe0
 entry_SYSCALL_64_after_hwframe+0x46/0x4e
 </TASK>
Fixes 45bf454ae884 ("net/mlx5e: Enabling aRFS mechanism")
---
 .../net/ethernet/mellanox/mlx5/core/en_arfs.c | 27 +++++++++++--------
 1 file changed, 16 insertions(+), 11 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c +16 -11
index c7f542d0b8f0..93cf23278d93 100644
@@ -46,6 +46,10 @@ struct arfs_table {
 	struct hlist_head	 rules_hash[ARFS_HASH_SIZE];
 };
 
+enum {
+	MLX5E_ARFS_STATE_ENABLED,
+};
+
 enum arfs_type {
 	ARFS_IPV4_TCP,
 	ARFS_IPV6_TCP,
@@ -60,6 +64,7 @@ struct mlx5e_arfs_tables {
 	spinlock_t                     arfs_lock;
 	int                            last_filter_id;
 	struct workqueue_struct        *wq;
+	unsigned long                  state;
 };
 
 struct arfs_tuple {
@@ -170,6 +175,8 @@ int mlx5e_arfs_enable(struct mlx5e_flow_steering *fs)
 			return err;
 		}
 	}
+	set_bit(MLX5E_ARFS_STATE_ENABLED, &arfs->state);
+
 	return 0;
 }
 
@@ -455,6 +462,8 @@ static void arfs_del_rules(struct mlx5e_flow_steering *fs)
 	int i;
 	int j;
 
+	clear_bit(MLX5E_ARFS_STATE_ENABLED, &arfs->state);
+
 	spin_lock_bh(&arfs->arfs_lock);
 	mlx5e_for_each_arfs_rule(rule, htmp, arfs->arfs_tables, i, j) {
 		hlist_del_init(&rule->hlist);
@@ -627,17 +636,8 @@ static void arfs_handle_work(struct work_struct *work)
 	struct mlx5_flow_handle *rule;
 
 	arfs = mlx5e_fs_get_arfs(priv->fs);
-	mutex_lock(&priv->state_lock);
-	if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
-		spin_lock_bh(&arfs->arfs_lock);
-		hlist_del(&arfs_rule->hlist);
-		spin_unlock_bh(&arfs->arfs_lock);
-
-		mutex_unlock(&priv->state_lock);
-		kfree(arfs_rule);
-		goto out;
-	}
-	mutex_unlock(&priv->state_lock);
+	if (!test_bit(MLX5E_ARFS_STATE_ENABLED, &arfs->state))
+		return;
 
 	if (!arfs_rule->rule) {
 		rule = arfs_add_rule(priv, arfs_rule);
@@ -753,6 +753,11 @@ int mlx5e_rx_flow_steer(struct net_device *dev, const struct sk_buff *skb,
 		return -EPROTONOSUPPORT;
 
 	spin_lock_bh(&arfs->arfs_lock);
+	if (!test_bit(MLX5E_ARFS_STATE_ENABLED, &arfs->state)) {
+		spin_unlock_bh(&arfs->arfs_lock);
+		return -EPERM;
+	}
+
 	arfs_rule = arfs_find_rule(arfs_t, &fk);
 	if (arfs_rule) {
 		if (arfs_rule->rxq == rxq_index || work_busy(&arfs_rule->arfs_work)) {
Signature · 1 line
2.44.0
#7 lore raw
In reply to Tariq Toukan
This series was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 8 lines · On Thu, 11 Apr 2024 14:54:38 +0300 you wrote:

Hi,

This patchset provides bug fixes to mlx5 core and Eth drivers.

Series generated against:
commit fe87922cee61 ("net/mlx5: fix possible stack overflows")

[...]

Here is the summary with links:
- [net,1/6] net/mlx5: Lag, restore buckets number to default after hash LAG deactivation

    https://git.kernel.org/netdev/net/c/37cc10da3a50
  - [net,2/6] net/mlx5: SD, Handle possible devcom ERR_PTR
    https://git.kernel.org/netdev/net/c/aa4ac90d04f4
  - [net,3/6] net/mlx5: Restore mistakenly dropped parts in register devlink flow
    https://git.kernel.org/netdev/net/c/bf729988303a
  - [net,4/6] net/mlx5e: Use channel mdev reference instead of global mdev instance for coalescing
    https://git.kernel.org/netdev/net/c/6c685bdb9e1a
  - [net,5/6] net/mlx5e: Acquire RTNL lock before RQs/SQs activation/deactivation
    https://git.kernel.org/netdev/net/c/fdce06bda7e5
  - [net,6/6] net/mlx5e: Prevent deadlock while disabling aRFS
    https://git.kernel.org/netdev/net/c/fef965764cf5

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html