Skip to contentSkip to conversation

iavf: fix temporary deadlock and failure to set MAC address

PATCH In review Michal Schmidt 3 participants 5 messages Dec 15, 2022 – Jan 19, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover iavf: fix temporary deadlock and failure to set MAC address

This fixes an issue where setting the MAC address on iavf runs into a timeout and fails with EAGAIN.

Changes in v2:
- Removed unused 'ret' variable in patch 1.
- Added patch 2 to fix another cause of the same timeout.

Michal Schmidt (2):
iavf: fix temporary deadlock and failure to set MAC address
iavf: avoid taking rtnl_lock in adminq_task

 drivers/net/ethernet/intel/iavf/iavf.h        |   4 +-
 .../net/ethernet/intel/iavf/iavf_ethtool.c    |  10 +-
 drivers/net/ethernet/intel/iavf/iavf_main.c   | 135 ++++++++++--------
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |   8 +-
 4 files changed, 86 insertions(+), 71 deletions(-)
Signature · 6 lines
2.37.2

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
#1 lore raw

PATCH v2 1/2 iavf: fix temporary deadlock and failure to set MAC address

We are seeing an issue where setting the MAC address on iavf fails with EAGAIN after the 2.5s timeout expires in iavf_set_mac().

There is the following deadlock scenario:

iavf_set_mac(), holding rtnl_lock, waits on:
iavf_watchdog_task (within iavf_wq) to send a message to the PF,
and
iavf_adminq_task (within iavf_wq) to receive a response from the PF.
In this adapter state (>=__IAVF_DOWN), these tasks do not need to take rtnl_lock, but iavf_wq is a global single-threaded workqueue, so they may get stuck waiting for another adapter's iavf_watchdog_task to run iavf_init_config_adapter(), which does take rtnl_lock.

The deadlock resolves itself by the timeout in iavf_set_mac(), which results in EAGAIN returned to userspace.

Let's break the deadlock loop by changing iavf_wq into a per-adapter workqueue, so that one adapter's tasks are not blocked by another's.

Fixes 35a2443d0910 ("iavf: Add waiting for response from PF in set mac")
---
 drivers/net/ethernet/intel/iavf/iavf.h        |  2 +-
 .../net/ethernet/intel/iavf/iavf_ethtool.c    | 10 +--
 drivers/net/ethernet/intel/iavf/iavf_main.c   | 86 +++++++++----------
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |  2 +-
 4 files changed, 49 insertions(+), 51 deletions(-)
drivers/net/ethernet/intel/iavf/iavf.h +1 -1
index 0d1bab4ac1b0..2a9f1eeeb701 100644
@@ -249,6 +249,7 @@ struct iavf_cloud_filter {
 
 /* board specific private data structure */
 struct iavf_adapter {
+	struct workqueue_struct *wq;
 	struct work_struct reset_task;
 	struct work_struct adminq_task;
 	struct delayed_work client_task;
@@ -459,7 +460,6 @@ struct iavf_device {
 
 /* needed by iavf_ethtool.c */
 extern char iavf_driver_name[];
-extern struct workqueue_struct *iavf_wq;
 
 static inline const char *iavf_state_str(enum iavf_state_t state)
 {
drivers/net/ethernet/intel/iavf/iavf_ethtool.c +5 -5
index a056e1545615..83cfc54a4706 100644
@@ -532,7 +532,7 @@ static int iavf_set_priv_flags(struct net_device *netdev, u32 flags)
 	if (changed_flags & IAVF_FLAG_LEGACY_RX) {
 		if (netif_running(netdev)) {
 			adapter->flags |= IAVF_FLAG_RESET_NEEDED;
-			queue_work(iavf_wq, &adapter->reset_task);
+			queue_work(adapter->wq, &adapter->reset_task);
 		}
 	}
 
@@ -672,7 +672,7 @@ static int iavf_set_ringparam(struct net_device *netdev,
 
 	if (netif_running(netdev)) {
 		adapter->flags |= IAVF_FLAG_RESET_NEEDED;
-		queue_work(iavf_wq, &adapter->reset_task);
+		queue_work(adapter->wq, &adapter->reset_task);
 	}
 
 	return 0;
@@ -1433,7 +1433,7 @@ static int iavf_add_fdir_ethtool(struct iavf_adapter *adapter, struct ethtool_rx
 	adapter->aq_required |= IAVF_FLAG_AQ_ADD_FDIR_FILTER;
 	spin_unlock_bh(&adapter->fdir_fltr_lock);
 
-	mod_delayed_work(iavf_wq, &adapter->watchdog_task, 0);
+	mod_delayed_work(adapter->wq, &adapter->watchdog_task, 0);
 
 ret:
 	if (err && fltr)
@@ -1474,7 +1474,7 @@ static int iavf_del_fdir_ethtool(struct iavf_adapter *adapter, struct ethtool_rx
 	spin_unlock_bh(&adapter->fdir_fltr_lock);
 
 	if (fltr && fltr->state == IAVF_FDIR_FLTR_DEL_REQUEST)
-		mod_delayed_work(iavf_wq, &adapter->watchdog_task, 0);
+		mod_delayed_work(adapter->wq, &adapter->watchdog_task, 0);
 
 	return err;
 }
@@ -1658,7 +1658,7 @@ iavf_set_adv_rss_hash_opt(struct iavf_adapter *adapter,
 	spin_unlock_bh(&adapter->adv_rss_lock);
 
 	if (!err)
-		mod_delayed_work(iavf_wq, &adapter->watchdog_task, 0);
+		mod_delayed_work(adapter->wq, &adapter->watchdog_task, 0);
 
 	mutex_unlock(&adapter->crit_lock);
 
drivers/net/ethernet/intel/iavf/iavf_main.c +42 -44

275 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1 -1
index 24a701fd140e..0752fd67c96e 100644
@@ -1952,7 +1952,7 @@ void iavf_virtchnl_completion(struct iavf_adapter *adapter,
 			if (!(adapter->flags & IAVF_FLAG_RESET_PENDING)) {
 				adapter->flags |= IAVF_FLAG_RESET_PENDING;
 				dev_info(&adapter->pdev->dev, "Scheduling reset task\n");
-				queue_work(iavf_wq, &adapter->reset_task);
+				queue_work(adapter->wq, &adapter->reset_task);
 			}
 			break;
 		default:
Signature · 6 lines
2.37.2

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Tested-by on v2 1/2
#2 lore raw

drivers/net/ethernet/intel/iavf/iavf.h

Quoted 45 lines

-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Michal Schmidt
Sent: czwartek, 15 grudnia 2022 23:51
To: [email protected]
Cc: ivecera <[email protected]>; [email protected]; Piotrowski, Patryk <[email protected]>
Subject: [Intel-wired-lan] [PATCH net 1/2] iavf: fix temporary deadlock and failure to set MAC address

We are seeing an issue where setting the MAC address on iavf fails with EAGAIN after the 2.5s timeout expires in iavf_set_mac().

There is the following deadlock scenario:

iavf_set_mac(), holding rtnl_lock, waits on:
iavf_watchdog_task (within iavf_wq) to send a message to the PF, and iavf_adminq_task (within iavf_wq) to receive a response from the PF.
In this adapter state (>=__IAVF_DOWN), these tasks do not need to take rtnl_lock, but iavf_wq is a global single-threaded workqueue, so they may get stuck waiting for another adapter's iavf_watchdog_task to run iavf_init_config_adapter(), which does take rtnl_lock.

The deadlock resolves itself by the timeout in iavf_set_mac(), which results in EAGAIN returned to userspace.

Let's break the deadlock loop by changing iavf_wq into a per-adapter workqueue, so that one adapter's tasks are not blocked by another's.

Fixes 35a2443d0910 ("iavf: Add waiting for response from PF in set mac")
---
 drivers/net/ethernet/intel/iavf/iavf.h        |  2 +-
 .../net/ethernet/intel/iavf/iavf_ethtool.c    | 10 +--
 drivers/net/ethernet/intel/iavf/iavf_main.c   | 86 +++++++++----------
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |  2 +-
 4 files changed, 49 insertions(+), 51 deletions(-)
drivers/net/ethernet/intel/iavf/iavf.h +0 -0
index 0d1bab4ac1b0..2a9f1eeeb701 100644

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#3 lore raw

PATCH v2 2/2 iavf: avoid taking rtnl_lock in adminq_task

In reply to Michal Schmidt

adminq_task processes virtchnl completions. iavf_set_mac() needs virtchnl communication to progress while it holds rtnl_lock. So adminq_task must not take rtnl_lock.

Do the handling of netdev features updates in a new work, features_task. The new work cannot run on the same ordered workqueue as adminq_task. The system-wide system_unbound_wq will do.

iavf_set_queue_vlan_tag_loc(), which iterates through queues, must run under crit_lock to prevent a concurrent iavf_free_queues() possibly called from watchdog_task or reset_task.

IAVF_FLAG_SETUP_NETDEV_FEATURES becomes unnecessary. features_task can be queued directly from iavf_virtchnl_completion().

Fixes 35a2443d0910 ("iavf: Add waiting for response from PF in set mac")
---
 drivers/net/ethernet/intel/iavf/iavf.h        |  2 +-
 drivers/net/ethernet/intel/iavf/iavf_main.c   | 49 ++++++++++++-------
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |  6 ++-
 3 files changed, 37 insertions(+), 20 deletions(-)
drivers/net/ethernet/intel/iavf/iavf.h +1 -1
index 2a9f1eeeb701..7dfd6dac74e4 100644
@@ -252,6 +252,7 @@ struct iavf_adapter {
 	struct workqueue_struct *wq;
 	struct work_struct reset_task;
 	struct work_struct adminq_task;
+	struct work_struct features_task;
 	struct delayed_work client_task;
 	wait_queue_head_t down_waitqueue;
 	wait_queue_head_t vc_waitqueue;
@@ -297,7 +298,6 @@ struct iavf_adapter {
 #define IAVF_FLAG_LEGACY_RX			BIT(15)
 #define IAVF_FLAG_REINIT_ITR_NEEDED		BIT(16)
 #define IAVF_FLAG_QUEUES_DISABLED		BIT(17)
-#define IAVF_FLAG_SETUP_NETDEV_FEATURES		BIT(18)
 #define IAVF_FLAG_REINIT_MSIX_NEEDED		BIT(20)
 /* duplicates for common code */
 #define IAVF_FLAG_DCB_ENABLED			0
drivers/net/ethernet/intel/iavf/iavf_main.c +31 -18

73 lines not on this page · raw message

drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +5 -1
index 0752fd67c96e..a644ab3804de 100644
@@ -2225,7 +2225,6 @@ void iavf_virtchnl_completion(struct iavf_adapter *adapter,
 				     sizeof(adapter->vlan_v2_caps)));
 
 		iavf_process_config(adapter);
-		adapter->flags |= IAVF_FLAG_SETUP_NETDEV_FEATURES;
 		was_mac_changed = !ether_addr_equal(netdev->dev_addr,
 						    adapter->hw.mac.addr);
 
@@ -2266,6 +2265,11 @@ void iavf_virtchnl_completion(struct iavf_adapter *adapter,
 
 		adapter->aq_required |= IAVF_FLAG_AQ_ADD_MAC_FILTER |
 			aq_required;
+
+		if (adapter->netdev_registered ||
+		    !test_bit(__IAVF_IN_REMOVE_TASK, &adapter->crit_section))
+			queue_work(system_unbound_wq,
+				   &adapter->features_task);
 		}
 		break;
 	case VIRTCHNL_OP_ENABLE_QUEUES:
Signature · 6 lines
2.37.2

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Reviewed-by Alexander Duyck on the v2 series
#4 lore raw
In reply to Michal Schmidt
Quoted 11 lines · Michal Schmidt

This fixes an issue where setting the MAC address on iavf runs into a timeout and fails with EAGAIN.

Changes in v2:
- Removed unused 'ret' variable in patch 1.
- Added patch 2 to fix another cause of the same timeout.

Michal Schmidt (2):
iavf: fix temporary deadlock and failure to set MAC address
iavf: avoid taking rtnl_lock in adminq_task

Quoted 6 lines
 drivers/net/ethernet/intel/iavf/iavf.h        |   4 +-
 .../net/ethernet/intel/iavf/iavf_ethtool.c    |  10 +-
 drivers/net/ethernet/intel/iavf/iavf_main.c   | 135 ++++++++++--------
 .../net/ethernet/intel/iavf/iavf_virtchnl.c   |   8 +-
 4 files changed, 86 insertions(+), 71 deletions(-)

The series looks good to me.

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan