iavf: fix temporary deadlock and failure to set MAC address
Subsystems (3)
v2 cover iavf: fix temporary deadlock and failure to set MAC address
This fixes an issue where setting the MAC address on iavf runs into a timeout and fails with EAGAIN.
Changes in v2:
- Removed unused 'ret' variable in patch 1.
- Added patch 2 to fix another cause of the same timeout.
Michal Schmidt (2):
iavf: fix temporary deadlock and failure to set MAC address
iavf: avoid taking rtnl_lock in adminq_task
drivers/net/ethernet/intel/iavf/iavf.h | 4 +- .../net/ethernet/intel/iavf/iavf_ethtool.c | 10 +- drivers/net/ethernet/intel/iavf/iavf_main.c | 135 ++++++++++-------- .../net/ethernet/intel/iavf/iavf_virtchnl.c | 8 +- 4 files changed, 86 insertions(+), 71 deletions(-)
Signature · 6 lines
2.37.2 _______________________________________________ Intel-wired-lan mailing list [email protected] https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
PATCH v2 1/2 iavf: fix temporary deadlock and failure to set MAC address
We are seeing an issue where setting the MAC address on iavf fails with EAGAIN after the 2.5s timeout expires in iavf_set_mac().
There is the following deadlock scenario:
iavf_set_mac(), holding rtnl_lock, waits on:
iavf_watchdog_task (within iavf_wq) to send a message to the PF,
and
iavf_adminq_task (within iavf_wq) to receive a response from the PF.
In this adapter state (>=__IAVF_DOWN), these tasks do not need to take rtnl_lock, but iavf_wq is a global single-threaded workqueue, so they may get stuck waiting for another adapter's iavf_watchdog_task to run iavf_init_config_adapter(), which does take rtnl_lock.
The deadlock resolves itself by the timeout in iavf_set_mac(), which results in EAGAIN returned to userspace.
Let's break the deadlock loop by changing iavf_wq into a per-adapter workqueue, so that one adapter's tasks are not blocked by another's.
--- drivers/net/ethernet/intel/iavf/iavf.h | 2 +- .../net/ethernet/intel/iavf/iavf_ethtool.c | 10 +-- drivers/net/ethernet/intel/iavf/iavf_main.c | 86 +++++++++---------- .../net/ethernet/intel/iavf/iavf_virtchnl.c | 2 +- 4 files changed, 49 insertions(+), 51 deletions(-)
drivers/net/ethernet/intel/iavf/iavf.h +1 -1
@@ -249,6 +249,7 @@ struct iavf_cloud_filter {/* board specific private data structure */ struct iavf_adapter { +struct workqueue_struct *wq;struct work_struct reset_task;struct work_struct adminq_task;struct delayed_work client_task;
@@ -459,7 +460,6 @@ struct iavf_device {/* needed by iavf_ethtool.c */ extern char iavf_driver_name[]; -extern struct workqueue_struct *iavf_wq;static inline const char *iavf_state_str (enum iavf_state_t state) {
drivers/net/ethernet/intel/iavf/iavf_ethtool.c +5 -5
@@ -532,7 +532,7 @@ static int iavf_set_priv_flags(struct net_device *netdev, u32 flags)if (changed_flags &IAVF_FLAG_LEGACY_RX ) {if (netif_running (netdev)) { adapter->flags |=IAVF_FLAG_RESET_NEEDED ; -queue_work (iavf_wq, &adapter->reset_task); +queue_work (adapter->wq, &adapter->reset_task); } }
@@ -672,7 +672,7 @@ static int iavf_set_ringparam(struct net_device *netdev,if (netif_running (netdev)) { adapter->flags |=IAVF_FLAG_RESET_NEEDED ; -queue_work (iavf_wq, &adapter->reset_task); +queue_work (adapter->wq, &adapter->reset_task); }return 0 ;
@@ -1433,7 +1433,7 @@ static int iavf_add_fdir_ethtool(struct iavf_adapter *adapter, struct ethtool_rx adapter->aq_required |=IAVF_FLAG_AQ_ADD_FDIR_FILTER ;spin_unlock_bh (&adapter->fdir_fltr_lock); -mod_delayed_work (iavf_wq, &adapter->watchdog_task,0 ); +mod_delayed_work (adapter->wq, &adapter->watchdog_task,0 ); ret:if (err && fltr)
@@ -1474,7 +1474,7 @@ static int iavf_del_fdir_ethtool(struct iavf_adapter *adapter, struct ethtool_rxspin_unlock_bh (&adapter->fdir_fltr_lock);if (fltr && fltr->state ==IAVF_FDIR_FLTR_DEL_REQUEST ) -mod_delayed_work (iavf_wq, &adapter->watchdog_task,0 ); +mod_delayed_work (adapter->wq, &adapter->watchdog_task,0 );return err; }
@@ -1658,7 +1658,7 @@ iavf_set_adv_rss_hash_opt(struct iavf_adapter *adapter,spin_unlock_bh (&adapter->adv_rss_lock);if (!err) -mod_delayed_work (iavf_wq, &adapter->watchdog_task,0 ); +mod_delayed_work (adapter->wq, &adapter->watchdog_task,0 );mutex_unlock (&adapter->crit_lock);
drivers/net/ethernet/intel/iavf/iavf_main.c +42 -44
275 lines not on this page · raw message
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +1 -1
@@ -1952,7 +1952,7 @@ void iavf_virtchnl_completion(struct iavf_adapter *adapter,if (!(adapter->flags &IAVF_FLAG_RESET_PENDING )) { adapter->flags |=IAVF_FLAG_RESET_PENDING ;dev_info (&adapter->pdev->dev,"Scheduling reset task\n" ); -queue_work (iavf_wq, &adapter->reset_task); +queue_work (adapter->wq, &adapter->reset_task); }break ;default :
Signature · 6 lines
2.37.2 _______________________________________________ Intel-wired-lan mailing list [email protected] https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
drivers/net/ethernet/intel/iavf/iavf.h
Quoted 45 lines
-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Michal Schmidt
Sent: czwartek, 15 grudnia 2022 23:51
To: [email protected]
Cc: ivecera <[email protected]>; [email protected]; Piotrowski, Patryk <[email protected]>
Subject: [Intel-wired-lan] [PATCH net 1/2] iavf: fix temporary deadlock and failure to set MAC addressWe are seeing an issue where setting the MAC address on iavf fails with EAGAIN after the 2.5s timeout expires in
iavf_set_mac().There is the following deadlock scenario:
iavf_set_mac(), holdingrtnl_lock, waits on:
iavf_watchdog_task(withiniavf_wq) to send a message to the PF, andiavf_adminq_task(withiniavf_wq) to receive a response from the PF.
In this adapter state (>=__IAVF_DOWN), these tasks do not need to takertnl_lock, butiavf_wqis a global single-threaded workqueue, so they may get stuck waiting for another adapter'siavf_watchdog_taskto runiavf_init_config_adapter(), which does takertnl_lock.The deadlock resolves itself by the timeout in
iavf_set_mac(), which results in EAGAIN returned to userspace.Let's break the deadlock loop by changing
iavf_wqinto a per-adapter workqueue, so that one adapter's tasks are not blocked by another's.Fixes 35a2443d0910 ("iavf: Add waiting for response from PF in set mac")Co-developed-by Ivan Vecera <[email protected]>Signed-off-by Ivan Vecera <[email protected]>Signed-off-by Michal Schmidt <[email protected]>--- drivers/net/ethernet/intel/iavf/iavf.h | 2 +- .../net/ethernet/intel/iavf/iavf_ethtool.c | 10 +-- drivers/net/ethernet/intel/iavf/iavf_main.c | 86 +++++++++---------- .../net/ethernet/intel/iavf/iavf_virtchnl.c | 2 +- 4 files changed, 49 insertions(+), 51 deletions(-)drivers/net/ethernet/intel/iavf/iavf.h +0
-0
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
PATCH v2 2/2 iavf: avoid taking rtnl_lock in adminq_task
In reply to Michal Schmidtadminq_task processes virtchnl completions. iavf_set_mac() needs virtchnl communication to progress while it holds rtnl_lock. So adminq_task must not take rtnl_lock.
Do the handling of netdev features updates in a new work, features_task. The new work cannot run on the same ordered workqueue as adminq_task. The system-wide system_unbound_wq will do.
iavf_set_queue_vlan_tag_loc(), which iterates through queues, must run under crit_lock to prevent a concurrent iavf_free_queues() possibly called from watchdog_task or reset_task.
IAVF_FLAG_SETUP_NETDEV_FEATURES becomes unnecessary. features_task can be queued directly from iavf_virtchnl_completion().
--- drivers/net/ethernet/intel/iavf/iavf.h | 2 +- drivers/net/ethernet/intel/iavf/iavf_main.c | 49 ++++++++++++------- .../net/ethernet/intel/iavf/iavf_virtchnl.c | 6 ++- 3 files changed, 37 insertions(+), 20 deletions(-)
drivers/net/ethernet/intel/iavf/iavf.h +1 -1
@@ -252,6 +252,7 @@ struct iavf_adapter {struct workqueue_struct *wq;struct work_struct reset_task;struct work_struct adminq_task; +struct work_struct features_task;struct delayed_work client_task;wait_queue_head_t down_waitqueue;wait_queue_head_t vc_waitqueue;
@@ -297,7 +298,6 @@ struct iavf_adapter {#define IAVF_FLAG_LEGACY_RX BIT(15)#define IAVF_FLAG_REINIT_ITR_NEEDED BIT(16)#define IAVF_FLAG_QUEUES_DISABLED BIT(17) -#define IAVF_FLAG_SETUP_NETDEV_FEATURES BIT(18)#define IAVF_FLAG_REINIT_MSIX_NEEDED BIT(20)/* duplicates for common code */ #define IAVF_FLAG_DCB_ENABLED 0
drivers/net/ethernet/intel/iavf/iavf_main.c +31 -18
73 lines not on this page · raw message
drivers/net/ethernet/intel/iavf/iavf_virtchnl.c +5 -1
@@ -2225,7 +2225,6 @@ void iavf_virtchnl_completion(struct iavf_adapter *adapter, sizeof(adapter->vlan_v2_caps)));iavf_process_config (adapter); - adapter->flags |=IAVF_FLAG_SETUP_NETDEV_FEATURES ; was_mac_changed = !ether_addr_equal (netdev->dev_addr, adapter->hw.mac.addr);
@@ -2266,6 +2265,11 @@ void iavf_virtchnl_completion(struct iavf_adapter *adapter, adapter->aq_required |=IAVF_FLAG_AQ_ADD_MAC_FILTER | aq_required; + +if (adapter->netdev_registered || + !test_bit (__IAVF_IN_REMOVE_TASK, &adapter->crit_section)) +queue_work (system_unbound_wq, + &adapter->features_task); }break ;case VIRTCHNL_OP_ENABLE_QUEUES :
Signature · 6 lines
2.37.2 _______________________________________________ Intel-wired-lan mailing list [email protected] https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Quoted 11 lines · Michal Schmidt
This fixes an issue where setting the MAC address on iavf runs into a timeout and fails with EAGAIN.
Changes in v2:
- Removed unused 'ret' variable in patch 1.
- Added patch 2 to fix another cause of the same timeout.Michal Schmidt (2):
iavf: fix temporary deadlock and failure to set MAC address
iavf: avoid takingrtnl_lockinadminq_task
Quoted 6 lines
drivers/net/ethernet/intel/iavf/iavf.h | 4 +- .../net/ethernet/intel/iavf/iavf_ethtool.c | 10 +- drivers/net/ethernet/intel/iavf/iavf_main.c | 135 ++++++++++-------- .../net/ethernet/intel/iavf/iavf_virtchnl.c | 8 +- 4 files changed, 86 insertions(+), 71 deletions(-)
The series looks good to me.
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan