igbvf: VF mailbox timeouts with multiple VF messages
Subsystems (2)
v2 cover igbvf: VF mailbox timeouts with multiple VF messages
We've encountered VF mailbox timeout issues with I350 VFs assigned to VMs, when NetworkManager or lldpad (LLDP) are assigning multicast addresses in rapid succession. A VF message may encounter a mailbox timeout waiting for the ACK from the PF, and the igbvf_watchdog_task will detect the condition and reset the VF. This pattern can continue for some time or indefinitely.
Changes from v1:
- fix in igbvf VF driver instead of igb PF driver, per suggestion by Alex.
Greg Edwards (2):
igbvf: add lock around mailbox ops
igbvf: after mailbox write, wait for reply
drivers/net/ethernet/intel/igbvf/ethtool.c | 4 +++ drivers/net/ethernet/intel/igbvf/mbx.c | 4 +++ drivers/net/ethernet/intel/igbvf/netdev.c | 47 ++++++++++++++++++++++++++++++ drivers/net/ethernet/intel/igbvf/vf.c | 10 +++++-- drivers/net/ethernet/intel/igbvf/vf.h | 1 + 5 files changed, 64 insertions(+), 2 deletions(-)
Signature · 1 line
2.13.3
PATCH v2 1/2 igbvf: add lock around mailbox ops
The PF driver assumes the VF will not send another mailbox message until the PF has written its reply to the previous message. If the VF does, that message will be silently dropped by the PF before it writes its reply to the mailbox. This results in a VF mailbox timeout for posted messages waiting for an ACK, and the VF is reset by the igbvf_watchdog_task in the VM.
Add a lock around the VF mailbox ops to prevent the VF from sending another message while the PF is still processing the previous one.
--- drivers/net/ethernet/intel/igbvf/ethtool.c | 4 +++ drivers/net/ethernet/intel/igbvf/mbx.c | 4 +++ drivers/net/ethernet/intel/igbvf/netdev.c | 47 ++++++++++++++++++++++++++++++ drivers/net/ethernet/intel/igbvf/vf.h | 1 + 4 files changed, 56 insertions(+)
drivers/net/ethernet/intel/igbvf/ethtool.c +4 -0
@@ -296,8 +296,12 @@ static int igbvf_link_test(struct igbvf_adapter *adapter, u64 *data)struct e1000_hw *hw = &adapter->hw; *data =0 ; +spin_lock_bh (&hw->mbx_lock); + hw->mac.ops.check_for_link (hw); +spin_unlock_bh (&hw->mbx_lock); +if (!(er32 (STATUS ) &E1000_STATUS_LU )) *data =1 ;
drivers/net/ethernet/intel/igbvf/mbx.c +4 -0
@@ -264,6 +264,8 @@ static s32 e1000_write_mbx_vf(struct e1000_hw *hw, u32 *msg, u16 size)s32 err;u16 i; +WARN_ON_ONCE (!spin_is_locked (&hw->mbx_lock)); +/* lock the mailbox to prevent pf/vf race condition */ err =e1000_obtain_mbx_lock_vf (hw);if (err)
@@ -300,6 +302,8 @@ static s32 e1000_read_mbx_vf(struct e1000_hw *hw, u32 *msg, u16 size)s32 err;u16 i; +WARN_ON_ONCE (!spin_is_locked (&hw->mbx_lock)); +/* lock the mailbox to prevent pf/vf race condition */ err =e1000_obtain_mbx_lock_vf (hw);if (err)
drivers/net/ethernet/intel/igbvf/netdev.c +47 -0
143 lines not on this page · raw message
drivers/net/ethernet/intel/igbvf/vf.h +1 -0
@@ -245,6 +245,7 @@ struct e1000_hw {struct e1000_mac_info mac;struct e1000_mbx_info mbx; +spinlock_t mbx_lock;/* serializes mailbox ops */ union {struct e1000_dev_spec_vf vf;
Signature · 1 line
2.13.3
Brown
PATCH 1/2 igbvf: add lock around mailbox ops
Quoted 17 lines
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Greg Edwards
Sent: Thursday, July 20, 2017 9:01 AM
To: intel-wired-lan at lists.osuosl.org
Cc: Greg Edwards <[email protected]>
Subject: [Intel-wired-lan] [PATCH 1/2] igbvf: add lock around mailbox opsThe PF driver assumes the VF will not send another mailbox message until the PF has written its reply to the previous message. If the VF does, that message will be silently dropped by the PF before it writes its reply to the mailbox. This results in a VF mailbox timeout for posted messages waiting for an ACK, and the VF is reset by the
igbvf_watchdog_taskin the VM.Add a lock around the VF mailbox ops to prevent the VF from sending another message while the PF is still processing the previous one.
Quoted 8 lines
Signed-off-by Greg Edwards <[email protected]>--- drivers/net/ethernet/intel/igbvf/ethtool.c | 4 +++ drivers/net/ethernet/intel/igbvf/mbx.c | 4 +++ drivers/net/ethernet/intel/igbvf/netdev.c | 47 ++++++++++++++++++++++++++++++ drivers/net/ethernet/intel/igbvf/vf.h | 1 + 4 files changed, 56 insertions(+)
PATCH v2 2/2 igbvf: after mailbox write, wait for reply
In reply to Greg EdwardsTwo of the VF mailbox commands were not waiting for a reply from the PF, which can result in a VF mailbox timeout in the VM for the next command.
--- drivers/net/ethernet/intel/igbvf/vf.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igbvf/vf.c +8 -2
@@ -230,6 +230,7 @@ static void e1000_update_mc_addr_list_vf(struct e1000_hw *hw,u16 *hash_list = (u16 *)&msgbuf[1 ];u32 hash_value;u32 cnt, i; +s32 ret_val;/* Each entry in the list uses 1 16 bit word. We have 30 * 16 bit words available in our HW msg buffer (minus 1 for the
@@ -250,7 +251,9 @@ static void e1000_update_mc_addr_list_vf(struct e1000_hw *hw, mc_addr_list +=ETH_ALEN ; } - mbx->ops.write_posted (hw, msgbuf,E1000_VFMAILBOX_SIZE ); + ret_val = mbx->ops.write_posted (hw, msgbuf,E1000_VFMAILBOX_SIZE ); +if (!ret_val) + mbx->ops.read_posted (hw, msgbuf,1 ); }/**
@@ -293,11 +296,14 @@ void e1000_rlpml_set_vf(struct e1000_hw *hw, u16 max_size) {struct e1000_mbx_info *mbx = &hw->mbx;u32 msgbuf[2 ]; +s32 ret_val; msgbuf[0 ] =E1000_VF_SET_LPE ; msgbuf[1 ] = max_size; - mbx->ops.write_posted (hw, msgbuf,2 ); + ret_val = mbx->ops.write_posted (hw, msgbuf,2 ); +if (!ret_val) + mbx->ops.read_posted (hw, msgbuf,1 ); }/**
Signature · 1 line
2.13.3
Brown
PATCH 2/2 igbvf: after mailbox write, wait for reply
Quoted 8 lines
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Greg Edwards
Sent: Thursday, July 20, 2017 9:01 AM
To: intel-wired-lan at lists.osuosl.org
Cc: Greg Edwards <[email protected]>
Subject: [Intel-wired-lan] [PATCH 2/2] igbvf: after mailbox write, wait for reply
Quoted 7 lines
Two of the VF mailbox commands were not waiting for a reply from the PF, which can result in a VF mailbox timeout in the VM for the next command.
Signed-off-by Greg Edwards <[email protected]>--- drivers/net/ethernet/intel/igbvf/vf.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)