Skip to contentSkip to conversation

igbvf: VF mailbox timeouts with multiple VF messages

PATCH New Greg Edwards 2 participants 5 messages Jul 20, 2017 – Jul 28, 2017

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover igbvf: VF mailbox timeouts with multiple VF messages

We've encountered VF mailbox timeout issues with I350 VFs assigned to VMs, when NetworkManager or lldpad (LLDP) are assigning multicast addresses in rapid succession. A VF message may encounter a mailbox timeout waiting for the ACK from the PF, and the igbvf_watchdog_task will detect the condition and reset the VF. This pattern can continue for some time or indefinitely.

Changes from v1:
- fix in igbvf VF driver instead of igb PF driver, per suggestion by Alex.

Greg Edwards (2):
igbvf: add lock around mailbox ops
igbvf: after mailbox write, wait for reply

 drivers/net/ethernet/intel/igbvf/ethtool.c |  4 +++
 drivers/net/ethernet/intel/igbvf/mbx.c     |  4 +++
 drivers/net/ethernet/intel/igbvf/netdev.c  | 47 ++++++++++++++++++++++++++++++
 drivers/net/ethernet/intel/igbvf/vf.c      | 10 +++++--
 drivers/net/ethernet/intel/igbvf/vf.h      |  1 +
 5 files changed, 64 insertions(+), 2 deletions(-)
Signature · 1 line
2.13.3
#1 lore raw

PATCH v2 1/2 igbvf: add lock around mailbox ops

The PF driver assumes the VF will not send another mailbox message until the PF has written its reply to the previous message. If the VF does, that message will be silently dropped by the PF before it writes its reply to the mailbox. This results in a VF mailbox timeout for posted messages waiting for an ACK, and the VF is reset by the igbvf_watchdog_task in the VM.

Add a lock around the VF mailbox ops to prevent the VF from sending another message while the PF is still processing the previous one.

---
 drivers/net/ethernet/intel/igbvf/ethtool.c |  4 +++
 drivers/net/ethernet/intel/igbvf/mbx.c     |  4 +++
 drivers/net/ethernet/intel/igbvf/netdev.c  | 47 ++++++++++++++++++++++++++++++
 drivers/net/ethernet/intel/igbvf/vf.h      |  1 +
 4 files changed, 56 insertions(+)
drivers/net/ethernet/intel/igbvf/ethtool.c +4 -0
index 34faa113a8a0..a127688e83e6 100644
@@ -296,8 +296,12 @@ static int igbvf_link_test(struct igbvf_adapter *adapter, u64 *data)
 	struct e1000_hw *hw = &adapter->hw;
 	*data = 0;
 
+	spin_lock_bh(&hw->mbx_lock);
+
 	hw->mac.ops.check_for_link(hw);
 
+	spin_unlock_bh(&hw->mbx_lock);
+
 	if (!(er32(STATUS) & E1000_STATUS_LU))
 		*data = 1;
 
drivers/net/ethernet/intel/igbvf/mbx.c +4 -0
index 01752f44ace2..c9a441632e9f 100644
@@ -264,6 +264,8 @@ static s32 e1000_write_mbx_vf(struct e1000_hw *hw, u32 *msg, u16 size)
 	s32 err;
 	u16 i;
 
+	WARN_ON_ONCE(!spin_is_locked(&hw->mbx_lock));
+
 	/* lock the mailbox to prevent pf/vf race condition */
 	err = e1000_obtain_mbx_lock_vf(hw);
 	if (err)
@@ -300,6 +302,8 @@ static s32 e1000_read_mbx_vf(struct e1000_hw *hw, u32 *msg, u16 size)
 	s32 err;
 	u16 i;
 
+	WARN_ON_ONCE(!spin_is_locked(&hw->mbx_lock));
+
 	/* lock the mailbox to prevent pf/vf race condition */
 	err = e1000_obtain_mbx_lock_vf(hw);
 	if (err)
drivers/net/ethernet/intel/igbvf/netdev.c +47 -0

143 lines not on this page · raw message

drivers/net/ethernet/intel/igbvf/vf.h +1 -0
index 4cf78b0dec50..d213eefb6169 100644
@@ -245,6 +245,7 @@ struct e1000_hw {
 
 	struct e1000_mac_info  mac;
 	struct e1000_mbx_info mbx;
+	spinlock_t mbx_lock;		/* serializes mailbox ops */
 
 	union {
 		struct e1000_dev_spec_vf vf;
Signature · 1 line
2.13.3
#2 lore raw

PATCH 1/2 igbvf: add lock around mailbox ops

Quoted 17 lines

From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Greg Edwards
Sent: Thursday, July 20, 2017 9:01 AM
To: intel-wired-lan at lists.osuosl.org
Cc: Greg Edwards <[email protected]>
Subject: [Intel-wired-lan] [PATCH 1/2] igbvf: add lock around mailbox ops

The PF driver assumes the VF will not send another mailbox message until the PF has written its reply to the previous message. If the VF does, that message will be silently dropped by the PF before it writes its reply to the mailbox. This results in a VF mailbox timeout for posted messages waiting for an ACK, and the VF is reset by the igbvf_watchdog_task in the VM.

Add a lock around the VF mailbox ops to prevent the VF from sending another message while the PF is still processing the previous one.

Quoted 8 lines
---
 drivers/net/ethernet/intel/igbvf/ethtool.c |  4 +++
 drivers/net/ethernet/intel/igbvf/mbx.c     |  4 +++
 drivers/net/ethernet/intel/igbvf/netdev.c  | 47
++++++++++++++++++++++++++++++
 drivers/net/ethernet/intel/igbvf/vf.h      |  1 +
 4 files changed, 56 insertions(+)
#3 lore raw

PATCH v2 2/2 igbvf: after mailbox write, wait for reply

In reply to Greg Edwards

Two of the VF mailbox commands were not waiting for a reply from the PF, which can result in a VF mailbox timeout in the VM for the next command.

---
 drivers/net/ethernet/intel/igbvf/vf.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igbvf/vf.c +8 -2
index 528be116184e..1d3aa9adcaa8 100644
@@ -230,6 +230,7 @@ static void e1000_update_mc_addr_list_vf(struct e1000_hw *hw,
 	u16 *hash_list = (u16 *)&msgbuf[1];
 	u32 hash_value;
 	u32 cnt, i;
+	s32 ret_val;
 
 	/* Each entry in the list uses 1 16 bit word.  We have 30
 	 * 16 bit words available in our HW msg buffer (minus 1 for the
@@ -250,7 +251,9 @@ static void e1000_update_mc_addr_list_vf(struct e1000_hw *hw,
 		mc_addr_list += ETH_ALEN;
 	}
 
-	mbx->ops.write_posted(hw, msgbuf, E1000_VFMAILBOX_SIZE);
+	ret_val = mbx->ops.write_posted(hw, msgbuf, E1000_VFMAILBOX_SIZE);
+	if (!ret_val)
+		mbx->ops.read_posted(hw, msgbuf, 1);
 }
 
 /**
@@ -293,11 +296,14 @@ void e1000_rlpml_set_vf(struct e1000_hw *hw, u16 max_size)
 {
 	struct e1000_mbx_info *mbx = &hw->mbx;
 	u32 msgbuf[2];
+	s32 ret_val;
 
 	msgbuf[0] = E1000_VF_SET_LPE;
 	msgbuf[1] = max_size;
 
-	mbx->ops.write_posted(hw, msgbuf, 2);
+	ret_val = mbx->ops.write_posted(hw, msgbuf, 2);
+	if (!ret_val)
+		mbx->ops.read_posted(hw, msgbuf, 1);
 }
 
 /**
Signature · 1 line
2.13.3
#4 lore raw

PATCH 2/2 igbvf: after mailbox write, wait for reply

Quoted 8 lines

From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Greg Edwards
Sent: Thursday, July 20, 2017 9:01 AM
To: intel-wired-lan at lists.osuosl.org
Cc: Greg Edwards <[email protected]>
Subject: [Intel-wired-lan] [PATCH 2/2] igbvf: after mailbox write, wait for reply

Quoted 7 lines

Two of the VF mailbox commands were not waiting for a reply from the PF, which can result in a VF mailbox timeout in the VM for the next command.

---
 drivers/net/ethernet/intel/igbvf/vf.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)