ptp cornercase cleanups
Subsystems (2)
v4 cover ptp cornercase cleanups
This series contains fixes for similar corner cases in the e1000e, igb, i40e, and ixgbe drivers reported by David Mirabito on the ptp4l mailing list.
The first few patches fix a race condition with skb_tstamp_tx(). The next few patches ensure that we cleanup the state bit when exiting the _xmit_frame_ring() function with a DMA or TSO failure. After this, we add new statistic to count how many times the Tx timestamp is skipped due to already outstanding requests. The final few patches add some handling in the potential case where we somehow never finish a timestamp request, ensuring that we don't permanently lock the Tx timestamps forever in this rare case.
Changes in v4
- Split the driver patches at request of Shannon Nelson
- Updated commit messages, but code content should be identical
I apologize in advance, the patchworks will not track this properly because I necessarily changed the commit subjects when splitting the patches apart by driver.
Jacob Keller (15):
e1000e: fix race condition around skb_tstamp_tx()
i40e: fix race condition with PTP_TX_IN_PROGRESS bits
igb: fix race condition with PTP_TX_IN_PROGRESS bits
ixgbe: fix race condition with PTP_TX_IN_PROGRESS bits
i40e: avoid permanent lock of *_PTP_TX_IN_PROGRESS
igb: avoid permanent lock of *_PTP_TX_IN_PROGRESS
ixgbe: avoid permanent lock of *_PTP_TX_IN_PROGRESS
e1000e: add statistic indicating number of skipped Tx timestamps
i40e: add statistic indicating number of skipped Tx timestamps
igb: add statistic indicating number of skipped Tx timestamps
ixgbe: add statistic indicating number of skipped Tx timestamps
i40e: use pf data structure directly in i40e_ptp_rx_hang
i40e: check for Tx timestamp timeouts during watchdog
igb: check for Tx timestamp timeouts during watchdog
ixgbe: check for Tx timestamp timeouts during watchdog
drivers/net/ethernet/intel/e1000e/e1000.h | 1 + drivers/net/ethernet/intel/e1000e/ethtool.c | 1 + drivers/net/ethernet/intel/e1000e/netdev.c | 27 ++++++++----- drivers/net/ethernet/intel/i40e/i40e.h | 5 ++- drivers/net/ethernet/intel/i40e/i40e_ethtool.c | 1 + drivers/net/ethernet/intel/i40e/i40e_main.c | 3 +- drivers/net/ethernet/intel/i40e/i40e_ptp.c | 48 +++++++++++++++++++++--- drivers/net/ethernet/intel/i40e/i40e_txrx.c | 28 +++++++++++--- drivers/net/ethernet/intel/igb/igb.h | 2 + drivers/net/ethernet/intel/igb/igb_ethtool.c | 1 + drivers/net/ethernet/intel/igb/igb_main.c | 26 ++++++++++--- drivers/net/ethernet/intel/igb/igb_ptp.c | 41 +++++++++++++++++++- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 2 + drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 3 ++ drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 42 ++++++++++++++------- drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 42 +++++++++++++++++++-- 16 files changed, 227 insertions(+), 46 deletions(-)
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
PATCH v4 1/15 e1000e: fix race condition around skb_tstamp_tx()
The e1000e driver and related hardware has a limitation on Tx PTP packets which requires we limit to timestamping a single packet at once. We do this by verifying that we never request a new Tx timestamp while we still have a tx_hwtstamp_skb pointer.
Unfortunately the driver suffers from a race condition around this. The tx_hwtstamp_skb pointer is not set to NULL until after skb_tstamp_tx() is called. This function notifies the stack and applications of a new timestamp. Even a well behaved application that only sends a new request when the first one is finished might be woken up and possibly send a packet before we can free the timestamp in the driver again. The result is that we needlessly ignore some Tx timestamp requests in this corner case.
Fix this by assigning the tx_hwtstamp_skb pointer prior to calling skb_tstamp_tx() and use a temporary pointer to hold the timestamped skb until that function finishes. This ensures that the application is not woken up until the driver is ready to begin timestamping a new packet.
This ensures that well behaved applications do not accidentally race with condition to skip Tx timestamps. Obviously an application which sends multiple Tx timestamp requests at once will still only timestamp one packet at a time. Unfortunately there is nothing we can do about this.
--- drivers/net/ethernet/intel/e1000e/netdev.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/e1000e/netdev.c +8 -2
@@ -1183,6 +1183,7 @@ static void e1000e_tx_hwtstamp_work(struct work_struct *work)struct e1000_hw *hw = &adapter->hw;if (er32 (TSYNCTXCTL ) &E1000_TSYNCTXCTL_VALID ) { +struct sk_buff *skb = adapter->tx_hwtstamp_skb;struct skb_shared_hwtstamps shhwtstamps;u64 txstmp;
@@ -1191,9 +1192,14 @@ static void e1000e_tx_hwtstamp_work(struct work_struct *work)e1000e_systim_to_hwtstamp (adapter, &shhwtstamps, txstmp); -skb_tstamp_tx (adapter->tx_hwtstamp_skb, &shhwtstamps); -dev_kfree_skb_any (adapter->tx_hwtstamp_skb); +/* Clear the global tx_hwtstamp_skb pointer and force writes +* prior to notifying the stack of a Tx timestamp. +*/ adapter->tx_hwtstamp_skb =NULL ; +wmb ();/* force write prior to skb_tstamp_tx */ + +skb_tstamp_tx (skb, &shhwtstamps); +dev_kfree_skb_any (skb); }else if (time_after (jiffies, adapter->tx_hwtstamp_start + adapter->tx_timeout_factor *HZ )) {dev_kfree_skb_any (adapter->tx_hwtstamp_skb);
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Brown
PATCH v4 1/15 e1000e: fix race condition around skb_tstamp_tx()
Quoted 33 lines
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 01/15] e1000e: fix race condition aroundskb_tstamp_tx()The e1000e driver and related hardware has a limitation on Tx PTP packets which requires we limit to timestamping a single packet at once. We do this by verifying that we never request a new Tx timestamp while we still have a
tx_hwtstamp_skbpointer.Unfortunately the driver suffers from a race condition around this. The
tx_hwtstamp_skbpointer is not set to NULL until afterskb_tstamp_tx()is called. This function notifies the stack and applications of a new timestamp. Even a well behaved application that only sends a new request when the first one is finished might be woken up and possibly send a packet before we can free the timestamp in the driver again. The result is that we needlessly ignore some Tx timestamp requests in this corner case.Fix this by assigning the
tx_hwtstamp_skbpointer prior to callingskb_tstamp_tx()and use a temporary pointer to hold the timestamped skb until that function finishes. This ensures that the application is not woken up until the driver is ready to begin timestamping a new packet.This ensures that well behaved applications do not accidentally race with condition to skip Tx timestamps. Obviously an application which sends multiple Tx timestamp requests at once will still only timestamp one packet at a time. Unfortunately there is nothing we can do about this.
Quoted 6 lines
Reported-by David Mirabito <[email protected]>Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/e1000e/netdev.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
PATCH v4 2/15 i40e: fix race condition with PTP_TX_IN_PROGRESS bits
In reply to Jacob KellerHardware related to the i40e driver has a limitation on Tx PTP packets. This requires us to limit the driver to timestamping a single packet at once. This is done using a state bitlock which enforces that only one timestamp request is honored at a time.
Unfortunately this suffers from a race condition. The bit lock is not cleared until after skb_tstamp_tx() is called notifying applications of a new Tx timestamp. Even a well behaved application sending only one packet at a time and waiting for a response can wake up and send a new timestamped packet request before the bit lock is cleared. This results in needlessly dropping some Tx timestamp requests.
We can fix this by unlocking the state bit as soon as we read the Timestamp register, as this is the first point at which it is safe to timestamp another packet.
To avoid issues with the skb pointer, we'll use a copy of the pointer and set the global variable in the driver structure to NULL first. This ensures that the next timestamp request does not modify our local copy of the skb pointer.
Now, a well behaved application which has at most one outstanding timestamp request will not accidentally race with the driver unlock bit. Obviously an application attempting to timestamp faster than one request at a time will have some timestamp requests skipped. Unfortunately there is nothing we can do about that.
--- drivers/net/ethernet/intel/i40e/i40e_ptp.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
drivers/net/ethernet/intel/i40e/i40e_ptp.c +11 -3
@@ -338,6 +338,7 @@ void i40e_ptp_rx_hang(struct i40e_vsi *vsi)void i40e_ptp_tx_hwtstamp (struct i40e_pf *pf) {struct skb_shared_hwtstamps shhwtstamps; +struct sk_buff *skb = pf->ptp_tx_skb;struct i40e_hw *hw = &pf->hw;u32 hi, lo;u64 ns;
@@ -353,12 +354,19 @@ void i40e_ptp_tx_hwtstamp(struct i40e_pf *pf) hi =rd32 (hw,I40E_PRTTSYN_TXTIME_H ); ns = (((u64 )hi) <<32 ) | lo; -i40e_ptp_convert_to_hwtstamp (&shhwtstamps, ns); -skb_tstamp_tx (pf->ptp_tx_skb, &shhwtstamps); -dev_kfree_skb_any (pf->ptp_tx_skb); + +/* Clear the bit lock as soon as possible after reading the register, +* and prior to notifying the stack via skb_tstamp_tx(). Otherwise +* applications might wake up and attempt to request another transmit +* timestamp prior to the bit lock being cleared. +*/ pf->ptp_tx_skb =NULL ;clear_bit_unlock (__I40E_PTP_TX_IN_PROGRESS, pf->state); + +/* Notify the stack and free the skb after we've unlocked */ +skb_tstamp_tx (skb, &shhwtstamps); +dev_kfree_skb_any (skb); }/**
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Bowers
PATCH v4 2/15 i40e: fix race condition with PTP_TX_IN_PROGRESS bits
Quoted 35 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 02/15] i40e: fix race condition withPTP_TX_IN_PROGRESSbitsHardware related to the i40e driver has a limitation on Tx PTP packets. This requires us to limit the driver to timestamping a single packet at once. This is done using a state bitlock which enforces that only one timestamp request is honored at a time.
Unfortunately this suffers from a race condition. The bit lock is not cleared until after
skb_tstamp_tx()is called notifying applications of a new Tx timestamp. Even a well behaved application sending only one packet at a time and waiting for a response can wake up and send a new timestamped packet request before the bit lock is cleared. This results in needlessly dropping some Tx timestamp requests.We can fix this by unlocking the state bit as soon as we read the Timestamp register, as this is the first point at which it is safe to timestamp another packet.
To avoid issues with the skb pointer, we'll use a copy of the pointer and set the global variable in the driver structure to NULL first. This ensures that the next timestamp request does not modify our local copy of the skb pointer.
Now, a well behaved application which has at most one outstanding timestamp request will not accidentally race with the driver unlock bit. Obviously an application attempting to timestamp faster than one request at a time will have some timestamp requests skipped. Unfortunately there is nothing we can do about that.
Quoted 5 lines
Reported-by David Mirabito <[email protected]>Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/i40e/i40e_ptp.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
PATCH v4 3/15 igb: fix race condition with PTP_TX_IN_PROGRESS bits
In reply to Jacob KellerHardware related to the igb driver has a limiation of only handling one Tx timestamp at a time. Thus, the driver uses a state bit lock to enforce that only one timestamp request is honored at a time.
Unfortunately this suffers from a simple race condition. The bit lock is not cleared until after skb_tstamp_tx() is called notifying the stack of a new Tx timestamp. Even a well behaved application which sends only one timestamp request at once and waits for a response might wake up and send a new packet before the bit lock is cleared. This results in needlessly dropping some Tx timestamp requests.
We can fix this by unlocking the state bit as soon as we read the Timestamp register, as this is the first point at which it is safe to unlock.
To avoid issues with the skb pointer, we'll use a copy of the pointer and set the global variable in the driver structure to NULL first. This ensures that the next timestamp request does not modify our local copy of the skb pointer.
This ensures that well behaved applications do not accidentally race with the unlock bit. Obviously an application which sends multiple Tx timestamp requests at once will still only timestamp one packet at a time. Unfortunately there is nothing we can do about this.
--- drivers/net/ethernet/intel/igb/igb_ptp.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igb/igb_ptp.c +10 -2
@@ -721,6 +721,7 @@ void igb_ptp_rx_hang(struct igb_adapter *adapter)**/ static void igb_ptp_tx_hwtstamp (struct igb_adapter *adapter) { +struct sk_buff *skb = adapter->ptp_tx_skb;struct e1000_hw *hw = &adapter->hw;struct skb_shared_hwtstamps shhwtstamps;u64 regval;
@@ -748,10 +749,17 @@ static void igb_ptp_tx_hwtstamp(struct igb_adapter *adapter) shhwtstamps.hwtstamp =ktime_add_ns (shhwtstamps.hwtstamp, adjust); -skb_tstamp_tx (adapter->ptp_tx_skb, &shhwtstamps); -dev_kfree_skb_any (adapter->ptp_tx_skb); +/* Clear the lock early before calling skb_tstamp_tx so that +* applications are not woken up before the lock bit is clear. We use +* a copy of the skb pointer to ensure other threads can't change it +* while we're notifying the stack. +*/ adapter->ptp_tx_skb =NULL ;clear_bit_unlock (__IGB_PTP_TX_IN_PROGRESS, &adapter->state); + +/* Notify the stack and free the skb after we've unlocked */ +skb_tstamp_tx (skb, &shhwtstamps); +dev_kfree_skb_any (skb); }/**
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Brown
PATCH v4 3/15 igb: fix race condition with PTP_TX_IN_PROGRESS bits
Quoted 33 lines
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 03/15] igb: fix race condition withPTP_TX_IN_PROGRESSbitsHardware related to the igb driver has a limiation of only handling one Tx timestamp at a time. Thus, the driver uses a state bit lock to enforce that only one timestamp request is honored at a time.
Unfortunately this suffers from a simple race condition. The bit lock is not cleared until after
skb_tstamp_tx()is called notifying the stack of a new Tx timestamp. Even a well behaved application which sends only one timestamp request at once and waits for a response might wake up and send a new packet before the bit lock is cleared. This results in needlessly dropping some Tx timestamp requests.We can fix this by unlocking the state bit as soon as we read the Timestamp register, as this is the first point at which it is safe to unlock.
To avoid issues with the skb pointer, we'll use a copy of the pointer and set the global variable in the driver structure to NULL first. This ensures that the next timestamp request does not modify our local copy of the skb pointer.
This ensures that well behaved applications do not accidentally race with the unlock bit. Obviously an application which sends multiple Tx timestamp requests at once will still only timestamp one packet at a time. Unfortunately there is nothing we can do about this.
Quoted 6 lines
Reported-by David Mirabito <[email protected]>Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/igb/igb_ptp.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
PATCH v4 4/15 ixgbe: fix race condition with PTP_TX_IN_PROGRESS bits
In reply to Jacob KellerHardware related to the ixgbe driver is limited to handling a single Tx timestamp request at a time. Thus, the driver ignores requests for Tx timestamp while waiting for the current request to finish. It usess a state bit lock which enforces that only one timestamp request is honored at a time.
Unfortunately this suffers from a simple race condition. The bit lock is not cleared until after skb_tstamp_tx() is called notifying applications of a new Tx timestamp. Even a well behaved application sending only one packet at a time and waiting for a response can wake up and send a new packet before the bit lock is cleared. This results in needlessly dropping some Tx timestamp requests.
We can fix this by unlocking the state bit as soon as we read the Timestamp register, as this is the first point at which it is safe to unlock.
To avoid issues with the skb pointer, we'll use a copy of the pointer and set the global variable in the driver structure to NULL first. This ensures that the next timestamp request does not modify our local copy of the skb pointer.
This ensures that well behaved applications do not accidentally race with the unlock bit. Obviously an application which sends multiple Tx timestamp requests at once will still only timestamp one packet at a time. Unfortunately there is nothing we can do about this.
--- drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c +12 -3
@@ -672,17 +672,26 @@ static void ixgbe_ptp_clear_tx_timestamp(struct ixgbe_adapter *adapter)*/ static void ixgbe_ptp_tx_hwtstamp (struct ixgbe_adapter *adapter) { +struct sk_buff *skb = adapter->ptp_tx_skb;struct ixgbe_hw *hw = &adapter->hw;struct skb_shared_hwtstamps shhwtstamps;u64 regval =0 ; regval |= (u64 )IXGBE_READ_REG (hw,IXGBE_TXSTMPL ); regval |= (u64 )IXGBE_READ_REG (hw,IXGBE_TXSTMPH ) <<32 ; -ixgbe_ptp_convert_to_hwtstamp (adapter, &shhwtstamps, regval); -skb_tstamp_tx (adapter->ptp_tx_skb, &shhwtstamps); -ixgbe_ptp_clear_tx_timestamp (adapter); +/* Handle cleanup of the ptp_tx_skb ourselves, and unlock the state +* bit prior to notifying the stack via skb_tstamp_tx(). This prevents +* well behaved applications from attempting to timestamp again prior +* to the lock bit being clear. +*/ + adapter->ptp_tx_skb =NULL ; +clear_bit_unlock (__IXGBE_PTP_TX_IN_PROGRESS, &adapter->state); + +/* Notify the stack and then free the skb after we've unlocked */ +skb_tstamp_tx (skb, &shhwtstamps); +dev_kfree_skb_any (skb); }/**
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Bowers
PATCH v4 4/15 ixgbe: fix race condition with PTP_TX_IN_PROGRESS bits
Quoted 33 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 04/15] ixgbe: fix race condition withPTP_TX_IN_PROGRESSbitsHardware related to the ixgbe driver is limited to handling a single Tx timestamp request at a time. Thus, the driver ignores requests for Tx timestamp while waiting for the current request to finish. It usess a state bit lock which enforces that only one timestamp request is honored at a time.
Unfortunately this suffers from a simple race condition. The bit lock is not cleared until after
skb_tstamp_tx()is called notifying applications of a new Tx timestamp. Even a well behaved application sending only one packet at a time and waiting for a response can wake up and send a new packet before the bit lock is cleared. This results in needlessly dropping some Tx timestamp requests.We can fix this by unlocking the state bit as soon as we read the Timestamp register, as this is the first point at which it is safe to unlock.
To avoid issues with the skb pointer, we'll use a copy of the pointer and set the global variable in the driver structure to NULL first. This ensures that the next timestamp request does not modify our local copy of the skb pointer.
This ensures that well behaved applications do not accidentally race with the unlock bit. Obviously an application which sends multiple Tx timestamp requests at once will still only timestamp one packet at a time. Unfortunately there is nothing we can do about this.
Quoted 5 lines
Reported-by David Mirabito <[email protected]>Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
PATCH v4 5/15 i40e: avoid permanent lock of *_PTP_TX_IN_PROGRESS
In reply to Jacob KellerThe i40e driver uses a bit lock to indicate when a Tx timestamp is in progress to avoid attempting to timestamp multiple packets at once. This is required because hardware only has registers to handle one request at a time.
There is a corner case where we failed to cleanup the bit lock after a failed transmit. This can potentially result in a state bit being locked forever.
Add some cleanup code to i40e_xmit_frame_ring to check and make sure we cleanup incase of these failures. We also modify i40e_tx_map to return an error code indication DMA failure.
--- drivers/net/ethernet/intel/i40e/i40e_txrx.c | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)
drivers/net/ethernet/intel/i40e/i40e_txrx.c +20 -6
@@ -2932,10 +2932,12 @@ bool __i40e_chk_linearize(struct sk_buff *skb)* @hdr_len: size of the packet header * @td_cmd: the command field in the descriptor * @td_offset: offset for checksum or crc +* +* Returns 0 on success, -1 on failure to DMA **/ -static inline void i40e_tx_map (struct i40e_ring *tx_ring,struct sk_buff *skb, -struct i40e_tx_buffer *first,u32 tx_flags, -const u8 hdr_len,u32 td_cmd,u32 td_offset) +static inline int i40e_tx_map (struct i40e_ring *tx_ring,struct sk_buff *skb, +struct i40e_tx_buffer *first,u32 tx_flags, +const u8 hdr_len,u32 td_cmd,u32 td_offset) {unsigned int data_len = skb->data_len;unsigned int size =skb_headlen (skb);
@@ -3093,7 +3095,7 @@ static inline void i40e_tx_map(struct i40e_ring *tx_ring, struct sk_buff *skb,mmiowb (); } -return ; +return 0 ; dma_error:dev_info (tx_ring->dev,"TX DMA map failed\n" );
@@ -3110,6 +3112,8 @@ static inline void i40e_tx_map(struct i40e_ring *tx_ring, struct sk_buff *skb, } tx_ring->next_to_use = i; + +return -1 ; }/**
@@ -3210,8 +3214,9 @@ static netdev_tx_t i40e_xmit_frame_ring(struct sk_buff *skb,*/ i40e_atr (tx_ring, skb, tx_flags); -i40e_tx_map (tx_ring, skb, first, tx_flags, hdr_len, - td_cmd, td_offset); +if (i40e_tx_map (tx_ring, skb, first, tx_flags, hdr_len, + td_cmd, td_offset)) + goto cleanup_tx_tstamp;return NETDEV_TX_OK ;
@@ -3219,6 +3224,15 @@ static netdev_tx_t i40e_xmit_frame_ring(struct sk_buff *skb,i40e_trace (xmit_frame_ring_drop, first->skb, tx_ring);dev_kfree_skb_any (first->skb); first->skb =NULL ; +cleanup_tx_tstamp: +if (unlikely (tx_flags &I40E_TX_FLAGS_TSYN )) { +struct i40e_pf *pf =i40e_netdev_to_pf (tx_ring->netdev); + +dev_kfree_skb_any (pf->ptp_tx_skb); + pf->ptp_tx_skb =NULL ; +clear_bit_unlock (__I40E_PTP_TX_IN_PROGRESS, pf->state); + } +return NETDEV_TX_OK ; }
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Bowers
PATCH v4 5/15 i40e: avoid permanent lock of *_PTP_TX_IN_PROGRESS
Quoted 20 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 05/15] i40e: avoid permanent lock of*_PTP_TX_IN_PROGRESSThe i40e driver uses a bit lock to indicate when a Tx timestamp is in progress to avoid attempting to timestamp multiple packets at once. This is required because hardware only has registers to handle one request at a time.
There is a corner case where we failed to cleanup the bit lock after a failed transmit. This can potentially result in a state bit being locked forever.
Add some cleanup code to
i40e_xmit_frame_ringto check and make sure we cleanup incase of these failures. We also modifyi40e_tx_mapto return an error code indication DMA failure.
Quoted 6 lines
Reported-by Reported-by: David Mirabito <[email protected]>Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/i40e/i40e_txrx.c | 26 ++++++++++++++++++++-- ---- 1 file changed, 20 insertions(+), 6 deletions(-)
PATCH v4 6/15 igb: avoid permanent lock of *_PTP_TX_IN_PROGRESS
In reply to Jacob KellerThe igb driver uses a state bit lock to avoid handling more than one Tx timestamp request at once. This is required because hardware is limited to a single set of registers for Tx timestamps.
The state bit lock is not properly cleaned up during igb_xmit_frame_ring() if the transmit fails such as due to DMA or TSO failure. In some hardware this results in blocking timestamps until the service task times out. In other hardware this results in a permanent lock of the timestamp bit because we never receive an interrupt indicating the timestamp occurred, since indeed the packet was never transmitted.
Fix this by checking for DMA and TSO errors in igb_xmit_frame_ring() and properly cleaning up after ourselves when these occur.
--- drivers/net/ethernet/intel/igb/igb_main.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-)
drivers/net/ethernet/intel/igb/igb_main.c +18 -5
@@ -5201,9 +5201,9 @@ static inline int igb_maybe_stop_tx(struct igb_ring *tx_ring, const u16 size)return __igb_maybe_stop_tx (tx_ring, size); } -static void igb_tx_map (struct igb_ring *tx_ring, -struct igb_tx_buffer *first, -const u8 hdr_len) +static int igb_tx_map (struct igb_ring *tx_ring, +struct igb_tx_buffer *first, +const u8 hdr_len) {struct sk_buff *skb = first->skb;struct igb_tx_buffer *tx_buffer;
@@ -5314,7 +5314,7 @@ static void igb_tx_map(struct igb_ring *tx_ring,*/ mmiowb (); } -return ; +return 0 ; dma_error:dev_err (tx_ring->dev,"TX DMA map failed\n" );
@@ -5345,6 +5345,8 @@ static void igb_tx_map(struct igb_ring *tx_ring, tx_buffer->skb =NULL ; tx_ring->next_to_use = i; + +return -1 ; } netdev_tx_t igb_xmit_frame_ring(struct sk_buff *skb,
@@ -5410,13 +5412,24 @@ netdev_tx_t igb_xmit_frame_ring(struct sk_buff *skb,else if (!tso)igb_tx_csum (tx_ring, first); -igb_tx_map (tx_ring, first, hdr_len); +if (igb_tx_map (tx_ring, first, hdr_len)) + goto cleanup_tx_tstamp;return NETDEV_TX_OK ; out_drop:dev_kfree_skb_any (first->skb); first->skb =NULL ; +cleanup_tx_tstamp: +if (unlikely (tx_flags &IGB_TX_FLAGS_TSTAMP )) { +struct igb_adapter *adapter =netdev_priv (tx_ring->netdev); + +dev_kfree_skb_any (adapter->ptp_tx_skb); + adapter->ptp_tx_skb =NULL ; +if (adapter->hw.mac.type == e1000_82576) +cancel_work_sync (&adapter->ptp_tx_work); +clear_bit_unlock (__IGB_PTP_TX_IN_PROGRESS, &adapter->state); + }return NETDEV_TX_OK ; }
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Brown
PATCH v4 6/15 igb: avoid permanent lock of *_PTP_TX_IN_PROGRESS
Quoted 20 lines
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 06/15] igb: avoid permanent lock of*_PTP_TX_IN_PROGRESSThe igb driver uses a state bit lock to avoid handling more than one Tx timestamp request at once. This is required because hardware is limited to a single set of registers for Tx timestamps.
The state bit lock is not properly cleaned up during
igb_xmit_frame_ring()if the transmit fails such as due to DMA or TSO failure. In some hardware this results in blocking timestamps until the service task times out. In other hardware this results in a permanent lock of the timestamp bit because we never receive an interrupt indicating the timestamp occurred, since indeed the packet was never transmitted.
Quoted 9 lines
Fix this by checking for DMA and TSO errors in
igb_xmit_frame_ring()and properly cleaning up after ourselves when these occur.Reported-by Reported-by: David Mirabito <[email protected]>Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/igb/igb_main.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-)
PATCH v4 7/15 ixgbe: avoid permanent lock of *_PTP_TX_IN_PROGRESS
In reply to Jacob KellerThe ixgbe driver uses a state bit lock to avoid handling more than one Tx timestamp request at once. This is required because hardware is limited to a single set of registers for Tx timestamps.
The state bit lock is not properly cleaned up during ixgbe_xmit_frame_ring() if the transmit fails such as due to DMA or TSO failure. In some hardware this results in blocking timestamps until the service task times out. In other hardware this results in a permanent lock of the timestamp bit because we never receive an interrupt indicating the timestamp occurred, since indeed the packet was never transmitted.
Fix this by checking for DMA and TSO errors in ixgbe_xmit_frame_ring() and properly cleaning up after ourselves when these occur.
--- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +15 -5
@@ -7885,9 +7885,9 @@ static inline int ixgbe_maybe_stop_tx(struct ixgbe_ring *tx_ring, u16 size)#define IXGBE_TXD_CMD (IXGBE_TXD_CMD_EOP | \ IXGBE_TXD_CMD_RS) -static void ixgbe_tx_map (struct ixgbe_ring *tx_ring, -struct ixgbe_tx_buffer *first, -const u8 hdr_len) +static int ixgbe_tx_map (struct ixgbe_ring *tx_ring, +struct ixgbe_tx_buffer *first, +const u8 hdr_len) {struct sk_buff *skb = first->skb;struct ixgbe_tx_buffer *tx_buffer;
@@ -8014,7 +8014,7 @@ static void ixgbe_tx_map(struct ixgbe_ring *tx_ring,mmiowb (); } -return ; +return 0 ; dma_error:dev_err (tx_ring->dev,"TX DMA map failed\n" ); tx_buffer = &tx_ring->tx_buffer_info[i];
@@ -8044,6 +8044,8 @@ static void ixgbe_tx_map(struct ixgbe_ring *tx_ring, first->skb =NULL ; tx_ring->next_to_use = i; + +return -1 ; } static void ixgbe_atr(struct ixgbe_ring *ring,
@@ -8416,13 +8418,21 @@ netdev_tx_t ixgbe_xmit_frame_ring(struct sk_buff *skb,#ifdef IXGBE_FCOE xmit_fcoe:#endif /* IXGBE_FCOE */ -ixgbe_tx_map (tx_ring, first, hdr_len); +if (ixgbe_tx_map (tx_ring, first, hdr_len)) + goto cleanup_tx_timestamp;return NETDEV_TX_OK ; out_drop:dev_kfree_skb_any (first->skb); first->skb =NULL ; +cleanup_tx_timestamp: +if (unlikely (tx_flags &IXGBE_TX_FLAGS_TSTAMP )) { +dev_kfree_skb_any (adapter->ptp_tx_skb); + adapter->ptp_tx_skb =NULL ; +cancel_work_sync (&adapter->ptp_tx_work); +clear_bit_unlock (__IXGBE_PTP_TX_IN_PROGRESS, &adapter->state); + }return NETDEV_TX_OK ; }
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Bowers
PATCH v4 7/15 ixgbe: avoid permanent lock of *_PTP_TX_IN_PROGRESS
Quoted 20 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 07/15] ixgbe: avoid permanent lock of*_PTP_TX_IN_PROGRESSThe ixgbe driver uses a state bit lock to avoid handling more than one Tx timestamp request at once. This is required because hardware is limited to a single set of registers for Tx timestamps.
The state bit lock is not properly cleaned up during
ixgbe_xmit_frame_ring()if the transmit fails such as due to DMA or TSO failure. In some hardware this results in blocking timestamps until the service task times out. In other hardware this results in a permanent lock of the timestamp bit because we never receive an interrupt indicating the timestamp occurred, since indeed the packet was never transmitted.
Quoted 8 lines
Fix this by checking for DMA and TSO errors in
ixgbe_xmit_frame_ring()and properly cleaning up after ourselves when these occur.Reported-by Reported-by: David Mirabito <[email protected]>Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)
PATCH v4 8/15 e1000e: add statistic indicating number of skipped Tx timestamps
In reply to Jacob KellerThe e1000e driver can only handle one Tx timestamp request at a time. This means it is possible for an application timestamp request to be ignored.
There is no easy way for an administrator to determine if this occurred. Add a new statistic which tracks this, tx_hwtstamp_skipped.
--- drivers/net/ethernet/intel/e1000e/e1000.h | 1 + drivers/net/ethernet/intel/e1000e/ethtool.c | 1 + drivers/net/ethernet/intel/e1000e/netdev.c | 17 ++++++++++------- 3 files changed, 12 insertions(+), 7 deletions(-)
drivers/net/ethernet/intel/e1000e/e1000.h +1 -0
@@ -268,6 +268,7 @@ struct e1000_adapter {u32 tx_fifo_size;u32 tx_dma_failed;u32 tx_hwtstamp_timeouts; +u32 tx_hwtstamp_skipped;/* Rx */ bool (*clean_rx)(struct e1000_ring *ring, int *work_done,
drivers/net/ethernet/intel/e1000e/ethtool.c +1 -0
@@ -105,6 +105,7 @@ static const struct e1000_stats e1000_gstrings_stats[] = {E1000_STAT ("uncorr_ecc_errors" , uncorr_errors),E1000_STAT ("corr_ecc_errors" , corr_errors),E1000_STAT ("tx_hwtstamp_timeouts" , tx_hwtstamp_timeouts), + E1000_STAT("tx_hwtstamp_skipped", tx_hwtstamp_skipped), };#define E1000_GLOBAL_STATS_LEN ARRAY_SIZE(e1000_gstrings_stats)
drivers/net/ethernet/intel/e1000e/netdev.c +10 -7
@@ -5866,13 +5866,16 @@ static netdev_tx_t e1000_xmit_frame(struct sk_buff *skb, nr_frags);if (count) {if (unlikely (skb_shinfo (skb)->tx_flags &SKBTX_HW_TSTAMP ) && - (adapter->flags &FLAG_HAS_HW_TIMESTAMP ) && - !adapter->tx_hwtstamp_skb) { -skb_shinfo (skb)->tx_flags |=SKBTX_IN_PROGRESS ; - tx_flags |=E1000_TX_FLAGS_HWTSTAMP ; - adapter->tx_hwtstamp_skb =skb_get (skb); - adapter->tx_hwtstamp_start = jiffies; -schedule_work (&adapter->tx_hwtstamp_work); + (adapter->flags &FLAG_HAS_HW_TIMESTAMP )) { +if (!adapter->tx_hwtstamp_skb) { +skb_shinfo (skb)->tx_flags |=SKBTX_IN_PROGRESS ; + tx_flags |=E1000_TX_FLAGS_HWTSTAMP ; + adapter->tx_hwtstamp_skb =skb_get (skb); + adapter->tx_hwtstamp_start = jiffies; +schedule_work (&adapter->tx_hwtstamp_work); + }else { + adapter->tx_hwtstamp_skipped++; + } }else {skb_tx_timestamp (skb); }
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Brown
PATCH v4 8/15 e1000e: add statistic indicating number of skipped Tx timestamps
Quoted 15 lines
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 08/15] e1000e: add statistic indicating number of skipped Tx timestampsThe e1000e driver can only handle one Tx timestamp request at a time. This means it is possible for an application timestamp request to be ignored.
There is no easy way for an administrator to determine if this occurred. Add a new statistic which tracks this,
tx_hwtstamp_skipped.
Quoted 7 lines
Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/e1000e/e1000.h | 1 + drivers/net/ethernet/intel/e1000e/ethtool.c | 1 + drivers/net/ethernet/intel/e1000e/netdev.c | 17 ++++++++++------- 3 files changed, 12 insertions(+), 7 deletions(-)
PATCH v4 9/15 i40e: add statistic indicating number of skipped Tx timestamps
In reply to Jacob KellerThe i40e driver can only handle one Tx timestamp request at a time. This means it is possible for an application timestamp request to be ignored.
There is no easy way for an administrator to determine if this occurred. Add a new statistic which tracks this, tx_hwtstamp_skipped.
--- drivers/net/ethernet/intel/i40e/i40e.h | 1 + drivers/net/ethernet/intel/i40e/i40e_ethtool.c | 1 + drivers/net/ethernet/intel/i40e/i40e_txrx.c | 1 + 3 files changed, 3 insertions(+)
drivers/net/ethernet/intel/i40e/i40e.h +1 -0
@@ -506,6 +506,7 @@ struct i40e_pf {struct mutex tmreg_lock;/* Used to protect the SYSTIME registers. */ u64 ptp_base_adj;u32 tx_hwtstamp_timeouts; +u32 tx_hwtstamp_skipped;u32 rx_hwtstamp_cleared;u32 latch_event_flags;spinlock_t ptp_rx_lock;/* Used to protect Rx timestamp registers. */
drivers/net/ethernet/intel/i40e/i40e_ethtool.c +1 -0
@@ -147,6 +147,7 @@ static const struct i40e_stats i40e_gstrings_stats[] = {I40E_PF_STAT ("VF_admin_queue_requests" , vf_aq_requests),I40E_PF_STAT ("arq_overflows" , arq_overflows),I40E_PF_STAT ("rx_hwtstamp_cleared" , rx_hwtstamp_cleared), +I40E_PF_STAT ("tx_hwtstamp_skipped" , tx_hwtstamp_skipped),I40E_PF_STAT ("fdir_flush_cnt" , fd_flush_cnt),I40E_PF_STAT ("fdir_atr_match" , stats.fd_atr_match), I40E_PF_STAT("fdir_atr_tunnel_match", stats.fd_atr_tunnel_match),
drivers/net/ethernet/intel/i40e/i40e_txrx.c +1 -0
@@ -2630,6 +2630,7 @@ static int i40e_tsyn(struct i40e_ring *tx_ring, struct sk_buff *skb,skb_shinfo (skb)->tx_flags |=SKBTX_IN_PROGRESS ; pf->ptp_tx_skb =skb_get (skb); }else { + pf->tx_hwtstamp_skipped++;return 0 ; }
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Bowers
PATCH v4 9/15 i40e: add statistic indicating number of skipped Tx timestamps
Quoted 15 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 09/15] i40e: add statistic indicating number of skipped Tx timestampsThe i40e driver can only handle one Tx timestamp request at a time. This means it is possible for an application timestamp request to be ignored.
There is no easy way for an administrator to determine if this occurred. Add a new statistic which tracks this,
tx_hwtstamp_skipped.
Quoted 6 lines
Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/i40e/i40e.h | 1 + drivers/net/ethernet/intel/i40e/i40e_ethtool.c | 1 + drivers/net/ethernet/intel/i40e/i40e_txrx.c | 1 + 3 files changed, 3 insertions(+)
PATCH v4 10/15 igb: add statistic indicating number of skipped Tx timestamps
In reply to Jacob KellerThe igb driver can only handle one Tx timestamp request at a time. This means it is possible for an application timestamp request to be ignored.
There is no easy way for an administrator to determine if this occurred. Add a new statistic which tracks this, tx_hwtstamp_skipped.
--- drivers/net/ethernet/intel/igb/igb.h | 1 + drivers/net/ethernet/intel/igb/igb_ethtool.c | 1 + drivers/net/ethernet/intel/igb/igb_main.c | 2 ++ 3 files changed, 4 insertions(+)
drivers/net/ethernet/intel/igb/igb.h +1 -0
@@ -563,6 +563,7 @@ struct igb_adapter {struct cyclecounter cc;struct timecounter tc;u32 tx_hwtstamp_timeouts; +u32 tx_hwtstamp_skipped;u32 rx_hwtstamp_cleared;bool pps_sys_wrap_on;
drivers/net/ethernet/intel/igb/igb_ethtool.c +1 -0
@@ -90,6 +90,7 @@ static const struct igb_stats igb_gstrings_stats[] = {IGB_STAT ("os2bmc_tx_by_host" , stats.o2bspc),IGB_STAT ("os2bmc_rx_by_host" , stats.b2ogprc),IGB_STAT ("tx_hwtstamp_timeouts" , tx_hwtstamp_timeouts), +IGB_STAT ("tx_hwtstamp_skipped" , tx_hwtstamp_skipped), IGB_STAT("rx_hwtstamp_cleared", rx_hwtstamp_cleared), };
drivers/net/ethernet/intel/igb/igb_main.c +2 -0
@@ -5392,6 +5392,8 @@ netdev_tx_t igb_xmit_frame_ring(struct sk_buff *skb, adapter->ptp_tx_start = jiffies;if (adapter->hw.mac.type == e1000_82576)schedule_work (&adapter->ptp_tx_work); + }else { + adapter->tx_hwtstamp_skipped++; } }
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Brown
PATCH v4 10/15 igb: add statistic indicating number of skipped Tx timestamps
Quoted 15 lines
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 10/15] igb: add statistic indicating number of skipped Tx timestampsThe igb driver can only handle one Tx timestamp request at a time. This means it is possible for an application timestamp request to be ignored.
There is no easy way for an administrator to determine if this occurred. Add a new statistic which tracks this,
tx_hwtstamp_skipped.
Quoted 7 lines
Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/igb/igb.h | 1 + drivers/net/ethernet/intel/igb/igb_ethtool.c | 1 + drivers/net/ethernet/intel/igb/igb_main.c | 2 ++ 3 files changed, 4 insertions(+)
PATCH v4 11/15 ixgbe: add statistic indicating number of skipped Tx timestamps
In reply to Jacob KellerThe ixgbe driver can only handle one Tx timestamp request at a time. This means it is possible for an application timestamp request to be ignored.
There is no easy way for an administrator to determine if this occurred. Add a new statistic which tracks this, tx_hwtstamp_skipped.
--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 1 + drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 3 +++ drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 21 ++++++++++++--------- 3 files changed, 16 insertions(+), 9 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe.h +1 -0
@@ -733,6 +733,7 @@ struct ixgbe_adapter {struct timecounter hw_tc;u32 base_incval;u32 tx_hwtstamp_timeouts; +u32 tx_hwtstamp_skipped;u32 rx_hwtstamp_cleared;void (*ptp_setup_sdp)(struct ixgbe_adapter *);
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +3 -0
@@ -111,6 +111,9 @@ static const struct ixgbe_stats ixgbe_gstrings_stats[] = { {"os2bmc_tx_by_bmc", IXGBE_STAT(stats.b2ospc)}, {"os2bmc_tx_by_host", IXGBE_STAT(stats.o2bspc)}, {"os2bmc_rx_by_host", IXGBE_STAT(stats.b2ogprc)}, + {"tx_hwtstamp_timeouts", IXGBE_STAT(tx_hwtstamp_timeouts)}, + {"tx_hwtstamp_skipped", IXGBE_STAT(tx_hwtstamp_skipped)}, + {"rx_hwtstamp_cleared", IXGBE_STAT(rx_hwtstamp_cleared)}, #ifdef IXGBE_FCOE {"fcoe_bad_fccrc", IXGBE_STAT(stats.fccrc)}, {"rx_fcoe_dropped", IXGBE_STAT(stats.fcoerpdc)},
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +12 -9
@@ -8346,16 +8346,19 @@ netdev_tx_t ixgbe_xmit_frame_ring(struct sk_buff *skb, protocol =vlan_get_protocol (skb);if (unlikely (skb_shinfo (skb)->tx_flags &SKBTX_HW_TSTAMP ) && - adapter->ptp_clock && - !test_and_set_bit_lock (__IXGBE_PTP_TX_IN_PROGRESS, - &adapter->state)) { -skb_shinfo (skb)->tx_flags |=SKBTX_IN_PROGRESS ; - tx_flags |=IXGBE_TX_FLAGS_TSTAMP ; + adapter->ptp_clock) { +if (!test_and_set_bit_lock (__IXGBE_PTP_TX_IN_PROGRESS, + &adapter->state)) { +skb_shinfo (skb)->tx_flags |=SKBTX_IN_PROGRESS ; + tx_flags |=IXGBE_TX_FLAGS_TSTAMP ; -/* schedule check for Tx timestamp */ - adapter->ptp_tx_skb =skb_get (skb); - adapter->ptp_tx_start = jiffies; -schedule_work (&adapter->ptp_tx_work); +/* schedule check for Tx timestamp */ + adapter->ptp_tx_skb =skb_get (skb); + adapter->ptp_tx_start = jiffies; +schedule_work (&adapter->ptp_tx_work); + }else { + adapter->tx_hwtstamp_skipped++; + } }skb_tx_timestamp (skb);
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
PATCH v4 12/15 i40e: use pf data structure directly in i40e_ptp_rx_hang
In reply to Jacob KellerThere's no reason to pass a *vsi pointer if we already have the *pf pointer in the only location where we call this function. Lets update the signature and directly pass the *pf data structure pointer.
--- drivers/net/ethernet/intel/i40e/i40e.h | 2 +- drivers/net/ethernet/intel/i40e/i40e_main.c | 2 +- drivers/net/ethernet/intel/i40e/i40e_ptp.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-)
drivers/net/ethernet/intel/i40e/i40e.h +1 -1
@@ -956,7 +956,7 @@ bool i40e_dcb_need_reconfig(struct i40e_pf *pf,struct i40e_dcbx_config *old_cfg, struct i40e_dcbx_config *new_cfg);#endif /* CONFIG_I40E_DCB */ -void i40e_ptp_rx_hang (struct i40e_vsi *vsi); +void i40e_ptp_rx_hang (struct i40e_pf *pf);void i40e_ptp_tx_hwtstamp (struct i40e_pf *pf);void i40e_ptp_rx_hwtstamp (struct i40e_pf *pf,struct sk_buff *skb,u8 index);void i40e_ptp_set_increment (struct i40e_pf *pf);
drivers/net/ethernet/intel/i40e/i40e_main.c +1 -1
@@ -6372,7 +6372,7 @@ static void i40e_watchdog_subtask(struct i40e_pf *pf)i40e_update_veb_stats (pf->veb[i]); } -i40e_ptp_rx_hang (pf->vsi[pf->lan_vsi]); +i40e_ptp_rx_hang (pf); }/**
drivers/net/ethernet/intel/i40e/i40e_ptp.c +2 -2
@@ -269,6 +269,7 @@ static u32 i40e_ptp_get_rx_events(struct i40e_pf *pf)/** * i40e_ptp_rx_hang - Detect error case when Rx timestamp registers are hung +* @pf: The PF private data structure * @vsi: The VSI with the rings relevant to 1588 * * This watchdog task is scheduled to detect error case where hardware has
@@ -276,9 +277,8 @@ static u32 i40e_ptp_get_rx_events(struct i40e_pf *pf)* particular error is rare but leaves the device in a state unable to timestamp * any future packets. **/ -void i40e_ptp_rx_hang (struct i40e_vsi *vsi) +void i40e_ptp_rx_hang (struct i40e_pf *pf) { -struct i40e_pf *pf = vsi->back;struct i40e_hw *hw = &pf->hw;unsigned int i, cleared =0 ;
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Bowers
PATCH v4 12/15 i40e: use pf data structure directly in i40e_ptp_rx_hang
Quoted 13 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 12/15] i40e: use pf data structure directly ini40e_ptp_rx_hangThere's no reason to pass a *vsi pointer if we already have the *pf pointer in the only location where we call this function. Lets update the signature and directly pass the *pf data structure pointer.
Quoted 6 lines
Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/i40e/i40e.h | 2 +- drivers/net/ethernet/intel/i40e/i40e_main.c | 2 +- drivers/net/ethernet/intel/i40e/i40e_ptp.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-)
PATCH v4 13/15 i40e: check for Tx timestamp timeouts during watchdog
In reply to Jacob KellerThe i40e driver has logic to handle only one Tx timestamp at a time, using a state bit lock to avoid multiple requests at once.
It may be possible, if incredibly unlikely, that a Tx timestamp event is requested but never completes. Since we use an interrupt scheme to determine when the Tx timestamp occurred we would never clear the state bit in this case.
Add an i40e_ptp_tx_hang() function similar to the already existing i40e_ptp_rx_hang() function. This function runs in the watchdog routine and makes sure we eventually recover from this case instead of permanently disabling Tx timestamps.
Note: there is no currently known way to cause this without hacking the driver code to force it.
--- drivers/net/ethernet/intel/i40e/i40e.h | 2 ++ drivers/net/ethernet/intel/i40e/i40e_main.c | 1 + drivers/net/ethernet/intel/i40e/i40e_ptp.c | 30 +++++++++++++++++++++++++++++ drivers/net/ethernet/intel/i40e/i40e_txrx.c | 1 + 4 files changed, 34 insertions(+)
drivers/net/ethernet/intel/i40e/i40e.h +2 -0
@@ -502,6 +502,7 @@ struct i40e_pf {struct ptp_clock *ptp_clock;struct ptp_clock_info ptp_caps;struct sk_buff *ptp_tx_skb; +unsigned long ptp_tx_start;struct hwtstamp_config tstamp_config;struct mutex tmreg_lock;/* Used to protect the SYSTIME registers. */ u64 ptp_base_adj;
@@ -957,6 +958,7 @@ bool i40e_dcb_need_reconfig(struct i40e_pf *pf, struct i40e_dcbx_config *new_cfg);#endif /* CONFIG_I40E_DCB */ void i40e_ptp_rx_hang (struct i40e_pf *pf); +void i40e_ptp_tx_hang (struct i40e_pf *pf);void i40e_ptp_tx_hwtstamp (struct i40e_pf *pf);void i40e_ptp_rx_hwtstamp (struct i40e_pf *pf,struct sk_buff *skb,u8 index);void i40e_ptp_set_increment (struct i40e_pf *pf);
drivers/net/ethernet/intel/i40e/i40e_main.c +1 -0
@@ -6373,6 +6373,7 @@ static void i40e_watchdog_subtask(struct i40e_pf *pf) }i40e_ptp_rx_hang (pf); +i40e_ptp_tx_hang (pf); }/**
drivers/net/ethernet/intel/i40e/i40e_ptp.c +30 -0
@@ -327,6 +327,36 @@ void i40e_ptp_rx_hang(struct i40e_pf *pf) pf->rx_hwtstamp_cleared += cleared; } +/** +* i40e_ptp_tx_hang - Detect error case when Tx timestamp register is hung +* @pf: The PF private data structure +* +* This watchdog task is run periodically to make sure that we clear the Tx +* timestamp logic if we don't obtain a timestamp in a reasonable amount of +* time. It is unexpected in the normal case but if it occurs it results in +* permanently prevent timestamps of future packets +**/ +void i40e_ptp_tx_hang (struct i40e_pf *pf) +{ +if (!(pf->flags &I40E_FLAG_PTP ) || !pf->ptp_tx) +return ; + +/* Nothing to do if we're not already waiting for a timestamp */ +if (!test_bit (__I40E_PTP_TX_IN_PROGRESS, pf->state)) +return ; + +/* We already have a handler routine which is run when we are notified +* of a Tx timestamp in the hardware. If we don't get an interrupt +* within a second it is reasonable to assume that we never will. +*/ +if (time_is_before_jiffies (pf->ptp_tx_start +HZ )) { +dev_kfree_skb_any (pf->ptp_tx_skb); + pf->ptp_tx_skb =NULL ; +clear_bit_unlock (__I40E_PTP_TX_IN_PROGRESS, pf->state); + pf->tx_hwtstamp_timeouts++; + } +} +/** * i40e_ptp_tx_hwtstamp - Utility function which returns the Tx timestamp * @pf: Board private structure
drivers/net/ethernet/intel/i40e/i40e_txrx.c +1 -0
@@ -2628,6 +2628,7 @@ static int i40e_tsyn(struct i40e_ring *tx_ring, struct sk_buff *skb,if (pf->ptp_tx && !test_and_set_bit_lock (__I40E_PTP_TX_IN_PROGRESS, pf->state)) {skb_shinfo (skb)->tx_flags |=SKBTX_IN_PROGRESS ; + pf->ptp_tx_start = jiffies; pf->ptp_tx_skb =skb_get (skb); }else { pf->tx_hwtstamp_skipped++;
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Bowers
PATCH v4 13/15 i40e: check for Tx timestamp timeouts during watchdog
Quoted 25 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 13/15] i40e: check for Tx timestamp timeouts during watchdogThe i40e driver has logic to handle only one Tx timestamp at a time, using a state bit lock to avoid multiple requests at once.
It may be possible, if incredibly unlikely, that a Tx timestamp event is requested but never completes. Since we use an interrupt scheme to determine when the Tx timestamp occurred we would never clear the state bit in this case.
Add an
i40e_ptp_tx_hang()function similar to the already existingi40e_ptp_rx_hang()function. This function runs in the watchdog routine and makes sure we eventually recover from this case instead of permanently disabling Tx timestamps.Note: there is no currently known way to cause this without hacking the driver code to force it.
Quoted 8 lines
Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/i40e/i40e.h | 2 ++ drivers/net/ethernet/intel/i40e/i40e_main.c | 1 + drivers/net/ethernet/intel/i40e/i40e_ptp.c | 30 +++++++++++++++++++++++++++++ drivers/net/ethernet/intel/i40e/i40e_txrx.c | 1 + 4 files changed, 34 insertions(+)
PATCH v4 14/15 igb: check for Tx timestamp timeouts during watchdog
In reply to Jacob KellerThe igb driver has logic to handle only one Tx timestamp at a time, using a state bit lock to avoid multiple requests at once.
It may be possible, if incredibly unlikely, that a Tx timestamp event is requested but never completes. Since we use an interrupt scheme to determine when the Tx timestamp occurred we would never clear the state bit in this case.
Add an igb_ptp_tx_hang() function similar to the already existing igb_ptp_rx_hang() function. This function runs in the watchdog routine and makes sure we eventually recover from this case instead of permanently disabling Tx timestamps.
Note: there is no currently known way to cause this without hacking the driver code to force it.
--- drivers/net/ethernet/intel/igb/igb.h | 1 + drivers/net/ethernet/intel/igb/igb_main.c | 1 + drivers/net/ethernet/intel/igb/igb_ptp.c | 29 +++++++++++++++++++++++++++++ 3 files changed, 31 insertions(+)
drivers/net/ethernet/intel/igb/igb.h +1 -0
@@ -677,6 +677,7 @@ void igb_ptp_stop(struct igb_adapter *adapter);void igb_ptp_reset (struct igb_adapter *adapter);void igb_ptp_suspend (struct igb_adapter *adapter);void igb_ptp_rx_hang (struct igb_adapter *adapter); +void igb_ptp_tx_hang (struct igb_adapter *adapter);void igb_ptp_rx_rgtstamp (struct igb_q_vector *q_vector,struct sk_buff *skb);void igb_ptp_rx_pktstamp (struct igb_q_vector *q_vector,void *va,struct sk_buff *skb);
drivers/net/ethernet/intel/igb/igb_main.c +1 -0
@@ -4726,6 +4726,7 @@ static void igb_watchdog_task(struct work_struct *work)igb_spoof_check (adapter);igb_ptp_rx_hang (adapter); +igb_ptp_tx_hang (adapter);/* Check LVMMC register on i350/i354 only */ if ((adapter->hw.mac.type == e1000_i350) ||
drivers/net/ethernet/intel/igb/igb_ptp.c +29 -0
@@ -711,6 +711,35 @@ void igb_ptp_rx_hang(struct igb_adapter *adapter) } } +/** +* igb_ptp_tx_hang - detect error case where Tx timestamp never finishes +* @adapter: private network adapter structure +*/ +void igb_ptp_tx_hang (struct igb_adapter *adapter) +{ +bool timeout =time_is_before_jiffies (adapter->ptp_tx_start + +IGB_PTP_TX_TIMEOUT ); + +if (!adapter->ptp_tx_skb) +return ; + +if (!test_bit (__IGB_PTP_TX_IN_PROGRESS, &adapter->state)) +return ; + +/* If we haven't received a timestamp within the timeout, it is +* reasonable to assume that it will never occur, so we can unlock the +* timestamp bit when this occurs. +*/ +if (timeout) { +cancel_work_sync (&adapter->ptp_tx_work); +dev_kfree_skb_any (adapter->ptp_tx_skb); + adapter->ptp_tx_skb =NULL ; +clear_bit_unlock (__IGB_PTP_TX_IN_PROGRESS, &adapter->state); + adapter->tx_hwtstamp_timeouts++; +dev_warn (&adapter->pdev->dev,"clearing Tx timestamp hang\n" ); + } +} +/** * igb_ptp_tx_hwtstamp - utility function which checks for TX time stamp * @adapter: Board private structure.
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Brown
PATCH v4 14/15 igb: check for Tx timestamp timeouts during watchdog
Quoted 24 lines
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 14/15] igb: check for Tx timestamp timeouts during watchdogThe igb driver has logic to handle only one Tx timestamp at a time, using a state bit lock to avoid multiple requests at once.
It may be possible, if incredibly unlikely, that a Tx timestamp event is requested but never completes. Since we use an interrupt scheme to determine when the Tx timestamp occurred we would never clear the state bit in this case.
Add an
igb_ptp_tx_hang()function similar to the already existingigb_ptp_rx_hang()function. This function runs in the watchdog routine and makes sure we eventually recover from this case instead of permanently disabling Tx timestamps.Note: there is no currently known way to cause this without hacking the driver code to force it.
Quoted 7 lines
Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/igb/igb.h | 1 + drivers/net/ethernet/intel/igb/igb_main.c | 1 + drivers/net/ethernet/intel/igb/igb_ptp.c | 29 +++++++++++++++++++++++++++++ 3 files changed, 31 insertions(+)
PATCH v4 15/15 ixgbe: check for Tx timestamp timeouts during watchdog
In reply to Jacob KellerThe ixgbe driver has logic to handle only one Tx timestamp at a time, using a state bit lock to avoid multiple requests at once.
It may be possible, if incredibly unlikely, that a Tx timestamp event is requested but never completes. Since we use an interrupt scheme to determine when the Tx timestamp occurred we would never clear the state bit in this case.
Add an ixgbe_ptp_tx_hang() function similar to the already existing ixgbe_ptp_rx_hang() function. This function runs in the watchdog routine and makes sure we eventually recover from this case instead of permanently disabling Tx timestamps.
Note: there is no currently known way to cause this without hacking the driver code to force it.
--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 1 + drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 1 + drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 27 +++++++++++++++++++++++++++ 3 files changed, 29 insertions(+)
drivers/net/ethernet/intel/ixgbe/ixgbe.h +1 -0
@@ -961,6 +961,7 @@ void ixgbe_ptp_suspend(struct ixgbe_adapter *adapter);void ixgbe_ptp_stop (struct ixgbe_adapter *adapter);void ixgbe_ptp_overflow_check (struct ixgbe_adapter *adapter);void ixgbe_ptp_rx_hang (struct ixgbe_adapter *adapter); +void ixgbe_ptp_tx_hang (struct ixgbe_adapter *adapter);void ixgbe_ptp_rx_pktstamp (struct ixgbe_q_vector *,struct sk_buff *);void ixgbe_ptp_rx_rgtstamp (struct ixgbe_q_vector *,struct sk_buff *skb); static inline void ixgbe_ptp_rx_hwtstamp(struct ixgbe_ring *rx_ring,
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +1 -0
@@ -7645,6 +7645,7 @@ static void ixgbe_service_task(struct work_struct *work)if (test_bit (__IXGBE_PTP_RUNNING, &adapter->state)) {ixgbe_ptp_overflow_check (adapter);ixgbe_ptp_rx_hang (adapter); +ixgbe_ptp_tx_hang (adapter); }ixgbe_service_event_complete (adapter);
drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c +27 -0
@@ -662,6 +662,33 @@ static void ixgbe_ptp_clear_tx_timestamp(struct ixgbe_adapter *adapter)clear_bit_unlock (__IXGBE_PTP_TX_IN_PROGRESS, &adapter->state); } +/** +* ixgbe_ptp_tx_hang - detect error case where Tx timestamp never finishes +* @adapter: private network adapter structure +*/ +void ixgbe_ptp_tx_hang (struct ixgbe_adapter *adapter) +{ +bool timeout =time_is_before_jiffies (adapter->ptp_tx_start + +IXGBE_PTP_TX_TIMEOUT ); + +if (!adapter->ptp_tx_skb) +return ; + +if (!test_bit (__IXGBE_PTP_TX_IN_PROGRESS, &adapter->state)) +return ; + +/* If we haven't received a timestamp within the timeout, it is +* reasonable to assume that it will never occur, so we can unlock the +* timestamp bit when this occurs. +*/ +if (timeout) { +cancel_work_sync (&adapter->ptp_tx_work); +ixgbe_ptp_clear_tx_timestamp (adapter); + adapter->tx_hwtstamp_timeouts++; +e_warn (drv,"clearing Tx timestamp hang\n" ); + } +} +/** * ixgbe_ptp_tx_hwtstamp - utility function which checks for TX time stamp * @adapter: the private adapter struct
Signature · 1 line
2.13.0.rc0.317.gcc792a6cad5a
Bowers
PATCH v4 15/15 ixgbe: check for Tx timestamp timeouts during watchdog
Quoted 25 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Wednesday, May 3, 2017 10:29 AM
To: Intel Wired LAN <[email protected]>
Cc: Richard Cochran <[email protected]>
Subject: [Intel-wired-lan] [PATCH v4 15/15] ixgbe: check for Tx timestamp timeouts during watchdogThe ixgbe driver has logic to handle only one Tx timestamp at a time, using a state bit lock to avoid multiple requests at once.
It may be possible, if incredibly unlikely, that a Tx timestamp event is requested but never completes. Since we use an interrupt scheme to determine when the Tx timestamp occurred we would never clear the state bit in this case.
Add an
ixgbe_ptp_tx_hang()function similar to the already existingixgbe_ptp_rx_hang()function. This function runs in the watchdog routine and makes sure we eventually recover from this case instead of permanently disabling Tx timestamps.Note: there is no currently known way to cause this without hacking the driver code to force it.
Quoted 7 lines
Signed-off-by Jacob Keller <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 1 + drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 1 + drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 27 +++++++++++++++++++++++++++ 3 files changed, 29 insertions(+)
v4 cover ptp cornercase cleanups
In reply to Jacob KellerQuoted 13 lines · Jacob Keller
This series contains fixes for similar corner cases in the e1000e, igb, i40e, and ixgbe drivers reported by David Mirabito on the ptp4l mailing list.
The first few patches fix a race condition with
skb_tstamp_tx(). The next few patches ensure that we cleanup the state bit when exiting the_xmit_frame_ring()function with a DMA or TSO failure. After this, we add new statistic to count how many times the Tx timestamp is skipped due to already outstanding requests. The final few patches add some handling in the potential case where we somehow never finish a timestamp request, ensuring that we don't permanently lock the Tx timestamps forever in this rare case.
Quoted 2 lines
Changes in v4
- Split the driver patches at request of Shannon Nelson
Thanks, looks good to me.
Quoted 41 lines
- Updated commit messages, but code content should be identical
I apologize in advance, the patchworks will not track this properly because I necessarily changed the commit subjects when splitting the patches apart by driver.
Jacob Keller (15):
e1000e: fix race condition aroundskb_tstamp_tx()
i40e: fix race condition withPTP_TX_IN_PROGRESSbits
igb: fix race condition withPTP_TX_IN_PROGRESSbits
ixgbe: fix race condition withPTP_TX_IN_PROGRESSbits
i40e: avoid permanent lock of*_PTP_TX_IN_PROGRESS
igb: avoid permanent lock of*_PTP_TX_IN_PROGRESS
ixgbe: avoid permanent lock of*_PTP_TX_IN_PROGRESS
e1000e: add statistic indicating number of skipped Tx timestamps
i40e: add statistic indicating number of skipped Tx timestamps
igb: add statistic indicating number of skipped Tx timestamps
ixgbe: add statistic indicating number of skipped Tx timestamps
i40e: use pf data structure directly ini40e_ptp_rx_hang
i40e: check for Tx timestamp timeouts during watchdog
igb: check for Tx timestamp timeouts during watchdog
ixgbe: check for Tx timestamp timeouts during watchdogdrivers/net/ethernet/intel/e1000e/e1000.h | 1 + drivers/net/ethernet/intel/e1000e/ethtool.c | 1 + drivers/net/ethernet/intel/e1000e/netdev.c | 27 ++++++++----- drivers/net/ethernet/intel/i40e/i40e.h | 5 ++- drivers/net/ethernet/intel/i40e/i40e_ethtool.c | 1 + drivers/net/ethernet/intel/i40e/i40e_main.c | 3 +- drivers/net/ethernet/intel/i40e/i40e_ptp.c | 48 +++++++++++++++++++++--- drivers/net/ethernet/intel/i40e/i40e_txrx.c | 28 +++++++++++--- drivers/net/ethernet/intel/igb/igb.h | 2 + drivers/net/ethernet/intel/igb/igb_ethtool.c | 1 + drivers/net/ethernet/intel/igb/igb_main.c | 26 ++++++++++--- drivers/net/ethernet/intel/igb/igb_ptp.c | 41 +++++++++++++++++++- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 2 + drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 3 ++ drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 42 ++++++++++++++------- drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 42 +++++++++++++++++++-- 16 files changed, 227 insertions(+), 46 deletions(-)
v4 cover ptp cornercase cleanups
In reply to Jacob KellerHi Jacob,
I'm just testing this patchset on IGB as applied to the upstream net-next repo (4.12.0-rc5+), and the problem persists. It seems to occur more seldom than without the patch, but I'm still seeing occasional messages along the lines of
ptp4l [1122.159]: timed out while polling for tx timestamp ptp4l [1122.159]: increasing tx_timestamp_timeout may correct this issue, but it is likely caused by a driver bug
roughly every 10 minutes. Other than ptp4l running, the machine is basically idle. Is there something missing in net-next?
Thanks,
Corinna
Quoted 47 lines · Jacob Keller
This series contains fixes for similar corner cases in the e1000e, igb, i40e, and ixgbe drivers reported by David Mirabito on the ptp4l mailing list.
The first few patches fix a race condition with
skb_tstamp_tx(). The next few patches ensure that we cleanup the state bit when exiting the_xmit_frame_ring()function with a DMA or TSO failure. After this, we add new statistic to count how many times the Tx timestamp is skipped due to already outstanding requests. The final few patches add some handling in the potential case where we somehow never finish a timestamp request, ensuring that we don't permanently lock the Tx timestamps forever in this rare case.Changes in v4
- Split the driver patches at request of Shannon Nelson
- Updated commit messages, but code content should be identicalI apologize in advance, the patchworks will not track this properly because I necessarily changed the commit subjects when splitting the patches apart by driver.
Jacob Keller (15):
e1000e: fix race condition aroundskb_tstamp_tx()
i40e: fix race condition withPTP_TX_IN_PROGRESSbits
igb: fix race condition withPTP_TX_IN_PROGRESSbits
ixgbe: fix race condition withPTP_TX_IN_PROGRESSbits
i40e: avoid permanent lock of*_PTP_TX_IN_PROGRESS
igb: avoid permanent lock of*_PTP_TX_IN_PROGRESS
ixgbe: avoid permanent lock of*_PTP_TX_IN_PROGRESS
e1000e: add statistic indicating number of skipped Tx timestamps
i40e: add statistic indicating number of skipped Tx timestamps
igb: add statistic indicating number of skipped Tx timestamps
ixgbe: add statistic indicating number of skipped Tx timestamps
i40e: use pf data structure directly ini40e_ptp_rx_hang
i40e: check for Tx timestamp timeouts during watchdog
igb: check for Tx timestamp timeouts during watchdog
ixgbe: check for Tx timestamp timeouts during watchdogdrivers/net/ethernet/intel/e1000e/e1000.h | 1 + drivers/net/ethernet/intel/e1000e/ethtool.c | 1 + drivers/net/ethernet/intel/e1000e/netdev.c | 27 ++++++++----- drivers/net/ethernet/intel/i40e/i40e.h | 5 ++- drivers/net/ethernet/intel/i40e/i40e_ethtool.c | 1 + drivers/net/ethernet/intel/i40e/i40e_main.c | 3 +- drivers/net/ethernet/intel/i40e/i40e_ptp.c | 48 +++++++++++++++++++++--- drivers/net/ethernet/intel/i40e/i40e_txrx.c | 28 +++++++++++--- drivers/net/ethernet/intel/igb/igb.h | 2 +
Quoted 16 lines
drivers/net/ethernet/intel/igb/igb_ethtool.c | 1 + drivers/net/ethernet/intel/igb/igb_main.c | 26 ++++++++++--- drivers/net/ethernet/intel/igb/igb_ptp.c | 41 +++++++++++++++++++- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 2 + drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 3 ++ drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 42 ++++++++++++++------- drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 42 +++++++++++++++++++-- 16 files changed, 227 insertions(+), 46 deletions(-)Signature · 6 lines
2.13.0.rc0.317.gcc792a6cad5a _______________________________________________ Intel-wired-lan mailing list Intel-wired-lan at lists.osuosl.org http://lists.osuosl.org/mailman/listinfo/intel-wired-lan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20170621/9aba6ac5/attachment.asc>
v4 cover ptp cornercase cleanups
For what it's worth, I found that although the patches solved the case where a tx timestamp would *never* come (on an I354/igb), we still had to set "tx_timestamp_timeout 750" in ptp4l.conf as there were still the occasional case where it'd come, just a bit later than ptp4l seems to expect. YMMV of course, but we needed both the patch and this config change at once in order to be stable.
Note 750 was just a number that worked. It may be that a smaller number would suffice but it seems OK for us.
Cheers,
- DavidM
Quoted 88 lines · Corinna Vinschen
Hi Jacob,
I'm just testing this patchset on IGB as applied to the upstream net-next repo (4.12.0-rc5+), and the problem persists. It seems to occur more seldom than without the patch, but I'm still seeing occasional messages along the lines of
ptp4l [1122.159]: timed out while polling for tx timestamp ptp4l [1122.159]: increasing
tx_timestamp_timeoutmay correct this issue, but it is likely caused by a driver bugroughly every 10 minutes. Other than ptp4l running, the machine is basically idle. Is there something missing in net-next?
Thanks,
CorinnaQuoted 38 lines · Jacob Keller
This series contains fixes for similar corner cases in the e1000e, igb, i40e, and ixgbe drivers reported by David Mirabito on the ptp4l mailing list.
The first few patches fix a race condition with
skb_tstamp_tx(). The next few patches ensure that we cleanup the state bit when exiting the_xmit_frame_ring()function with a DMA or TSO failure. After this, we add new statistic to count how many times the Tx timestamp is skipped due to already outstanding requests. The final few patches add some handling in the potential case where we somehow never finish a timestamp request, ensuring that we don't permanently lock the Tx timestamps forever in this rare case.Changes in v4
- Split the driver patches at request of Shannon Nelson
- Updated commit messages, but code content should be identicalI apologize in advance, the patchworks will not track this properly because I necessarily changed the commit subjects when splitting the patches apart by driver.
Jacob Keller (15):
e1000e: fix race condition aroundskb_tstamp_tx()
i40e: fix race condition withPTP_TX_IN_PROGRESSbits
igb: fix race condition withPTP_TX_IN_PROGRESSbits
ixgbe: fix race condition withPTP_TX_IN_PROGRESSbits
i40e: avoid permanent lock of*_PTP_TX_IN_PROGRESS
igb: avoid permanent lock of*_PTP_TX_IN_PROGRESS
ixgbe: avoid permanent lock of*_PTP_TX_IN_PROGRESS
e1000e: add statistic indicating number of skipped Tx timestamps
i40e: add statistic indicating number of skipped Tx timestamps
igb: add statistic indicating number of skipped Tx timestamps
ixgbe: add statistic indicating number of skipped Tx timestamps
i40e: use pf data structure directly ini40e_ptp_rx_hang
i40e: check for Tx timestamp timeouts during watchdog
igb: check for Tx timestamp timeouts during watchdog
ixgbe: check for Tx timestamp timeouts during watchdogQuoted 7 lines
drivers/net/ethernet/intel/e1000e/e1000.h | 1 + drivers/net/ethernet/intel/e1000e/ethtool.c | 1 + drivers/net/ethernet/intel/e1000e/netdev.c | 27 ++++++++----- drivers/net/ethernet/intel/i40e/i40e.h | 5 ++- drivers/net/ethernet/intel/i40e/i40e_ethtool.c | 1 + drivers/net/ethernet/intel/i40e/i40e_main.c | 3 +- drivers/net/ethernet/intel/i40e/i40e_ptp.c | 48+++++++++++++++++++++---
Quoted 5 lines
drivers/net/ethernet/intel/i40e/i40e_txrx.c | 28 +++++++++++--- drivers/net/ethernet/intel/igb/igb.h | 2 + drivers/net/ethernet/intel/igb/igb_ethtool.c | 1 + drivers/net/ethernet/intel/igb/igb_main.c | 26 ++++++++++--- drivers/net/ethernet/intel/igb/igb_ptp.c | 41+++++++++++++++++++-
Quoted 3 lines
drivers/net/ethernet/intel/ixgbe/ixgbe.h | 2 + drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 3 ++ drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 42++++++++++++++-------
Quoted 1 line
drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 42+++++++++++++++++++--
Quoted 9 lines
16 files changed, 227 insertions(+), 46 deletions(-)--
2.13.0.rc0.317.gcc792a6cad5a_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at lists.osuosl.org
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Quoted 6 lines
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20170622/75da2d2f/attachment-0001.html>
v4 cover ptp cornercase cleanups
Hi David,
Quoted 9 lines · David Mirabito
For what it's worth, I found that although the patches solved the case where a tx timestamp would *never* come (on an I354/igb), we still had to set "
tx_timestamp_timeout750" in ptp4l.conf as there were still the occasional case where it'd come, just a bit later than ptp4l seems to expect. YMMV of course, but we needed both the patch and this config change at once in order to be stable.Note 750 was just a number that worked. It may be that a smaller number would suffice but it seems OK for us.
Thanks for the info. I'm just giving it a try. A value of 100 appears to work fine in my environment. Interestingly, I only have to raise tx_timestamp_timeout on i350, not on an i210 in the same machine.
Thanks,
Corinna
Quoted 101 lines
Cheers,
- DavidMQuoted 88 lines · Corinna Vinschen
Hi Jacob,
I'm just testing this patchset on IGB as applied to the upstream net-next repo (4.12.0-rc5+), and the problem persists. It seems to occur more seldom than without the patch, but I'm still seeing occasional messages along the lines of
ptp4l [1122.159]: timed out while polling for tx timestamp ptp4l [1122.159]: increasing
tx_timestamp_timeoutmay correct this issue, but it is likely caused by a driver bugroughly every 10 minutes. Other than ptp4l running, the machine is basically idle. Is there something missing in net-next?
Thanks,
CorinnaQuoted 38 lines · Jacob Keller
This series contains fixes for similar corner cases in the e1000e, igb, i40e, and ixgbe drivers reported by David Mirabito on the ptp4l mailing list.
The first few patches fix a race condition with
skb_tstamp_tx(). The next few patches ensure that we cleanup the state bit when exiting the_xmit_frame_ring()function with a DMA or TSO failure. After this, we add new statistic to count how many times the Tx timestamp is skipped due to already outstanding requests. The final few patches add some handling in the potential case where we somehow never finish a timestamp request, ensuring that we don't permanently lock the Tx timestamps forever in this rare case.Changes in v4
- Split the driver patches at request of Shannon Nelson
- Updated commit messages, but code content should be identicalI apologize in advance, the patchworks will not track this properly because I necessarily changed the commit subjects when splitting the patches apart by driver.
Jacob Keller (15):
e1000e: fix race condition aroundskb_tstamp_tx()
i40e: fix race condition withPTP_TX_IN_PROGRESSbits
igb: fix race condition withPTP_TX_IN_PROGRESSbits
ixgbe: fix race condition withPTP_TX_IN_PROGRESSbits
i40e: avoid permanent lock of*_PTP_TX_IN_PROGRESS
igb: avoid permanent lock of*_PTP_TX_IN_PROGRESS
ixgbe: avoid permanent lock of*_PTP_TX_IN_PROGRESS
e1000e: add statistic indicating number of skipped Tx timestamps
i40e: add statistic indicating number of skipped Tx timestamps
igb: add statistic indicating number of skipped Tx timestamps
ixgbe: add statistic indicating number of skipped Tx timestamps
i40e: use pf data structure directly ini40e_ptp_rx_hang
i40e: check for Tx timestamp timeouts during watchdog
igb: check for Tx timestamp timeouts during watchdog
ixgbe: check for Tx timestamp timeouts during watchdogQuoted 7 lines
drivers/net/ethernet/intel/e1000e/e1000.h | 1 + drivers/net/ethernet/intel/e1000e/ethtool.c | 1 + drivers/net/ethernet/intel/e1000e/netdev.c | 27 ++++++++----- drivers/net/ethernet/intel/i40e/i40e.h | 5 ++- drivers/net/ethernet/intel/i40e/i40e_ethtool.c | 1 + drivers/net/ethernet/intel/i40e/i40e_main.c | 3 +- drivers/net/ethernet/intel/i40e/i40e_ptp.c | 48+++++++++++++++++++++---
Quoted 5 lines
drivers/net/ethernet/intel/i40e/i40e_txrx.c | 28 +++++++++++--- drivers/net/ethernet/intel/igb/igb.h | 2 + drivers/net/ethernet/intel/igb/igb_ethtool.c | 1 + drivers/net/ethernet/intel/igb/igb_main.c | 26 ++++++++++--- drivers/net/ethernet/intel/igb/igb_ptp.c | 41+++++++++++++++++++-
Quoted 3 lines
drivers/net/ethernet/intel/ixgbe/ixgbe.h | 2 + drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 3 ++ drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 42++++++++++++++-------
Quoted 1 line
drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 42+++++++++++++++++++--
Quoted 9 lines
16 files changed, 227 insertions(+), 46 deletions(-)--
2.13.0.rc0.317.gcc792a6cad5a_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at lists.osuosl.org
http://lists.osuosl.org/mailman/listinfo/intel-wired-lanQuoted 6 lines
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Quoted 4 lines
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20170622/c8c79e5d/attachment.asc>