Skip to contentSkip to conversation

igc: PTP tx fixes

PATCH New Andre Guedes 2 participants 11 messages Jul 28, 2020 – Aug 24, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover igc: PTP tx fixes

Hi all,

This patch series fixes some issues in the PTP tx code from the IGC driver. The most relevant one is a race condition between igc_xmit_frame_ring() and igc_ptp_tx_hang() which is fixed by the last patch.

- Andre

Andre Guedes (4):
igc: Rename IGC_TSYNCTXCTL_VALID macro
igc: Don't reschedule ptp_tx work
igc: Remove timeout check from ptp_tx work
igc: Fix race condition in PTP tx code

 drivers/net/ethernet/intel/igc/igc.h         |  5 +-
 drivers/net/ethernet/intel/igc/igc_defines.h |  2 +-
 drivers/net/ethernet/intel/igc/igc_main.c    |  7 ++-
 drivers/net/ethernet/intel/igc/igc_ptp.c     | 60 +++++++++++---------
 4 files changed, 43 insertions(+), 31 deletions(-)
Signature · 1 line
2.26.2
#1 lore raw

PATCH 1/4 igc: Rename IGC_TSYNCTXCTL_VALID macro

This patch renames the IGC_TSYNCTXCTL_VALID macro to IGC_TSYNCTXCTL_ TXTT_0 so it matches the datasheet.

---
 drivers/net/ethernet/intel/igc/igc_defines.h | 2 +-
 drivers/net/ethernet/intel/igc/igc_ptp.c     | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igc/igc_defines.h +1 -1
index 1c032c1ce4d3..28885be15ee8 100644
@@ -409,7 +409,7 @@
 #define IGC_IMIREXT_SIZE_BP	0x00001000  /* Packet size bypass */
 
 /* Time Sync Transmit Control bit definitions */
-#define IGC_TSYNCTXCTL_VALID			0x00000001  /* Tx timestamp valid */
+#define IGC_TSYNCTXCTL_TXTT_0			0x00000001  /* Tx timestamp reg 0 valid */
 #define IGC_TSYNCTXCTL_ENABLED			0x00000010  /* enable Tx timestamping */
 #define IGC_TSYNCTXCTL_MAX_ALLOWED_DLY_MASK	0x0000F000  /* max delay */
 #define IGC_TSYNCTXCTL_SYNC_COMP_ERR		0x20000000  /* sync err */
drivers/net/ethernet/intel/igc/igc_ptp.c +1 -1
index e67d4655b47e..79802fb4ea83 100644
@@ -410,7 +410,7 @@ static void igc_ptp_tx_work(struct work_struct *work)
 	}
 
 	tsynctxctl = rd32(IGC_TSYNCTXCTL);
-	if (tsynctxctl & IGC_TSYNCTXCTL_VALID)
+	if (tsynctxctl & IGC_TSYNCTXCTL_TXTT_0)
 		igc_ptp_tx_hwtstamp(adapter);
 	else
 		/* reschedule to check later */
Signature · 1 line
2.26.2
#2 lore raw

PATCH 1/4 igc: Rename IGC_TSYNCTXCTL_VALID macro

Quoted 7 lines

From: Intel-wired-lan <[email protected]> On Behalf Of Andre Guedes
Sent: Tuesday, July 28, 2020 4:38 PM
To: intel-wired-lan at lists.osuosl.org
Subject: [Intel-wired-lan] [PATCH 1/4] igc: Rename IGC_TSYNCTXCTL_VALID macro

Quoted 5 lines

This patch renames the IGC_TSYNCTXCTL_VALID macro to IGC_TSYNCTXCTL_ TXTT_0 so it matches the datasheet.

---

#3 lore raw

PATCH 2/4 igc: Don't reschedule ptp_tx work

In reply to Andre Guedes

The ptp_tx work is scheduled only if TSICR.TXTS bit is set, therefore TSYNCTXCTL.TXTT_0 bit is expected to be set when we check it igc_ptp_tx_ work(). If it isn't, something is really off and rescheduling the ptp_tx work to check it later doesn't help much. This patch changes the code to WARN_ON_ONCE() if this situation ever happens.

---
 drivers/net/ethernet/intel/igc/igc_ptp.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
drivers/net/ethernet/intel/igc/igc_ptp.c +4 -5
index 79802fb4ea83..b912830ea7bb 100644
@@ -410,11 +410,10 @@ static void igc_ptp_tx_work(struct work_struct *work)
 	}
 
 	tsynctxctl = rd32(IGC_TSYNCTXCTL);
-	if (tsynctxctl & IGC_TSYNCTXCTL_TXTT_0)
-		igc_ptp_tx_hwtstamp(adapter);
-	else
-		/* reschedule to check later */
-		schedule_work(&adapter->ptp_tx_work);
+	if (WARN_ON_ONCE(!(tsynctxctl & IGC_TSYNCTXCTL_TXTT_0)))
+		return;
+
+	igc_ptp_tx_hwtstamp(adapter);
 }
 
 /**
Signature · 1 line
2.26.2
#4 lore raw

PATCH 2/4 igc: Don't reschedule ptp_tx work

Quoted 6 lines

From: Intel-wired-lan <[email protected]> On Behalf Of Andre Guedes
Sent: Tuesday, July 28, 2020 4:38 PM
To: intel-wired-lan at lists.osuosl.org
Subject: [Intel-wired-lan] [PATCH 2/4] igc: Don't reschedule ptp_tx work

Quoted 8 lines

The ptp_tx work is scheduled only if TSICR.TXTS bit is set, therefore TSYNCTXCTL.TXTT_0 bit is expected to be set when we check it igc_ptp_tx_ work(). If it isn't, something is really off and rescheduling the ptp_tx work to check it later doesn't help much. This patch changes the code to WARN_ON_ONCE() if this situation ever happens.

---

#5 lore raw

PATCH 3/4 igc: Remove timeout check from ptp_tx work

In reply to Andre Guedes

The tx timestamp timeout is already checked by the watchdog_task work which runs periodically. In addition to that, from the ptp_tx work perspective, if __IGC_PTP_TX_IN_PROGRESS flag is set we always want handle the timestamp stored in hardware and update the skb. So this patch removes the timeout check in igc_ptp_tx_work() function.

---
 drivers/net/ethernet/intel/igc/igc_ptp.c | 6 ------
 1 file changed, 6 deletions(-)
drivers/net/ethernet/intel/igc/igc_ptp.c +0 -6
index b912830ea7bb..22b2cf80b22f 100644
@@ -403,12 +403,6 @@ static void igc_ptp_tx_work(struct work_struct *work)
 	if (!test_bit(__IGC_PTP_TX_IN_PROGRESS, &adapter->state))
 		return;
 
-	if (time_is_before_jiffies(adapter->ptp_tx_start +
-				   IGC_PTP_TX_TIMEOUT)) {
-		igc_ptp_tx_timeout(adapter);
-		return;
-	}
-
 	tsynctxctl = rd32(IGC_TSYNCTXCTL);
 	if (WARN_ON_ONCE(!(tsynctxctl & IGC_TSYNCTXCTL_TXTT_0)))
 		return;
Signature · 1 line
2.26.2
#6 lore raw

PATCH 3/4 igc: Remove timeout check from ptp_tx work

Quoted 13 lines

From: Intel-wired-lan <[email protected]> On Behalf Of Andre Guedes
Sent: Tuesday, July 28, 2020 4:38 PM
To: intel-wired-lan at lists.osuosl.org
Subject: [Intel-wired-lan] [PATCH 3/4] igc: Remove timeout check from ptp_tx work

The tx timestamp timeout is already checked by the watchdog_task work which runs periodically. In addition to that, from the ptp_tx work perspective, if __IGC_PTP_TX_IN_PROGRESS flag is set we always want handle the timestamp stored in hardware and update the skb. So this patch removes the timeout check in igc_ptp_tx_work() function.

Quoted 5 lines
---
 drivers/net/ethernet/intel/igc/igc_ptp.c | 6 ------
 1 file changed, 6 deletions(-)
#7 lore raw

PATCH 4/4 igc: Fix race condition in PTP tx code

In reply to Andre Guedes

Currently, the igc driver supports timestamping only one tx packet at a time. During the transmission flow, the skb that requires hardware timestamping is saved in adapter->ptp_tx_skb. Once hardware has the timestamp, an interrupt is delivered, and adapter->ptp_tx_work is scheduled. In igc_ptp_tx_work(), we read the timestamp register, update adapter->ptp_tx_skb, and notify the network stack.

While the thread executing the transmission flow (the user process running in kernel mode) and the thread executing ptp_tx_work don't access adapter->ptp_tx_skb concurrently, there are two other places where adapter->ptp_tx_skb is accessed: igc_ptp_tx_hang() and igc_ptp_suspend().

igc_ptp_tx_hang() is executed by the adapter->watchdog_task worker thread which runs periodically so it is possible we have two threads accessing ptp_tx_skb at the same time. Consider the following scenario:
right after __IGC_PTP_TX_IN_PROGRESS is set in igc_xmit_frame_ring(), igc_ptp_tx_hang() is executed. Since adapter->ptp_tx_start hasn't been written yet, this is considered a timeout and adapter->ptp_tx_skb is cleaned up.

This patch fixes the issue described above by adding the ptp_tx_lock to protect access to ptp_tx_skb and ptp_tx_start fields from igc_adapter. Since igc_xmit_frame_ring() called in atomic context by the networking stack, ptp_tx_lock is defined as a spinlock.

With the introduction of the ptp_tx_lock, the __IGC_PTP_TX_IN_PROGRESS flag doesn't provide much of a use anymore so this patch gets rid of it.

---
 drivers/net/ethernet/intel/igc/igc.h      |  5 ++-
 drivers/net/ethernet/intel/igc/igc_main.c |  7 +++-
 drivers/net/ethernet/intel/igc/igc_ptp.c  | 49 ++++++++++++++---------
 3 files changed, 40 insertions(+), 21 deletions(-)
drivers/net/ethernet/intel/igc/igc.h +4 -1
index 3070dfdb7eb4..19f88f705ec3 100644
@@ -207,6 +207,10 @@ struct igc_adapter {
 	struct ptp_clock *ptp_clock;
 	struct ptp_clock_info ptp_caps;
 	struct work_struct ptp_tx_work;
+	/* Access to ptp_tx_skb and ptp_tx_start is protected by the
+	 * ptp_tx_lock.
+	 */
+	spinlock_t ptp_tx_lock;
 	struct sk_buff *ptp_tx_skb;
 	struct hwtstamp_config tstamp_config;
 	unsigned long ptp_tx_start;
@@ -361,7 +365,6 @@ enum igc_state_t {
 	__IGC_TESTING,
 	__IGC_RESETTING,
 	__IGC_DOWN,
-	__IGC_PTP_TX_IN_PROGRESS,
 };
 
 enum igc_tx_flags {
drivers/net/ethernet/intel/igc/igc_main.c +5 -2
index ab5b302d6655..d6b37d91cad9 100644
@@ -1347,13 +1347,14 @@ static netdev_tx_t igc_xmit_frame_ring(struct sk_buff *skb,
 	if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)) {
 		struct igc_adapter *adapter = netdev_priv(tx_ring->netdev);
 
+		spin_lock(&adapter->ptp_tx_lock);
+
 		/* FIXME: add support for retrieving timestamps from
 		 * the other timer registers before skipping the
 		 * timestamping request.
 		 */
 		if (adapter->tstamp_config.tx_type == HWTSTAMP_TX_ON &&
-		    !test_and_set_bit_lock(__IGC_PTP_TX_IN_PROGRESS,
-					   &adapter->state)) {
+		    !adapter->ptp_tx_skb) {
 			skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
 			tx_flags |= IGC_TX_FLAGS_TSTAMP;
 
@@ -1362,6 +1363,8 @@ static netdev_tx_t igc_xmit_frame_ring(struct sk_buff *skb,
 		} else {
 			adapter->tx_hwtstamp_skipped++;
 		}
+
+		spin_unlock(&adapter->ptp_tx_lock);
 	}
 
 	/* record initial flags and protocol */
drivers/net/ethernet/intel/igc/igc_ptp.c +31 -18

108 lines not on this page · raw message

Signature · 1 line
2.26.2
#8 lore raw

PATCH 4/4 igc: Fix race condition in PTP tx code

Quoted 35 lines

From: Intel-wired-lan <[email protected]> On Behalf Of Andre Guedes
Sent: Tuesday, July 28, 2020 4:38 PM
To: intel-wired-lan at lists.osuosl.org
Subject: [Intel-wired-lan] [PATCH 4/4] igc: Fix race condition in PTP tx code

Currently, the igc driver supports timestamping only one tx packet at a time. During the transmission flow, the skb that requires hardware timestamping is saved in adapter->ptp_tx_skb. Once hardware has the timestamp, an interrupt is delivered, and adapter->ptp_tx_work is scheduled. In igc_ptp_tx_work(), we read the timestamp register, update adapter->ptp_tx_skb, and notify the network stack.

While the thread executing the transmission flow (the user process running in kernel mode) and the thread executing ptp_tx_work don't access adapter->ptp_tx_skb concurrently, there are two other places where adapter->ptp_tx_skb is accessed: igc_ptp_tx_hang() and igc_ptp_suspend().

igc_ptp_tx_hang() is executed by the adapter->watchdog_task worker thread which runs periodically so it is possible we have two threads accessing ptp_tx_skb at the same time. Consider the following scenario:
right after __IGC_PTP_TX_IN_PROGRESS is set in igc_xmit_frame_ring(), igc_ptp_tx_hang() is executed. Since adapter->ptp_tx_start hasn't been written yet, this is considered a timeout and adapter->ptp_tx_skb is cleaned up.

This patch fixes the issue described above by adding the ptp_tx_lock to protect access to ptp_tx_skb and ptp_tx_start fields from igc_adapter. Since igc_xmit_frame_ring() called in atomic context by the networking stack, ptp_tx_lock is defined as a spinlock.

With the introduction of the ptp_tx_lock, the __IGC_PTP_TX_IN_PROGRESS flag doesn't provide much of a use anymore so this patch gets rid of it.

Quoted 7 lines
---
 drivers/net/ethernet/intel/igc/igc.h      |  5 ++-
 drivers/net/ethernet/intel/igc/igc_main.c |  7 +++-
 drivers/net/ethernet/intel/igc/igc_ptp.c  | 49 ++++++++++++++---------
 3 files changed, 40 insertions(+), 21 deletions(-)
#9 lore raw

PATCH 4/4 igc: Fix race condition in PTP tx code

Hi Jeff/Tony,

Quoting Brown, Aaron F (2020-08-13 20:09:42)

Quoted 35 lines
Quoted 35 lines

From: Intel-wired-lan <[email protected]> On Behalf Of Andre Guedes
Sent: Tuesday, July 28, 2020 4:38 PM
To: intel-wired-lan at lists.osuosl.org
Subject: [Intel-wired-lan] [PATCH 4/4] igc: Fix race condition in PTP tx code

Currently, the igc driver supports timestamping only one tx packet at a time. During the transmission flow, the skb that requires hardware timestamping is saved in adapter->ptp_tx_skb. Once hardware has the timestamp, an interrupt is delivered, and adapter->ptp_tx_work is scheduled. In igc_ptp_tx_work(), we read the timestamp register, update adapter->ptp_tx_skb, and notify the network stack.

While the thread executing the transmission flow (the user process running in kernel mode) and the thread executing ptp_tx_work don't access adapter->ptp_tx_skb concurrently, there are two other places where adapter->ptp_tx_skb is accessed: igc_ptp_tx_hang() and igc_ptp_suspend().

igc_ptp_tx_hang() is executed by the adapter->watchdog_task worker thread which runs periodically so it is possible we have two threads accessing ptp_tx_skb at the same time. Consider the following scenario:
right after __IGC_PTP_TX_IN_PROGRESS is set in igc_xmit_frame_ring(), igc_ptp_tx_hang() is executed. Since adapter->ptp_tx_start hasn't been written yet, this is considered a timeout and adapter->ptp_tx_skb is cleaned up.

This patch fixes the issue described above by adding the ptp_tx_lock to protect access to ptp_tx_skb and ptp_tx_start fields from igc_adapter. Since igc_xmit_frame_ring() called in atomic context by the networking stack, ptp_tx_lock is defined as a spinlock.

With the introduction of the ptp_tx_lock, the __IGC_PTP_TX_IN_PROGRESS flag doesn't provide much of a use anymore so this patch gets rid of it.

Quoted 8 lines
Quoted 7 lines
---
 drivers/net/ethernet/intel/igc/igc.h      |  5 ++-
 drivers/net/ethernet/intel/igc/igc_main.c |  7 +++-
 drivers/net/ethernet/intel/igc/igc_ptp.c  | 49 ++++++++++++++---------
 3 files changed, 40 insertions(+), 21 deletions(-)

Please hold this patch back. I think I found an issue with it. I'm investigating it and should send a v2 soon.

Cheers,
Andre

#10 lore raw

PATCH 4/4 igc: Fix race condition in PTP tx code

Quoted 67 lines · Andre Guedes

Hi Jeff/Tony,

Quoting Brown, Aaron F (2020-08-13 20:09:42)

Quoted 64 lines
Quoted 63 lines

From: Intel-wired-lan <[email protected]> On Behalf Of
Andre Guedes
Sent: Tuesday, July 28, 2020 4:38 PM
To: intel-wired-lan at lists.osuosl.org
Subject: [Intel-wired-lan] [PATCH 4/4] igc: Fix race condition in PTP tx code

Currently, the igc driver supports timestamping only one tx packet at a
time. During the transmission flow, the skb that requires hardware
timestamping is saved in adapter->ptp_tx_skb. Once hardware has the
timestamp, an interrupt is delivered, and adapter->ptp_tx_work is scheduled. In igc_ptp_tx_work(), we read the timestamp register, update
adapter->ptp_tx_skb, and notify the network stack.

While the thread executing the transmission flow (the user process
running in kernel mode) and the thread executing ptp_tx_work don't
access adapter->ptp_tx_skb concurrently, there are two other places
where adapter->ptp_tx_skb is accessed: igc_ptp_tx_hang() and igc_ptp_suspend().

igc_ptp_tx_hang() is executed by the adapter->watchdog_task worker
thread which runs periodically so it is possible we have two threads
accessing ptp_tx_skb at the same time. Consider the following scenario:
right after __IGC_PTP_TX_IN_PROGRESS is set in igc_xmit_frame_ring(),
igc_ptp_tx_hang() is executed. Since adapter->ptp_tx_start hasn't been
written yet, this is considered a timeout and adapter->ptp_tx_skb is
cleaned up.

This patch fixes the issue described above by adding the ptp_tx_lock to
protect access to ptp_tx_skb and ptp_tx_start fields from igc_adapter.
Since igc_xmit_frame_ring() called in atomic context by the networking
stack, ptp_tx_lock is defined as a spinlock.

With the introduction of the ptp_tx_lock, the __IGC_PTP_TX_IN_PROGRESS
flag doesn't provide much of a use anymore so this patch gets rid of it.

---
 drivers/net/ethernet/intel/igc/igc.h      |  5 ++-
 drivers/net/ethernet/intel/igc/igc_main.c |  7 +++-
 drivers/net/ethernet/intel/igc/igc_ptp.c  | 49 ++++++++++++++---
------
 3 files changed, 40 insertions(+), 21 deletions(-)
Quoted 4 lines
Quoted 1 line

Please hold this patch back. I think I found an issue with it. I'm investigating it and should send a v2 soon.

Thanks Andre, I've dropped this patch from dev-queue.