Skip to contentSkip to conversation

ice: Improve miscellaneous interrupt code

PATCH In review Jacob Keller 3 participants 12 messages Jun 1, 2023 – Jun 8, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v3 cover ice: Improve miscellaneous interrupt code

This series improves the driver's use of the threaded IRQ and the communication between ice_misc_intr() and the ice_misc_intr_thread_fn() which was previously introduced by commit 1229b33973c7 ("ice: Add low latency Tx timestamp read").

First, a new custom enumerated return value is used instead of a boolean for ice_ptp_process_ts(). This significantly reduces the cognitive burden when reviewing the logic for this function, as the expected action is clear from the return value name.

Second, the unconditional loop in ice_misc_intr_thread_fn() is removed, replacing it with a write to the Other Interrupt Cause register. This causes the MAC to trigger the Tx timestamp interrupt again. This makes it possible to safely use the ice_misc_intr_thread_fn() to handle other tasks beyond just the Tx timestamps. It is also easier to reason about since the thread function will exit cleanly if we do something like disable the interrupt and call synchronize_irq().

Third, refactor the handling for external timestamp events to use the miscellaneous thread function. This resolves an issue with the external time stamps getting blocked while processing the periodic work function task.

Fourth, a simplification of the ice_misc_intr() function to always return IRQ_WAKE_THREAD, and schedule the ice service task in the ice_misc_intr_thread_fn() instead.

Finally, the Other Interrupt Cause is kept disabled over the thread function processing, rather than immediately re-enabled.

Special thanks to Michal Schmidt for the careful review of the series and pointing out my misunderstandings of the kernel IRQ code. It has been determined that the race outlined as being fixed in previous series was actually introduced by this series itself, which I've since corrected.

Changes since v2:
* heavily re-ordered patches for clarity and a better flow of implementation
* used atomic bits when communicating between ice_misc_intr() and ice_misc_intr_thread_fn()
* re-wrote commit messages to improve clarity and remove mistakes about how threaded IRQs work.

Jacob Keller (3):
ice: introduce ICE_TX_TSTAMP_WORK enumeration
ice: trigger PFINT_OICR_TSYN_TX interrupt instead of polling
ice: do not re-enable miscellaneous interrupt until thread_fn
completes

Karol Kolacinski (2):
ice: handle extts in the miscellaneous interrupt thread
ice: always return IRQ_WAKE_THREAD in ice_misc_intr()

 drivers/net/ethernet/intel/ice/ice.h      |  7 +++
 drivers/net/ethernet/intel/ice/ice_main.c | 47 +++++++++++------
 drivers/net/ethernet/intel/ice/ice_ptp.c  | 62 ++++++++++++-----------
 drivers/net/ethernet/intel/ice/ice_ptp.h  | 16 ++++--
 4 files changed, 84 insertions(+), 48 deletions(-)

base-commit: f08ff053a47a51d4f391d407bdda6adb4e7ed499

Signature · 6 lines
2.40.0.471.gbd7f14d9353b

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
#1 lore raw

PATCH v3 1/5 ice: handle extts in the miscellaneous interrupt thread

From: Karol Kolacinski <[email protected]>

The ice_ptp_extts_work() and ice_ptp_periodic_work() functions are both scheduled on the same kthread worker, pf.ptp.kworker. The ice_ptp_periodic_work() function sends to the firmware to interact with the PHY, and must block to wait for responses.

This can cause delay in responding to the PFINT_OICR_TSYN_EVNT interrupt cause, ultimately resulting in disruption to processing an input signal of the frequency is high enough. In our testing, even 100 Hz signals get disrupted.

Fix this by instead processing the signal inside the miscellaneous interrupt thread prior to handling Tx timestamps.

Use atomic bits in a new pf->misc_thread bitmap in order to safely communicate which tasks require processing within the ice_misc_intr_thread_fn(). This ensures the communication of desired tasks from the ice_misc_intr() are correctly processed without racing even in the event that the interrupt triggers again before the thread function exits.

Fixes 172db5f91d5f ("ice: add support for auxiliary input/output pins")
---
 drivers/net/ethernet/intel/ice/ice.h      |  7 ++++++
 drivers/net/ethernet/intel/ice/ice_main.c | 29 ++++++++++++++++-------
 drivers/net/ethernet/intel/ice/ice_ptp.c  | 12 +++-------
 drivers/net/ethernet/intel/ice/ice_ptp.h  |  4 ++--
 4 files changed, 33 insertions(+), 19 deletions(-)
drivers/net/ethernet/intel/ice/ice.h +7 -0
index 176e281dfa24..9109006336f0 100644
@@ -509,6 +509,12 @@ enum ice_pf_flags {
 	ICE_PF_FLAGS_NBITS		/* must be last */
 };
 
+enum ice_misc_thread_tasks {
+	ICE_MISC_THREAD_EXTTS_EVENT,
+	ICE_MISC_THREAD_TX_TSTAMP,
+	ICE_MISC_THREAD_NBITS		/* must be last */
+};
+
 struct ice_switchdev_info {
 	struct ice_vsi *control_vsi;
 	struct ice_vsi *uplink_vsi;
@@ -552,6 +558,7 @@ struct ice_pf {
 	DECLARE_BITMAP(features, ICE_F_MAX);
 	DECLARE_BITMAP(state, ICE_STATE_NBITS);
 	DECLARE_BITMAP(flags, ICE_PF_FLAGS_NBITS);
+	DECLARE_BITMAP(misc_thread, ICE_MISC_THREAD_NBITS);
 	unsigned long *avail_txqs;	/* bitmap to track PF Tx queue usage */
 	unsigned long *avail_rxqs;	/* bitmap to track PF Rx queue usage */
 	unsigned long serv_tmr_period;
drivers/net/ethernet/intel/ice/ice_main.c +21 -8
index c7d7104dbadc..0beb6afc0b2d 100644
@@ -3139,20 +3139,28 @@ static irqreturn_t ice_misc_intr(int __always_unused irq, void *data)
 
 	if (oicr & PFINT_OICR_TSYN_TX_M) {
 		ena_mask &= ~PFINT_OICR_TSYN_TX_M;
-		if (!hw->reset_ongoing)
+		if (!hw->reset_ongoing) {
+			set_bit(ICE_MISC_THREAD_TX_TSTAMP, pf->misc_thread);
 			ret = IRQ_WAKE_THREAD;
+		}
 	}
 
 	if (oicr & PFINT_OICR_TSYN_EVNT_M) {
 		u8 tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
 		u32 gltsyn_stat = rd32(hw, GLTSYN_STAT(tmr_idx));
 
-		/* Save EVENTs from GTSYN register */
-		pf->ptp.ext_ts_irq |= gltsyn_stat & (GLTSYN_STAT_EVENT0_M |
-						     GLTSYN_STAT_EVENT1_M |
-						     GLTSYN_STAT_EVENT2_M);
 		ena_mask &= ~PFINT_OICR_TSYN_EVNT_M;
-		kthread_queue_work(pf->ptp.kworker, &pf->ptp.extts_work);
+
+		if (hw->func_caps.ts_func_info.src_tmr_owned) {
+			/* Save EVENTs from GLTSYN register */
+			pf->ptp.ext_ts_irq |= gltsyn_stat &
+					      (GLTSYN_STAT_EVENT0_M |
+					       GLTSYN_STAT_EVENT1_M |
+					       GLTSYN_STAT_EVENT2_M);
+
+			set_bit(ICE_MISC_THREAD_EXTTS_EVENT, pf->misc_thread);
+			ret = IRQ_WAKE_THREAD;
+		}
 	}
 
 #define ICE_AUX_CRIT_ERR (PFINT_OICR_PE_CRITERR_M | PFINT_OICR_HMC_ERR_M | PFINT_OICR_PE_PUSH_M)
@@ -3196,8 +3204,13 @@ static irqreturn_t ice_misc_intr_thread_fn(int __always_unused irq, void *data)
 	if (ice_is_reset_in_progress(pf->state))
 		return IRQ_HANDLED;
 
-	while (!ice_ptp_process_ts(pf))
-		usleep_range(50, 100);
+	if (test_and_clear_bit(ICE_MISC_THREAD_EXTTS_EVENT, pf->misc_thread))
+		ice_ptp_extts_event(pf);
+
+	if (test_and_clear_bit(ICE_MISC_THREAD_TX_TSTAMP, pf->misc_thread)) {
+		while (!ice_ptp_process_ts(pf))
+			usleep_range(50, 100);
+	}
 
 	return IRQ_HANDLED;
 }
drivers/net/ethernet/intel/ice/ice_ptp.c +3 -9
index d4b6c997141d..6f51ebaf1d70 100644
@@ -1458,15 +1458,11 @@ static int ice_ptp_adjfine(struct ptp_clock_info *info, long scaled_ppm)
 }
 
 /**
- * ice_ptp_extts_work - Workqueue task function
- * @work: external timestamp work structure
- *
- * Service for PTP external clock event
+ * ice_ptp_extts_event - Process PTP external clock event
+ * @pf: Board private structure
  */
-static void ice_ptp_extts_work(struct kthread_work *work)
+void ice_ptp_extts_event(struct ice_pf *pf)
 {
-	struct ice_ptp *ptp = container_of(work, struct ice_ptp, extts_work);
-	struct ice_pf *pf = container_of(ptp, struct ice_pf, ptp);
 	struct ptp_clock_event event;
 	struct ice_hw *hw = &pf->hw;
 	u8 chan, tmr_idx;
@@ -2558,7 +2554,6 @@ void ice_ptp_prepare_for_reset(struct ice_pf *pf)
 	ice_ptp_cfg_timestamp(pf, false);
 
 	kthread_cancel_delayed_work_sync(&ptp->work);
-	kthread_cancel_work_sync(&ptp->extts_work);
 
 	if (test_bit(ICE_PFR_REQ, pf->state))
 		return;
@@ -2656,7 +2651,6 @@ static int ice_ptp_init_work(struct ice_pf *pf, struct ice_ptp *ptp)
 
 	/* Initialize work functions */
 	kthread_init_delayed_work(&ptp->work, ice_ptp_periodic_work);
-	kthread_init_work(&ptp->extts_work, ice_ptp_extts_work);
 
 	/* Allocate a kworker for handling work required for the ports
 	 * connected to the PTP hardware clock.
drivers/net/ethernet/intel/ice/ice_ptp.h +2 -2
index 9cda2f43e0e5..9f8902c1e743 100644
@@ -169,7 +169,6 @@ struct ice_ptp_port {
  * struct ice_ptp - data used for integrating with CONFIG_PTP_1588_CLOCK
  * @port: data for the PHY port initialization procedure
  * @work: delayed work function for periodic tasks
- * @extts_work: work function for handling external Tx timestamps
  * @cached_phc_time: a cached copy of the PHC time for timestamp extension
  * @cached_phc_jiffies: jiffies when cached_phc_time was last updated
  * @ext_ts_chan: the external timestamp channel in use
@@ -190,7 +189,6 @@ struct ice_ptp_port {
 struct ice_ptp {
 	struct ice_ptp_port port;
 	struct kthread_delayed_work work;
-	struct kthread_work extts_work;
 	u64 cached_phc_time;
 	unsigned long cached_phc_jiffies;
 	u8 ext_ts_chan;
@@ -256,6 +254,7 @@ int ice_ptp_get_ts_config(struct ice_pf *pf, struct ifreq *ifr);
 void ice_ptp_cfg_timestamp(struct ice_pf *pf, bool ena);
 int ice_get_ptp_clock_index(struct ice_pf *pf);
 
+void ice_ptp_extts_event(struct ice_pf *pf);
 s8 ice_ptp_request_ts(struct ice_ptp_tx *tx, struct sk_buff *skb);
 bool ice_ptp_process_ts(struct ice_pf *pf);
 
@@ -284,6 +283,7 @@ static inline int ice_get_ptp_clock_index(struct ice_pf *pf)
 	return -1;
 }
 
+static inline void ice_ptp_extts_event(struct ice_pf *pf) { }
 static inline s8
 ice_ptp_request_ts(struct ice_ptp_tx *tx, struct sk_buff *skb)
 {
Signature · 6 lines
2.40.0.471.gbd7f14d9353b

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Tested-by on v3 1/5
#2 lore raw

Re: [Intel-wired-lan] [PATCH iwl-next v3 1/5] ice: handle extts in the miscellaneous interrupt thread

Quoted 18 lines

-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Jacob Keller
Sent: Friday, June 2, 2023 2:45 AM
To: Intel Wired LAN <[email protected]>; Nguyen, Anthony L <[email protected]>
Cc: Kolacinski, Karol <[email protected]>
Subject: [Intel-wired-lan] [PATCH iwl-next v3 1/5] ice: handle extts in the miscellaneous interrupt thread

From: Karol Kolacinski <[email protected]>

The ice_ptp_extts_work() and ice_ptp_periodic_work() functions are both scheduled on the same kthread worker, pf.ptp.kworker. The ice_ptp_periodic_work() function sends to the firmware to interact with the PHY, and must block to wait for responses.

This can cause delay in responding to the PFINT_OICR_TSYN_EVNT interrupt cause, ultimately resulting in disruption to processing an input signal of the frequency is high enough. In our testing, even 100 Hz > signals get disrupted.

Fix this by instead processing the signal inside the miscellaneous interrupt thread prior to handling Tx timestamps.

Use atomic bits in a new pf->misc_thread bitmap in order to safely communicate which tasks require processing within the ice_misc_intr_thread_fn(). This ensures the communication of desired tasks from > the ice_misc_intr() are correctly processed without racing even in the event that the interrupt triggers again before the thread function exits.

Quoted 8 lines
Fixes 172db5f91d5f ("ice: add support for auxiliary input/output pins")
---
 drivers/net/ethernet/intel/ice/ice.h      |  7 ++++++
 drivers/net/ethernet/intel/ice/ice_main.c | 29 ++++++++++++++++-------  drivers/net/ethernet/intel/ice/ice_ptp.c  | 12 +++-------  drivers/net/ethernet/intel/ice/ice_ptp.h  |  4 ++--
 4 files changed, 33 insertions(+), 19 deletions(-)

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#3 lore raw

PATCH v3 2/5 ice: always return IRQ_WAKE_THREAD in ice_misc_intr()

In reply to Jacob Keller

From: Karol Kolacinski <[email protected]>

Refactor the ice_misc_intr() function to always return IRQ_WAKE_THREAD, and schedule the service task during the soft IRQ thread function instead of at the end of the hard IRQ handler.

Remove the duplicate call to ice_service_task_schedule() that happened when we got a PCI exception.

---
 drivers/net/ethernet/intel/ice/ice_main.c | 14 ++++----------
 1 file changed, 4 insertions(+), 10 deletions(-)
drivers/net/ethernet/intel/ice/ice_main.c +4 -10
index 0beb6afc0b2d..34599088633d 100644
@@ -3058,7 +3058,6 @@ static irqreturn_t ice_misc_intr(int __always_unused irq, void *data)
 {
 	struct ice_pf *pf = (struct ice_pf *)data;
 	struct ice_hw *hw = &pf->hw;
-	irqreturn_t ret = IRQ_NONE;
 	struct device *dev;
 	u32 oicr, ena_mask;
 
@@ -3139,10 +3138,8 @@ static irqreturn_t ice_misc_intr(int __always_unused irq, void *data)
 
 	if (oicr & PFINT_OICR_TSYN_TX_M) {
 		ena_mask &= ~PFINT_OICR_TSYN_TX_M;
-		if (!hw->reset_ongoing) {
+		if (!hw->reset_ongoing)
 			set_bit(ICE_MISC_THREAD_TX_TSTAMP, pf->misc_thread);
-			ret = IRQ_WAKE_THREAD;
-		}
 	}
 
 	if (oicr & PFINT_OICR_TSYN_EVNT_M) {
@@ -3159,7 +3156,6 @@ static irqreturn_t ice_misc_intr(int __always_unused irq, void *data)
 					       GLTSYN_STAT_EVENT2_M);
 
 			set_bit(ICE_MISC_THREAD_EXTTS_EVENT, pf->misc_thread);
-			ret = IRQ_WAKE_THREAD;
 		}
 	}
 
@@ -3180,16 +3176,12 @@ static irqreturn_t ice_misc_intr(int __always_unused irq, void *data)
 		if (oicr & (PFINT_OICR_PCI_EXCEPTION_M |
 			    PFINT_OICR_ECC_ERR_M)) {
 			set_bit(ICE_PFR_REQ, pf->state);
-			ice_service_task_schedule(pf);
 		}
 	}
-	if (!ret)
-		ret = IRQ_HANDLED;
 
-	ice_service_task_schedule(pf);
 	ice_irq_dynamic_ena(hw, NULL, NULL);
 
-	return ret;
+	return IRQ_WAKE_THREAD;
 }
 
 /**
@@ -3204,6 +3196,8 @@ static irqreturn_t ice_misc_intr_thread_fn(int __always_unused irq, void *data)
 	if (ice_is_reset_in_progress(pf->state))
 		return IRQ_HANDLED;
 
+	ice_service_task_schedule(pf);
+
 	if (test_and_clear_bit(ICE_MISC_THREAD_EXTTS_EVENT, pf->misc_thread))
 		ice_ptp_extts_event(pf);
 
Signature · 6 lines
2.40.0.471.gbd7f14d9353b

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Tested-by on v3 2/5
#4 lore raw

Re: [Intel-wired-lan] [PATCH iwl-next v3 2/5] ice: always return IRQ_WAKE_THREAD in ice_misc_intr()

Quoted 11 lines

-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Jacob Keller
Sent: Friday, June 2, 2023 2:45 AM
To: Intel Wired LAN <[email protected]>; Nguyen, Anthony L <[email protected]>
Cc: Kolacinski, Karol <[email protected]>
Subject: [Intel-wired-lan] [PATCH iwl-next v3 2/5] ice: always return IRQ_WAKE_THREAD in ice_misc_intr()

From: Karol Kolacinski <[email protected]>

Refactor the ice_misc_intr() function to always return IRQ_WAKE_THREAD, and schedule the service task during the soft IRQ thread function instead of at the end of the hard IRQ handler.

Quoted 8 lines

Remove the duplicate call to ice_service_task_schedule() that happened when we got a PCI exception.

---
 drivers/net/ethernet/intel/ice/ice_main.c | 14 ++++----------
 1 file changed, 4 insertions(+), 10 deletions(-)

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#5 lore raw

PATCH v3 3/5 ice: introduce ICE_TX_TSTAMP_WORK enumeration

In reply to Jacob Keller

The ice_ptp_process_ts() function and its various helper functions return a boolean value indicating whether any work is remaining. This use of a boolean has grown confusing as we have multiple helpers that pass status between each other. Readers must be aware of what "true" and "false" mean, and it is very easy to get their meaning inverted. The names of the functions are not standard "yes/no" questions, which is the best practice for boolean returns.

Replace this use of an enumeration with a custom type, enum ice_tx_tstamp_work. This enumeration clearly indicates whether all work is done, or if more work is pending.

To aid in readability, factor the actual list iteration and processing out into ice_ptp_process_tx_tstamp(), making it void. Then call this in ice_ptp_tx_tstamp() ensuring that we always check the Tracker list at the end when determining the appropriate return value.

Now the return value is an explicit name instead of the true or false value. This is easier to follow and makes reading the resulting callers much simpler.

In addition, this paves the way for future work to allow E822 hardware to process timestamps for all functions using a single interrupt on the clock owning PF.

---
 drivers/net/ethernet/intel/ice/ice_main.c |  2 +-
 drivers/net/ethernet/intel/ice/ice_ptp.c  | 50 ++++++++++++++---------
 drivers/net/ethernet/intel/ice/ice_ptp.h  | 12 +++++-
 3 files changed, 42 insertions(+), 22 deletions(-)
drivers/net/ethernet/intel/ice/ice_main.c +1 -1
index 34599088633d..9df53dd300a9 100644
@@ -3202,7 +3202,7 @@ static irqreturn_t ice_misc_intr_thread_fn(int __always_unused irq, void *data)
 		ice_ptp_extts_event(pf);
 
 	if (test_and_clear_bit(ICE_MISC_THREAD_TX_TSTAMP, pf->misc_thread)) {
-		while (!ice_ptp_process_ts(pf))
+		while (ice_ptp_process_ts(pf) == ICE_TX_TSTAMP_WORK_PENDING)
 			usleep_range(50, 100);
 	}
 
drivers/net/ethernet/intel/ice/ice_ptp.c +30 -20

100 lines not on this page · raw message

drivers/net/ethernet/intel/ice/ice_ptp.h +11 -1
index 9f8902c1e743..6c90775e1eb0 100644
@@ -108,6 +108,16 @@ struct ice_tx_tstamp {
 	u64 cached_tstamp;
 };
 
+/**
+ * enum ice_tx_tstamp_work - Status of Tx timestamp work function
+ * @ICE_TX_TSTAMP_WORK_DONE - Tx timestamp processing is complete
+ * @ICE_TX_TSTAMP_WORK_PENDING - More Tx timestamps are pending
+ */
+enum ice_tx_tstamp_work {
+	ICE_TX_TSTAMP_WORK_DONE = 0,
+	ICE_TX_TSTAMP_WORK_PENDING,
+};
+
 /**
  * struct ice_ptp_tx - Tracking structure for all Tx timestamp requests on a port
  * @lock: lock to prevent concurrent access to fields of this struct
@@ -256,7 +266,7 @@ int ice_get_ptp_clock_index(struct ice_pf *pf);
 
 void ice_ptp_extts_event(struct ice_pf *pf);
 s8 ice_ptp_request_ts(struct ice_ptp_tx *tx, struct sk_buff *skb);
-bool ice_ptp_process_ts(struct ice_pf *pf);
+enum ice_tx_tstamp_work ice_ptp_process_ts(struct ice_pf *pf);
 
 void
 ice_ptp_rx_hwtstamp(struct ice_rx_ring *rx_ring,
Signature · 6 lines
2.40.0.471.gbd7f14d9353b

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Tested-by on v3 3/5
#6 lore raw

Re: [Intel-wired-lan] [PATCH iwl-next v3 3/5] ice: introduce ICE_TX_TSTAMP_WORK enumeration

Quoted 13 lines

-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Jacob Keller
Sent: Friday, June 2, 2023 2:45 AM
To: Intel Wired LAN <[email protected]>; Nguyen, Anthony L <[email protected]>
Subject: [Intel-wired-lan] [PATCH iwl-next v3 3/5] ice: introduce ICE_TX_TSTAMP_WORK enumeration

The ice_ptp_process_ts() function and its various helper functions return a boolean value indicating whether any work is remaining. This use of a boolean has grown confusing as we have multiple helpers that pass status between each other. Readers must be aware of what "true" and "false" mean, and it is very easy to get their meaning inverted. The names of the functions are not standard "yes/no" questions, which is the best practice for boolean returns.

Replace this use of an enumeration with a custom type, enum ice_tx_tstamp_work. This enumeration clearly indicates whether all work is done, or if more work is pending.

To aid in readability, factor the actual list iteration and processing out into ice_ptp_process_tx_tstamp(), making it void. Then call this in ice_ptp_tx_tstamp() ensuring that we always check the Tracker list at the end when determining the appropriate return value.

Quoted 9 lines

Now the return value is an explicit name instead of the true or false value. This is easier to follow and makes reading the resulting callers much simpler.

In addition, this paves the way for future work to allow E822 hardware to process timestamps for all functions using a single interrupt on the clock owning PF.

---
 drivers/net/ethernet/intel/ice/ice_main.c |  2 +-  drivers/net/ethernet/intel/ice/ice_ptp.c  | 50 ++++++++++++++---------  drivers/net/ethernet/intel/ice/ice_ptp.h  | 12 +++++-
 3 files changed, 42 insertions(+), 22 deletions(-)

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#7 lore raw

PATCH v3 4/5 ice: trigger PFINT_OICR_TSYN_TX interrupt instead of polling

In reply to Jacob Keller

In ice_misc_intr_thread_fn(), if we do not complete all Tx timestamp work, the thread function will poll continuously forever.

For E822 hardware, this wastes time as the return value from ice_ptp_process_ts() is accurate and always reports correctly that the PHY actually has new timestamp data.

In addition, if we receive enough timestamps with the right pacing, we may never exit this polling. Should this occur, other tasks handled by the ice_misc_intr_thread_fn() will never be processed.

Fix this by instead writing to PFINT_OICR, causing an emulated interrupt to be triggered immediately. This does take slightly more processing than just re-checking the timestamps. However, it allows all of the other interrupt causes a chance to be processed first in the hard IRQ function.

Note that the OICR interrupt is configured to be throttled to no more than once every 124 microseconds. This gives an effective interrupt rate of ~8000 interrupts per second. This should thus not cause a significant increase in overall CPU usage when compared to sleeping.

---
 drivers/net/ethernet/intel/ice/ice_main.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/ice/ice_main.c +9 -2
index 9df53dd300a9..8f3ddc8451bd 100644
@@ -3202,8 +3202,15 @@ static irqreturn_t ice_misc_intr_thread_fn(int __always_unused irq, void *data)
 		ice_ptp_extts_event(pf);
 
 	if (test_and_clear_bit(ICE_MISC_THREAD_TX_TSTAMP, pf->misc_thread)) {
-		while (ice_ptp_process_ts(pf) == ICE_TX_TSTAMP_WORK_PENDING)
-			usleep_range(50, 100);
+		struct ice_hw *hw = &pf->hw;
+
+		/* Process outstanding Tx timestamps. If there is more work,
+		 * re-arm the interrupt to trigger again.
+		 */
+		if (ice_ptp_process_ts(pf) == ICE_TX_TSTAMP_WORK_PENDING) {
+			wr32(hw, PFINT_OICR, PFINT_OICR_TSYN_TX_M);
+			ice_flush(hw);
+		}
 	}
 
 	return IRQ_HANDLED;
Signature · 6 lines
2.40.0.471.gbd7f14d9353b

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Tested-by on v3 4/5
#8 lore raw

Re: [Intel-wired-lan] [PATCH iwl-next v3 4/5] ice: trigger PFINT_OICR_TSYN_TX interrupt instead of polling

Quoted 16 lines

-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Jacob Keller
Sent: Friday, June 2, 2023 2:45 AM
To: Intel Wired LAN <[email protected]>; Nguyen, Anthony L <[email protected]>
Subject: [Intel-wired-lan] [PATCH iwl-next v3 4/5] ice: trigger PFINT_OICR_TSYN_TX interrupt instead of polling

In ice_misc_intr_thread_fn(), if we do not complete all Tx timestamp work, the thread function will poll continuously forever.

For E822 hardware, this wastes time as the return value from
ice_ptp_process_ts() is accurate and always reports correctly that the PHY actually has new timestamp data.

In addition, if we receive enough timestamps with the right pacing, we may never exit this polling. Should this occur, other tasks handled by the ice_misc_intr_thread_fn() will never be processed.

Fix this by instead writing to PFINT_OICR, causing an emulated interrupt to be triggered immediately. This does take slightly more processing than just re-checking the timestamps. However, it allows all of the other interrupt causes a chance to be processed first in the hard IRQ function.

Quoted 8 lines

Note that the OICR interrupt is configured to be throttled to no more than once every 124 microseconds. This gives an effective interrupt rate of ~8000 interrupts per second. This should thus not cause a significant increase in overall CPU usage when compared to sleeping.

---
 drivers/net/ethernet/intel/ice/ice_main.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#9 lore raw

PATCH v3 5/5 ice: do not re-enable miscellaneous interrupt until thread_fn completes

In reply to Jacob Keller

The ice driver uses threaded IRQ for managing Tx timestamps via the devm_request_threaded_irq() interface. The ice_misc_intr() handler function is responsible for processing the hard interrupt context, and can wake the ice_misc_intr_thread_fn() by returning IRQ_WAKE_THREAD.

The request_threaded_irq() function comment says:

@handler is still called in hard interrupt context and has to check whether the interrupt originates from the device. If yes, it needs to disable the interrupt on the device and return IRQ_WAKE_THREAD which will wake up the handler thread and run the @thread_fn.

We currently re-enable the Other Interrupt Cause Register (OCIR) at the end of ice_misc_intr(). In practice, this seems to be ok, but it can make communicating between the handler function and the thread function difficult. This is because the interrupt can trigger again while the thread function is still processing.

Move the OICR update to the end of the thread function, leaving the other interrupt cause disabled in hardware until we complete one pass of the thread function. This prevents the miscellaneous interrupt from firing until after we finish the thread function.

---
 drivers/net/ethernet/intel/ice/ice_main.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/net/ethernet/intel/ice/ice_main.c +5 -4
index 8f3ddc8451bd..6eae43828c46 100644
@@ -3179,8 +3179,6 @@ static irqreturn_t ice_misc_intr(int __always_unused irq, void *data)
 		}
 	}
 
-	ice_irq_dynamic_ena(hw, NULL, NULL);
-
 	return IRQ_WAKE_THREAD;
 }
 
@@ -3192,6 +3190,9 @@ static irqreturn_t ice_misc_intr(int __always_unused irq, void *data)
 static irqreturn_t ice_misc_intr_thread_fn(int __always_unused irq, void *data)
 {
 	struct ice_pf *pf = data;
+	struct ice_hw *hw;
+
+	hw = &pf->hw;
 
 	if (ice_is_reset_in_progress(pf->state))
 		return IRQ_HANDLED;
@@ -3202,8 +3203,6 @@ static irqreturn_t ice_misc_intr_thread_fn(int __always_unused irq, void *data)
 		ice_ptp_extts_event(pf);
 
 	if (test_and_clear_bit(ICE_MISC_THREAD_TX_TSTAMP, pf->misc_thread)) {
-		struct ice_hw *hw = &pf->hw;
-
 		/* Process outstanding Tx timestamps. If there is more work,
 		 * re-arm the interrupt to trigger again.
 		 */
@@ -3213,6 +3212,8 @@ static irqreturn_t ice_misc_intr_thread_fn(int __always_unused irq, void *data)
 		}
 	}
 
+	ice_irq_dynamic_ena(hw, NULL, NULL);
+
 	return IRQ_HANDLED;
 }
 
Signature · 6 lines
2.40.0.471.gbd7f14d9353b

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Tested-by on v3 5/5
#10 lore raw

Re: [Intel-wired-lan] [PATCH iwl-next v3 5/5] ice: do not re-enable miscellaneous interrupt until thread_fn completes

Quoted 17 lines

-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Jacob Keller
Sent: Friday, June 2, 2023 2:45 AM
To: Intel Wired LAN <[email protected]>; Nguyen, Anthony L <[email protected]>
Subject: [Intel-wired-lan] [PATCH iwl-next v3 5/5] ice: do not re-enable miscellaneous interrupt until thread_fn completes

The ice driver uses threaded IRQ for managing Tx timestamps via the
devm_request_threaded_irq() interface. The ice_misc_intr() handler function is responsible for processing the hard interrupt context, and can wake the ice_misc_intr_thread_fn() by returning IRQ_WAKE_THREAD.

The request_threaded_irq() function comment says:

@handler is still called in hard interrupt context and has to check whether the interrupt originates from the device. If yes, it needs to disable the interrupt on the device and return IRQ_WAKE_THREAD which will wake up the handler thread and run the @thread_fn.

Quoted 9 lines

We currently re-enable the Other Interrupt Cause Register (OCIR) at the end of ice_misc_intr(). In practice, this seems to be ok, but it can make communicating between the handler function and the thread > function difficult. This is because the interrupt can trigger again while the thread function is still processing.

Move the OICR update to the end of the thread function, leaving the other interrupt cause disabled in hardware until we complete one pass of the thread function. This prevents the miscellaneous interrupt > from firing until after we finish the thread function.

---
 drivers/net/ethernet/intel/ice/ice_main.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Reviewed-by on the v3 series
#11 lore raw
In reply to Jacob Keller
Quoted 60 lines · Jacob Keller

This series improves the driver's use of the threaded IRQ and the communication between ice_misc_intr() and the ice_misc_intr_thread_fn() which was previously introduced by commit 1229b33973c7 ("ice: Add low latency Tx timestamp read").

First, a new custom enumerated return value is used instead of a boolean for ice_ptp_process_ts(). This significantly reduces the cognitive burden when reviewing the logic for this function, as the expected action is clear from the return value name.

Second, the unconditional loop in ice_misc_intr_thread_fn() is removed, replacing it with a write to the Other Interrupt Cause register. This causes the MAC to trigger the Tx timestamp interrupt again. This makes it possible to safely use the ice_misc_intr_thread_fn() to handle other tasks beyond just the Tx timestamps. It is also easier to reason about since the thread function will exit cleanly if we do something like disable the interrupt and call synchronize_irq().

Third, refactor the handling for external timestamp events to use the miscellaneous thread function. This resolves an issue with the external time stamps getting blocked while processing the periodic work function task.

Fourth, a simplification of the ice_misc_intr() function to always return IRQ_WAKE_THREAD, and schedule the ice service task in the ice_misc_intr_thread_fn() instead.

Finally, the Other Interrupt Cause is kept disabled over the thread function processing, rather than immediately re-enabled.

Special thanks to Michal Schmidt for the careful review of the series and pointing out my misunderstandings of the kernel IRQ code. It has been determined that the race outlined as being fixed in previous series was actually introduced by this series itself, which I've since corrected.

Changes since v2:
* heavily re-ordered patches for clarity and a better flow of implementation
* used atomic bits when communicating between ice_misc_intr() and ice_misc_intr_thread_fn()
* re-wrote commit messages to improve clarity and remove mistakes about how threaded IRQs work.

Jacob Keller (3):
ice: introduce ICE_TX_TSTAMP_WORK enumeration
ice: trigger PFINT_OICR_TSYN_TX interrupt instead of polling
ice: do not re-enable miscellaneous interrupt until thread_fn
completes

Karol Kolacinski (2):
ice: handle extts in the miscellaneous interrupt thread
ice: always return IRQ_WAKE_THREAD in ice_misc_intr()

 drivers/net/ethernet/intel/ice/ice.h      |  7 +++
 drivers/net/ethernet/intel/ice/ice_main.c | 47 +++++++++++------
 drivers/net/ethernet/intel/ice/ice_ptp.c  | 62 ++++++++++++-----------
 drivers/net/ethernet/intel/ice/ice_ptp.h  | 16 ++++--
 4 files changed, 84 insertions(+), 48 deletions(-)
Quoted 3 lines

base-commit: f08ff053a47a51d4f391d407bdda6adb4e7ed499 --
2.40.0.471.gbd7f14d9353b

Thanks, it looks good now. To the series:

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan