Skip to contentSkip to conversation

net/intel: fix link status when interrupt delivery unavailable

PATCH Superseded Ciara Loftus 2 participants 10 messages Jun 30 – Jul 10 Network drivers

lore
Share & tools
lore
#0 lore raw

cover net/intel: fix link status when interrupt delivery unavailable

On FreeBSD with nic_uio, interrupt registration fails silently and hardware events, including link state notifications, are never processed via the interrupt path.

For ice, this affects the typical polling-mode configuration: the interrupt handler is never invoked, AdminQ messages go unprocessed, and link status is not reliable. A periodic alarm is added to drain the AdminQ when interrupt delivery is unavailable. For i40e, an alarm already runs unconditionally in polling mode, so that configuration is unaffected. The gap is the interrupt-mode configuration, where the existing alarm was not activated as a fallback when interrupt enable fails; this patch extends it to do so.

Ciara Loftus (2):
net/ice: poll AdminQ if interrupt delivery unavailable net/i40e: activate alarm if interrupt delivery unavailable

 drivers/net/intel/i40e/i40e_ethdev.c |  9 +++++-
 drivers/net/intel/i40e/i40e_ethdev.h |  2 ++
 drivers/net/intel/ice/ice_ethdev.c   | 45 +++++++++++++++++++++++++++-
 drivers/net/intel/ice/ice_ethdev.h   |  1 +
 4 files changed, 55 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/2 net/ice: poll AdminQ if interrupt delivery unavailable

If interrupt registration fails eg. on FreeBSD with nic_uio, the ice interrupt handler is never called and AdminQ messages, including unsolicited link state notifications, are never processed.

This gap has always existed but was partially masked by the blocking link status poll in the ice dev_start function, which would delay some time until the link was up before returning. In this case, the link status was typically correct after dev_start. However after that delay was removed in 2dd7e99855 ("net/ice: revert fix link up when starting device"), the link status was often wrong after dev_start, and there was no mechanism to self-correct.

Rather than restoring the blocking wait removed by that commit which would re-impose a startup delay on every platform, instead activate a periodic alarm that drains the AdminQ on a 50ms interval, replicating the role of the interrupt handler on platforms where interrupt delivery is not supported.

Fixes 2dd7e99855 ("net/ice: revert fix link up when starting device")
---
 drivers/net/intel/ice/ice_ethdev.c | 45 +++++++++++++++++++++++++++++-
 drivers/net/intel/ice/ice_ethdev.h |  1 +
 2 files changed, 45 insertions(+), 1 deletion(-)
drivers/net/intel/ice/ice_ethdev.c +44 -1
index 022d92a2f6..447742ff8d 100644
@@ -3,6 +3,7 @@
  */
 
 #include <rte_string_fns.h>
+#include <rte_alarm.h>
 #include <ethdev_pci.h>
 
 #include <ctype.h>
@@ -108,7 +109,11 @@ enum ice_link_state_on_close {
 
 #define ICE_RSS_LUT_GLOBAL_QUEUE_NB	64
 
+/* Polling interval used when the interrupt path is unavailable */
+#define ICE_AQ_ALARM_INTERVAL 50000 /* us */
+
 static int ice_dev_configure(struct rte_eth_dev *dev);
+static void ice_aq_alarm_handler(void *param);
 static int ice_dev_start(struct rte_eth_dev *dev);
 static int ice_dev_stop(struct rte_eth_dev *dev);
 static int ice_dev_close(struct rte_eth_dev *dev);
@@ -1473,6 +1478,28 @@ ice_handle_aq_msg(struct rte_eth_dev *dev)
 	}
 }
 
+/*
+ * Alarm-based AdminQ polling handler that drains the AdminQ to process
+ * async events, then re-arms itself.
+ *
+ * @param param
+ *  The address of parameter (struct rte_eth_dev *) registered before.
+ *
+ * @return
+ *  void
+ */
+static void
+ice_aq_alarm_handler(void *param)
+{
+	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
+	struct ice_pf *pf = ICE_DEV_PRIVATE_TO_PF(dev->data->dev_private);
+
+	if (!pf->adapter_stopped) {
+		ice_handle_aq_msg(dev);
+		rte_eal_alarm_set(ICE_AQ_ALARM_INTERVAL, ice_aq_alarm_handler, dev);
+	}
+}
+
 /**
  * Interrupt handler triggered by NIC for handling
  * specific interrupt.
@@ -2913,6 +2940,11 @@ ice_dev_stop(struct rte_eth_dev *dev)
 	rte_intr_efd_disable(intr_handle);
 	rte_intr_vec_list_free(intr_handle);
 
+	if (pf->use_aq_polling) {
+		rte_eal_alarm_cancel(ice_aq_alarm_handler, dev);
+		pf->use_aq_polling = false;
+	}
+
 	pf->adapter_stopped = true;
 	dev->data->dev_started = 0;
 
@@ -4434,7 +4466,13 @@ ice_dev_start(struct rte_eth_dev *dev)
 	if (ice_rxq_intr_setup(dev))
 		return -EIO;
 
-	rte_intr_enable(pci_dev->intr_handle);
+	/* Fall back to polling the AdminQ via an alarm on platforms where
+	 * interrupt delivery is unavailable.
+	 */
+	if (rte_intr_enable(pci_dev->intr_handle) != 0) {
+		pf->use_aq_polling = true;
+		rte_eal_alarm_set(ICE_AQ_ALARM_INTERVAL, ice_aq_alarm_handler, dev);
+	}
 
 	/* Enable receiving broadcast packets and transmitting packets */
 	ice_set_bit(ICE_PROMISC_BCAST_RX, pmask);
@@ -4497,6 +4535,11 @@ ice_dev_start(struct rte_eth_dev *dev)
 	for (i = 0; i < nb_txq; i++)
 		ice_tx_queue_stop(dev, i);
 
+	if (pf->use_aq_polling) {
+		rte_eal_alarm_cancel(ice_aq_alarm_handler, dev);
+		pf->use_aq_polling = false;
+	}
+
 	return -EIO;
 }
 
drivers/net/intel/ice/ice_ethdev.h +1 -0
index 20e8a13fe9..86b256fc3f 100644
@@ -599,6 +599,7 @@ struct ice_pf {
 	struct ice_eth_stats internal_stats;
 	bool offset_loaded;
 	bool adapter_stopped;
+	bool use_aq_polling; /* true when interrupt path unavailable */
 	struct ice_flow_list flow_list;
 	rte_spinlock_t flow_ops_lock;
 	bool init_link_up;
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH 1/2] net/ice: poll AdminQ if interrupt delivery unavailable

Quoted 18 lines · Ciara Loftus

If interrupt registration fails eg. on FreeBSD with nic_uio, the ice interrupt handler is never called and AdminQ messages, including unsolicited link state notifications, are never processed.

This gap has always existed but was partially masked by the blocking link status poll in the ice dev_start function, which would delay some time until the link was up before returning. In this case, the link status was typically correct after dev_start. However after that delay was removed in 2dd7e99855 ("net/ice: revert fix link up when starting device"), the link status was often wrong after dev_start, and there was no mechanism to self-correct.

Rather than restoring the blocking wait removed by that commit which would re-impose a startup delay on every platform, instead activate a periodic alarm that drains the AdminQ on a 50ms interval, replicating the role of the interrupt handler on platforms where interrupt delivery is not supported.

Quoted 5 lines
Fixes 2dd7e99855 ("net/ice: revert fix link up when starting device")

---

This solution seems reasonable. Some thoughts and comments inline below.

/Bruce

Quoted 58 lines
 drivers/net/intel/ice/ice_ethdev.c | 45 +++++++++++++++++++++++++++++-
 drivers/net/intel/ice/ice_ethdev.h |  1 +
 2 files changed, 45 insertions(+), 1 deletion(-)
drivers/net/intel/ice/ice_ethdev.c +27 -0
index 022d92a2f6..447742ff8d 100644
@@ -3,6 +3,7 @@
  */
 
 #include <rte_string_fns.h>
+#include <rte_alarm.h>
 #include <ethdev_pci.h>
 
 #include <ctype.h>
@@ -108,7 +109,11 @@ enum ice_link_state_on_close {
 
 #define ICE_RSS_LUT_GLOBAL_QUEUE_NB	64
 
+/* Polling interval used when the interrupt path is unavailable */
+#define ICE_AQ_ALARM_INTERVAL 50000 /* us */
+
 static int ice_dev_configure(struct rte_eth_dev *dev);
+static void ice_aq_alarm_handler(void *param);
 static int ice_dev_start(struct rte_eth_dev *dev);
 static int ice_dev_stop(struct rte_eth_dev *dev);
 static int ice_dev_close(struct rte_eth_dev *dev);
@@ -1473,6 +1478,28 @@ ice_handle_aq_msg(struct rte_eth_dev *dev)
 	}
 }
 
+/*
+ * Alarm-based AdminQ polling handler that drains the AdminQ to process
+ * async events, then re-arms itself.
+ *
+ * @param param
+ *  The address of parameter (struct rte_eth_dev *) registered before.
+ *
+ * @return
+ *  void
+ */
+static void
+ice_aq_alarm_handler(void *param)
+{
+	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
+	struct ice_pf *pf = ICE_DEV_PRIVATE_TO_PF(dev->data->dev_private);
+
+	if (!pf->adapter_stopped) {
+		ice_handle_aq_msg(dev);
+		rte_eal_alarm_set(ICE_AQ_ALARM_INTERVAL, ice_aq_alarm_handler, dev);
+	}
+}
+
 /**
  * Interrupt handler triggered by NIC for handling
  * specific interrupt.

drivers/net/intel/ice/ice_ethdev.c ice_dev_stop(struct rte_eth_dev *dev) · line 2947

Quoted 8 lines
drivers/net/intel/ice/ice_ethdev.c +5 -0
@@ -2913,6 +2940,11 @@ ice_dev_stop(struct rte_eth_dev *dev)
 	rte_intr_efd_disable(intr_handle);
 	rte_intr_vec_list_free(intr_handle);
 
+	if (pf->use_aq_polling) {
+		rte_eal_alarm_cancel(ice_aq_alarm_handler, dev);
+		pf->use_aq_polling = false;
+	}
+

Do we actually need to cancel the timer here? Since the callback itself only rearms on started devices, we can let the timer expire and become a no-op, rather than needing to explicitly cancel it here.

If so, and based on other suggestions I have below, we may be able to remove this block from stop function entirely.

Quoted 5 lines
drivers/net/intel/ice/ice_ethdev.c +0 -0
 	pf->adapter_stopped = true;
 	dev->data->dev_started = 0;
 
@@ -4434,7 +4466,13 @@ ice_dev_start(struct rte_eth_dev *dev)
 	if (ice_rxq_intr_setup(dev))

drivers/net/intel/ice/ice_ethdev.c ice_dev_start(struct rte_eth_dev *dev) · line 4475

Quoted 10 lines
drivers/net/intel/ice/ice_ethdev.c +7 -1
 		return -EIO;
 
-	rte_intr_enable(pci_dev->intr_handle);
+	/* Fall back to polling the AdminQ via an alarm on platforms where
+	 * interrupt delivery is unavailable.
+	 */
+	if (rte_intr_enable(pci_dev->intr_handle) != 0) {
+		pf->use_aq_polling = true;
+		rte_eal_alarm_set(ICE_AQ_ALARM_INTERVAL, ice_aq_alarm_handler, dev);
+	}

If alarm set fails, we should at least emit a warning, even if there is nothing we can really do about it.

Also, I don't think we need to track use_aq_polling at all. Its only use is to cancel the timer, but all places where we cancel the timer are places where we mark the port as stopped (with one exception, which I believe is a separate error in the code), so the timer will self-cancel at next timeout expiry.

drivers/net/intel/ice/ice_ethdev.c ice_dev_start(struct rte_eth_dev *dev) · line 4542

Quoted 12 lines
drivers/net/intel/ice/ice_ethdev.c +5 -0
 	/* Enable receiving broadcast packets and transmitting packets */
 	ice_set_bit(ICE_PROMISC_BCAST_RX, pmask);
@@ -4497,6 +4535,11 @@ ice_dev_start(struct rte_eth_dev *dev)
 	for (i = 0; i < nb_txq; i++)
 		ice_tx_queue_stop(dev, i);
 
+	if (pf->use_aq_polling) {
+		rte_eal_alarm_cancel(ice_aq_alarm_handler, dev);
+		pf->use_aq_polling = false;
+	}
+

As explained above, I don't think we need this block either. If we have an error on start, then the port should be marked as stopped and the timer won't rearm itself after it expires.

NOTE: there is an unrelated bug in the code here - at line 4550 in start, we set "adapter_stopped = false", but there is still error handling at line 4564 which causes us to jump to rx_err and stop all the rx and tx queues again, but neglecting to reset adapter_stopped back to true. I think we need to either reset it immediately after the rx_err label, or move the assignment down till after all potential rollback points are passed.

Quoted 18 lines
drivers/net/intel/ice/ice_ethdev.c +0 -0
 	return -EIO;
 }
 
drivers/net/intel/ice/ice_ethdev.h +1 -0
index 20e8a13fe9..86b256fc3f 100644
@@ -599,6 +599,7 @@ struct ice_pf {
 	struct ice_eth_stats internal_stats;
 	bool offset_loaded;
 	bool adapter_stopped;
+	bool use_aq_polling; /* true when interrupt path unavailable */
 	struct ice_flow_list flow_list;
 	rte_spinlock_t flow_ops_lock;
 	bool init_link_up;
Signature · 1 line
2.43.0
#3 lore raw

RE: [PATCH 1/2] net/ice: poll AdminQ if interrupt delivery unavailable

Quoted 95 lines

Subject: Re: [PATCH 1/2] net/ice: poll AdminQ if interrupt delivery unavailable

Quoted 18 lines · Ciara Loftus

If interrupt registration fails eg. on FreeBSD with nic_uio, the ice interrupt handler is never called and AdminQ messages, including unsolicited link state notifications, are never processed.

This gap has always existed but was partially masked by the blocking link status poll in the ice dev_start function, which would delay some time until the link was up before returning. In this case, the link status was typically correct after dev_start. However after that delay was removed in 2dd7e99855 ("net/ice: revert fix link up when starting device"), the link status was often wrong after dev_start, and there was no mechanism to self-correct.

Rather than restoring the blocking wait removed by that commit which would re-impose a startup delay on every platform, instead activate a periodic alarm that drains the AdminQ on a 50ms interval, replicating the role of the interrupt handler on platforms where interrupt delivery is not supported.

Quoted 5 lines
Fixes 2dd7e99855 ("net/ice: revert fix link up when starting device")

---

This solution seems reasonable. Some thoughts and comments inline below.

/Bruce

Quoted 1 line
 drivers/net/intel/ice/ice_ethdev.c | 45

+++++++++++++++++++++++++++++-

Review

Quoted 4 lines
 drivers/net/intel/ice/ice_ethdev.h |  1 +
 2 files changed, 45 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/drivers/net/intel/ice/ice_ethdev.c

Quoted 36 lines

index 022d92a2f6..447742ff8d 100644

drivers/net/intel/ice/ice_ethdev.c +14 -0
@@ -3,6 +3,7 @@
  */
 
 #include <rte_string_fns.h>
+#include <rte_alarm.h>
 #include <ethdev_pci.h>
 
 #include <ctype.h>
@@ -108,7 +109,11 @@ enum ice_link_state_on_close {
 
 #define ICE_RSS_LUT_GLOBAL_QUEUE_NB	64
 
+/* Polling interval used when the interrupt path is unavailable */
+#define ICE_AQ_ALARM_INTERVAL 50000 /* us */
+
 static int ice_dev_configure(struct rte_eth_dev *dev);
+static void ice_aq_alarm_handler(void *param);
 static int ice_dev_start(struct rte_eth_dev *dev);
 static int ice_dev_stop(struct rte_eth_dev *dev);
 static int ice_dev_close(struct rte_eth_dev *dev);
@@ -1473,6 +1478,28 @@ ice_handle_aq_msg(struct rte_eth_dev *dev)
 	}
 }
 
+/*
+ * Alarm-based AdminQ polling handler that drains the AdminQ to process
+ * async events, then re-arms itself.
+ *
+ * @param param
+ *  The address of parameter (struct rte_eth_dev *) registered before.
+ *
+ * @return
+ *  void

drivers/net/intel/ice/ice_ethdev.c ice_handle_aq_msg(struct rte_eth_dev *dev) · line 1499

Quoted 10 lines
drivers/net/intel/ice/ice_ethdev.c +10 -0
+ */
+static void
+ice_aq_alarm_handler(void *param)
+{
+	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
+	struct ice_pf *pf = ICE_DEV_PRIVATE_TO_PF(dev->data->dev_private);
+
+	if (!pf->adapter_stopped) {
+		ice_handle_aq_msg(dev);
+		rte_eal_alarm_set(ICE_AQ_ALARM_INTERVAL,

ice_aq_alarm_handler, dev);

Quoted 10 lines
drivers/net/intel/ice/ice_ethdev.c +3 -0
+	}
+}
+
 /**
  * Interrupt handler triggered by NIC for handling
  * specific interrupt.
@@ -2913,6 +2940,11 @@ ice_dev_stop(struct rte_eth_dev *dev)
 	rte_intr_efd_disable(intr_handle);
 	rte_intr_vec_list_free(intr_handle);
 
Quoted 9 lines

drivers/net/intel/ice/ice_ethdev.c ice_dev_stop(struct rte_eth_dev *dev) · line 2947

Quoted 5 lines
drivers/net/intel/ice/ice_ethdev.c +5 -0
+	if (pf->use_aq_polling) {
+		rte_eal_alarm_cancel(ice_aq_alarm_handler, dev);
+		pf->use_aq_polling = false;
+	}
+

Do we actually need to cancel the timer here? Since the callback itself only rearms on started devices, we can let the timer expire and become a no-op, rather than needing to explicitly cancel it here.

If we don't cancel the timer, it may fire after device memory is freed on exit. When dev->data->dev_private is accessed in the handler, it will SEGV. So we need the cancel in dev_stop.

Quoted 19 lines

If so, and based on other suggestions I have below, we may be able to remove this block from stop function entirely.

Quoted 4 lines
drivers/net/intel/ice/ice_ethdev.c +0 -0
 	pf->adapter_stopped = true;
 	dev->data->dev_started = 0;
 

drivers/net/intel/ice/ice_ethdev.c ice_dev_start(struct rte_eth_dev *dev) · line 4474

Quoted 10 lines
drivers/net/intel/ice/ice_ethdev.c +6 -1
@@ -4434,7 +4466,13 @@ ice_dev_start(struct rte_eth_dev *dev)
 	if (ice_rxq_intr_setup(dev))
 		return -EIO;
 
-	rte_intr_enable(pci_dev->intr_handle);
+	/* Fall back to polling the AdminQ via an alarm on platforms where
+	 * interrupt delivery is unavailable.
+	 */
+	if (rte_intr_enable(pci_dev->intr_handle) != 0) {
+		pf->use_aq_polling = true;
+		rte_eal_alarm_set(ICE_AQ_ALARM_INTERVAL,

ice_aq_alarm_handler, dev);

Quoted 4 lines

drivers/net/intel/ice/ice_ethdev.c ice_dev_start(struct rte_eth_dev *dev) · line 4475

Quoted 1 line
drivers/net/intel/ice/ice_ethdev.c +1 -0
+	}

If alarm set fails, we should at least emit a warning, even if there is nothing we can really do about it.

Agree, will implement.

Quoted 6 lines

Also, I don't think we need to track use_aq_polling at all. Its only use is to cancel the timer, but all places where we cancel the timer are places where we mark the port as stopped (with one exception, which I believe is a separate error in the code), so the timer will self-cancel at next timeout expiry.

As explained above I believe we need to keep the cancel in ice_dev_stop so I will keep use_aq_polling to guard that cancel. I think it would probably be fine to unconditionally cancel but I think it is cleaner to only cancel if we actually set the alarm.

Quoted 8 lines
Quoted 8 lines
drivers/net/intel/ice/ice_ethdev.c +0 -0
 	/* Enable receiving broadcast packets and transmitting packets */
 	ice_set_bit(ICE_PROMISC_BCAST_RX, pmask);
@@ -4497,6 +4535,11 @@ ice_dev_start(struct rte_eth_dev *dev)
 	for (i = 0; i < nb_txq; i++)
 		ice_tx_queue_stop(dev, i);
 
Quoted 9 lines

drivers/net/intel/ice/ice_ethdev.c ice_dev_stop(struct rte_eth_dev *dev) · line 2947

Quoted 5 lines
drivers/net/intel/ice/ice_ethdev.c +5 -0
+	if (pf->use_aq_polling) {
+		rte_eal_alarm_cancel(ice_aq_alarm_handler, dev);
+		pf->use_aq_polling = false;
+	}
+

As explained above, I don't think we need this block either. If we have an error on start, then the port should be marked as stopped and the timer won't rearm itself after it expires.

Will remove.

Quoted 7 lines

NOTE: there is an unrelated bug in the code here - at line 4550 in start, we set "adapter_stopped = false", but there is still error handling at line 4564 which causes us to jump to rx_err and stop all the rx and tx queues again, but neglecting to reset adapter_stopped back to true. I think we need to either reset it immediately after the rx_err label, or move the assignment down till after all potential rollback points are passed.

Will fix in a separate patch.

Quoted 20 lines

Review

Quoted 5 lines
drivers/net/intel/ice/ice_ethdev.c +0 -0
 	return -EIO;
 }
 
(quoted diff) +0 -0

b/drivers/net/intel/ice/ice_ethdev.h

Quoted 14 lines

index 20e8a13fe9..86b256fc3f 100644

drivers/net/intel/ice/ice_ethdev.h +1 -1
@@ -599,6 +599,7 @@ struct ice_pf {
 	struct ice_eth_stats internal_stats;
 	bool offset_loaded;
 	bool adapter_stopped;
+	bool use_aq_polling; /* true when interrupt path unavailable */
 	struct ice_flow_list flow_list;
 	rte_spinlock_t flow_ops_lock;
 	bool init_link_up;
--

2.43.0

#4 lore raw

PATCH 2/2 net/i40e: activate alarm if interrupt delivery unavailable

In reply to Ciara Loftus

If interrupt registration fails eg. on FreeBSD with nic_uio, and Rx queue interrupts are configured (intr_conf.rxq != 0), the interrupt handler is never called and ICR0 events, including AdminQ messages carrying link state notifications, are never processed. In polling mode (rxq == 0) a periodic alarm already drains these events, but in the interrupt-enabled configuration no such fallback existed.

This gap has always existed but was partially masked by the blocking link status poll in the i40e dev_start function, which would delay some time until the link was up before returning. In this case, the link status was typically correct after dev_start. However after that delay was removed in 111965395b ("net/i40e: fix blocking link wait on device start"), the link status was often wrong after dev_start, and there was no mechanism to self-correct.

Rather than restoring the blocking wait removed by that commit which would re-impose a startup delay on every platform, activate the existing periodic alarm on platforms where interrupt delivery is unavailable.

Fixes 111965395b ("net/i40e: fix blocking link wait on device start")
---
 drivers/net/intel/i40e/i40e_ethdev.c | 9 ++++++++-
 drivers/net/intel/i40e/i40e_ethdev.h | 2 ++
 2 files changed, 10 insertions(+), 1 deletion(-)
drivers/net/intel/i40e/i40e_ethdev.c +8 -1
index b2694cd33a..86eb7468c8 100644
@@ -2579,7 +2579,11 @@ i40e_dev_start(struct rte_eth_dev *dev)
 		i40e_dev_link_update(dev, !rte_intr_allow_others(intr_handle));
 		pf->mac_config_on_link_up = !dev->data->dev_link.link_status;
 		/* enable uio intr after callback register */
-		rte_intr_enable(intr_handle);
+		if (rte_intr_enable(intr_handle) != 0) {
+			pf->use_aq_polling = true;
+			rte_eal_alarm_set(I40E_ALARM_INTERVAL,
+					  i40e_dev_alarm_handler, dev);
+		}
 	}
 
 	i40e_filter_restore(pf);
@@ -2625,6 +2629,9 @@ i40e_dev_stop(struct rte_eth_dev *dev)
 	if (dev->data->dev_conf.intr_conf.rxq == 0) {
 		rte_eal_alarm_cancel(i40e_dev_alarm_handler, dev);
 		rte_intr_enable(intr_handle);
+	} else if (pf->use_aq_polling) {
+		rte_eal_alarm_cancel(i40e_dev_alarm_handler, dev);
+		pf->use_aq_polling = false;
 	}
 
 	/* Disable all queues */
drivers/net/intel/i40e/i40e_ethdev.h +2 -0
index 5a009393b0..9d9bde6aeb 100644
@@ -1193,6 +1193,8 @@ struct i40e_pf {
 	bool fw8_3gt;
 	/* MAC config needs re-applying when link first comes up */
 	bool mac_config_on_link_up;
+	/* true when interrupt path unavailable */
+	bool use_aq_polling;
 
 	struct i40e_vf_msg_cfg vf_msg_cfg;
 	uint64_t prev_rx_bytes;
Signature · 1 line
2.43.0
#5 lore raw

v2 cover net/intel: fix link status when interrupt delivery unavailable

In reply to Ciara Loftus

On FreeBSD with nic_uio, interrupt registration fails silently and hardware events, including link state notifications, are never processed via the interrupt path.

For ice, this affects the typical polling-mode configuration: the interrupt handler is never invoked, AdminQ messages go unprocessed, and link status is not reliable. A periodic alarm is added to drain the AdminQ when interrupt delivery is unavailable. For i40e, an alarm already runs unconditionally in polling mode, so that configuration is unaffected. The gap is the interrupt-mode configuration, where the existing alarm was not activated as a fallback when interrupt enable fails; this patch extends it to do so.

v2:
* Added warning on rte_eal_alarm_set failure (i40e & ice)
* Only set use_aq_polling if rte_eal_alarm_set succeeds (i40e & ice)
* Move alarm cancel in ice_dev_stop to prevent a race (ice)
* Remove unnecessary alarm cancel from ice_dev_start error path (ice)
* Added general fix for ice dev_start error path

Ciara Loftus (3):
net/ice: poll AdminQ if interrupt delivery unavailable net/i40e: activate alarm if interrupt delivery unavailable net/ice: fix adapter stopped on device start error

 drivers/net/intel/i40e/i40e_ethdev.c | 11 ++++++-
 drivers/net/intel/i40e/i40e_ethdev.h |  2 ++
 drivers/net/intel/ice/ice_ethdev.c   | 44 +++++++++++++++++++++++++++-
 drivers/net/intel/ice/ice_ethdev.h   |  1 +
 4 files changed, 56 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0
#6 lore raw

PATCH v2 1/3 net/ice: poll AdminQ if interrupt delivery unavailable

If interrupt registration fails eg. on FreeBSD with nic_uio, the ice interrupt handler is never called and AdminQ messages, including unsolicited link state notifications, are never processed.

This gap has always existed but was partially masked by the blocking link status poll in the ice dev_start function, which would delay some time until the link was up before returning. In this case, the link status was typically correct after dev_start. However after that delay was removed in commit 2dd7e998550e ("net/ice: revert fix link up when starting device"), the link status was often wrong after dev_start, and there was no mechanism to self-correct.

Rather than restoring the blocking wait removed by that commit which would re-impose a startup delay on every platform, instead activate a periodic alarm that drains the AdminQ on a 50ms interval, replicating the role of the interrupt handler on platforms where interrupt delivery is not supported.

Fixes 2dd7e998550e ("net/ice: revert fix link up when starting device")

---
* Added warning on rte_eal_alarm_set failure
* Only set use_aq_polling if rte_eal_alarm_set succeeds
* Move alarm cancel in ice_dev_stop to prevent a race
* Remove unnecessary alarm cancel from ice_dev_start error path

---
 drivers/net/intel/ice/ice_ethdev.c | 42 +++++++++++++++++++++++++++++-
 drivers/net/intel/ice/ice_ethdev.h |  1 +
 2 files changed, 42 insertions(+), 1 deletion(-)
drivers/net/intel/ice/ice_ethdev.c +41 -1
index 4be5b63c9a..8db5168b88 100644
@@ -3,6 +3,7 @@
  */
 
 #include <rte_string_fns.h>
+#include <rte_alarm.h>
 #include <ethdev_pci.h>
 
 #include <ctype.h>
@@ -110,7 +111,11 @@ enum ice_link_state_on_close {
 
 #define ICE_RSS_LUT_GLOBAL_QUEUE_NB	64
 
+/* Polling interval used when the interrupt path is unavailable */
+#define ICE_AQ_ALARM_INTERVAL 50000 /* us */
+
 static int ice_dev_configure(struct rte_eth_dev *dev);
+static void ice_aq_alarm_handler(void *param);
 static int ice_dev_start(struct rte_eth_dev *dev);
 static int ice_dev_stop(struct rte_eth_dev *dev);
 static int ice_dev_close(struct rte_eth_dev *dev);
@@ -1475,6 +1480,28 @@ ice_handle_aq_msg(struct rte_eth_dev *dev)
 	}
 }
 
+/*
+ * Alarm-based AdminQ polling handler that drains the AdminQ to process
+ * async events, then re-arms itself.
+ *
+ * @param param
+ *  The address of parameter (struct rte_eth_dev *) registered before.
+ *
+ * @return
+ *  void
+ */
+static void
+ice_aq_alarm_handler(void *param)
+{
+	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
+	struct ice_pf *pf = ICE_DEV_PRIVATE_TO_PF(dev->data->dev_private);
+
+	if (!pf->adapter_stopped) {
+		ice_handle_aq_msg(dev);
+		rte_eal_alarm_set(ICE_AQ_ALARM_INTERVAL, ice_aq_alarm_handler, dev);
+	}
+}
+
 /**
  * Interrupt handler triggered by NIC for handling
  * specific interrupt.
@@ -2961,6 +2988,11 @@ ice_dev_stop(struct rte_eth_dev *dev)
 	pf->adapter_stopped = true;
 	dev->data->dev_started = 0;
 
+	if (pf->use_aq_polling) {
+		rte_eal_alarm_cancel(ice_aq_alarm_handler, dev);
+		pf->use_aq_polling = false;
+	}
+
 	return 0;
 }
 
@@ -4479,7 +4511,15 @@ ice_dev_start(struct rte_eth_dev *dev)
 	if (ice_rxq_intr_setup(dev))
 		return -EIO;
 
-	rte_intr_enable(pci_dev->intr_handle);
+	/* Fall back to polling the AdminQ via an alarm on platforms where
+	 * interrupt delivery is unavailable.
+	 */
+	if (rte_intr_enable(pci_dev->intr_handle) != 0) {
+		if (rte_eal_alarm_set(ICE_AQ_ALARM_INTERVAL, ice_aq_alarm_handler, dev) != 0)
+			PMD_DRV_LOG(WARNING, "Failed to set alarm for AdminQ polling");
+		else
+			pf->use_aq_polling = true;
+	}
 
 	/* Enable receiving broadcast packets and transmitting packets */
 	ice_set_bit(ICE_PROMISC_BCAST_RX, pmask);
drivers/net/intel/ice/ice_ethdev.h +1 -0
index 0a9d75b9cd..7ee3ea8a70 100644
@@ -599,6 +599,7 @@ struct ice_pf {
 	struct ice_eth_stats internal_stats;
 	bool offset_loaded;
 	bool adapter_stopped;
+	bool use_aq_polling; /* true when alarm polls AdminQ in place of interrupt */
 	struct ice_flow_list flow_list;
 	rte_spinlock_t flow_ops_lock;
 	bool init_link_up;
Signature · 1 line
2.43.0
#7 lore raw

PATCH v2 2/3 net/i40e: activate alarm if interrupt delivery unavailable

In reply to Ciara Loftus

If interrupt registration fails eg. on FreeBSD with nic_uio, and Rx queue interrupts are configured (intr_conf.rxq != 0), the interrupt handler is never called and ICR0 events, including AdminQ messages carrying link state notifications, are never processed. In polling mode (rxq == 0) a periodic alarm already drains these events, but in the interrupt-enabled configuration no such fallback existed.

This gap has always existed but was partially masked by the blocking link status poll in the i40e dev_start function, which would delay some time until the link was up before returning. In this case, the link status was typically correct after dev_start. However after that delay was removed in commit 111965395b37 ("net/i40e: fix blocking link wait on device start"), the link status was often wrong after dev_start, and there was no mechanism to self-correct.

Rather than restoring the blocking wait removed by that commit which would re-impose a startup delay on every platform, activate the existing periodic alarm on platforms where interrupt delivery is unavailable.

Fixes 111965395b37 ("net/i40e: fix blocking link wait on device start")

---
* Added warning on rte_eal_alarm_set failure
* Only set use_aq_polling if rte_eal_alarm_set succeeds

---
 drivers/net/intel/i40e/i40e_ethdev.c | 11 ++++++++++-
 drivers/net/intel/i40e/i40e_ethdev.h |  2 ++
 2 files changed, 12 insertions(+), 1 deletion(-)
drivers/net/intel/i40e/i40e_ethdev.c +10 -1
index b2694cd33a..b6b2d291ee 100644
@@ -2579,7 +2579,13 @@ i40e_dev_start(struct rte_eth_dev *dev)
 		i40e_dev_link_update(dev, !rte_intr_allow_others(intr_handle));
 		pf->mac_config_on_link_up = !dev->data->dev_link.link_status;
 		/* enable uio intr after callback register */
-		rte_intr_enable(intr_handle);
+		if (rte_intr_enable(intr_handle) != 0) {
+			if (rte_eal_alarm_set(I40E_ALARM_INTERVAL,
+					  i40e_dev_alarm_handler, dev) != 0)
+				PMD_DRV_LOG(WARNING, "Failed to set alarm");
+			else
+				pf->use_aq_polling = true;
+		}
 	}
 
 	i40e_filter_restore(pf);
@@ -2625,6 +2631,9 @@ i40e_dev_stop(struct rte_eth_dev *dev)
 	if (dev->data->dev_conf.intr_conf.rxq == 0) {
 		rte_eal_alarm_cancel(i40e_dev_alarm_handler, dev);
 		rte_intr_enable(intr_handle);
+	} else if (pf->use_aq_polling) {
+		rte_eal_alarm_cancel(i40e_dev_alarm_handler, dev);
+		pf->use_aq_polling = false;
 	}
 
 	/* Disable all queues */
drivers/net/intel/i40e/i40e_ethdev.h +2 -0
index 5a009393b0..9d9bde6aeb 100644
@@ -1193,6 +1193,8 @@ struct i40e_pf {
 	bool fw8_3gt;
 	/* MAC config needs re-applying when link first comes up */
 	bool mac_config_on_link_up;
+	/* true when interrupt path unavailable */
+	bool use_aq_polling;
 
 	struct i40e_vf_msg_cfg vf_msg_cfg;
 	uint64_t prev_rx_bytes;
Signature · 1 line
2.43.0
#8 lore raw

PATCH v2 3/3 net/ice: fix adapter stopped on device start error

In reply to Ciara Loftus

Currently, adapter_stopped is cleared during device startup but is not restored to true if startup fails partway through. Reset adapter_stopped to true on the device startup error path to accurately reflect device state after a failed start.

Fixes 437dbd2fd428 ("net/ice: support 1PPS")
---
 drivers/net/intel/ice/ice_ethdev.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/intel/ice/ice_ethdev.c +2 -0
index 8db5168b88..76b8ff0a72 100644
@@ -4582,6 +4582,8 @@ ice_dev_start(struct rte_eth_dev *dev)
 	for (i = 0; i < nb_txq; i++)
 		ice_tx_queue_stop(dev, i);
 
+	pf->adapter_stopped = true;
+
 	return -EIO;
 }
 
Signature · 1 line
2.43.0
#9 lore raw
In reply to Ciara Loftus
Quoted 20 lines · Ciara Loftus

On FreeBSD with nic_uio, interrupt registration fails silently and hardware events, including link state notifications, are never processed via the interrupt path.

For ice, this affects the typical polling-mode configuration: the interrupt handler is never invoked, AdminQ messages go unprocessed, and link status is not reliable. A periodic alarm is added to drain the AdminQ when interrupt delivery is unavailable. For i40e, an alarm already runs unconditionally in polling mode, so that configuration is unaffected. The gap is the interrupt-mode configuration, where the existing alarm was not activated as a fallback when interrupt enable fails; this patch extends it to do so.

v2:
* Added warning on rte_eal_alarm_set failure (i40e & ice)
* Only set use_aq_polling if rte_eal_alarm_set succeeds (i40e & ice)
* Move alarm cancel in ice_dev_stop to prevent a race (ice)
* Remove unnecessary alarm cancel from ice_dev_start error path (ice)
* Added general fix for ice dev_start error path

Quoted 5 lines

Ciara Loftus (3):
net/ice: poll AdminQ if interrupt delivery unavailable net/i40e: activate alarm if interrupt delivery unavailable net/ice: fix adapter stopped on device start error

Applied to next-net-intel.

Thanks,
/Bruce

Quoted 9 lines
 drivers/net/intel/i40e/i40e_ethdev.c | 11 ++++++-
 drivers/net/intel/i40e/i40e_ethdev.h |  2 ++
 drivers/net/intel/ice/ice_ethdev.c   | 44 +++++++++++++++++++++++++++-
 drivers/net/intel/ice/ice_ethdev.h   |  1 +
 4 files changed, 56 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0