Skip to contentSkip to conversation

net/ixgbe: fix link state timing issue on fiber ports

PATCH Superseded Phil Yang 3 participants 7 messages Mar 19, 2020 – May 11, 2020 Network drivers

lore
Share & tools
lore

Phil Yang

author
#0 lore raw

PATCH 1/1 net/ixgbe: fix link state timing issue on fiber ports

With some models of fiber ports (e.g. X520-2 device ID 0x10fb), it is possible when a port is started to experience a timing issue which prevents the link from ever being fully set up.

In ixgbe_dev_link_update_share(), if the media type is fiber and the link is down, a flag (IXGBE_FLAG_NEED_LINK_CONFIG) is set. A callback to ixgbe_dev_setup_link_thread_handler() is scheduled which should try to set up the link and clear the flag afterwards.

If the device is started before the flag is cleared, the scheduled callback is cancelled. This causes the flag to remain set and subsequent calls to ixgbe_dev_link_update_share() return without trying to retrieve the link state because the flag is set.

In ixgbe_dev_cancel_link_thread(), after cancelling the callback, unset the flag on the device to avoid this condition.

Fixes 819d0d1d57f1 ("net/ixgbe: fix blocking system events")

Bugzilla ID: 388

---
 drivers/net/ixgbe/ixgbe_ethdev.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
drivers/net/ixgbe/ixgbe_ethdev.c +13 -1
index 23b3f5b..2b65750 100644
@@ -4147,11 +4147,19 @@ static void
 ixgbe_dev_cancel_link_thread(struct rte_eth_dev *dev)
 {
 	struct ixgbe_adapter *ad = dev->data->dev_private;
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	void *retval;
 
 	if (rte_atomic32_read(&ad->link_thread_running)) {
 		pthread_cancel(ad->link_thread_tid);
 		pthread_join(ad->link_thread_tid, &retval);
+		/* clear this flag once the thread has been
+		 * cancelled, to avoid link status error in
+		 * case unfinished threads cannot clean up
+		 * this flag.
+		 */
+		intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
 		rte_atomic32_clear(&ad->link_thread_running);
 	}
 }
@@ -4262,8 +4270,12 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 
 	if (link_up == 0) {
 		if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
-			intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
 			if (rte_atomic32_test_and_set(&ad->link_thread_running)) {
+				/* To avoid race condition between threads, set
+				 * the IXGBE_FLAG_NEED_LINK_CONFIG flag only
+				 * when there is no link thread running.
+				 */
+				intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
 				if (rte_ctrl_thread_create(&ad->link_thread_tid,
 					"ixgbe-link-handler",
 					NULL,
Signature · 1 line
2.7.4
#1 lore raw

This issue is firstly observed with an ixgbe NIC in VPP project, which is software switching application based on DPDK. There's a daemon thread running in background keeping polling hardware link status, using ixgbe_dev_link_update_share().
Once flag IXGBE_FLAG_NEED_LINK_CONFIG is set, ixgbe_dev_link_update_share() will just return link down status without actually polling hardware status.

In the issue, flag IXGBE_FLAG_NEED_LINK_CONFIG is always set, and never be cleared, meaning ixgbe_dev_link_update_share() cannot get hardware status, but always get link down status.

The condition causing IXGBE_FLAG_NEED_LINK_CONFIG always set is as below.

The ixgbe_dev_link_update_share() is always running in the background.
1. In the beginning, IXGBE_FLAG_NEED_LINK_CONFIG is 0 and it is link down status.
2. ixgbe_dev_link_update_share() will set IXGBE_FLAG_NEED_LINK_CONFIG to 1
3. Then it triggers ixgbe_dev_setup_link_thread_handler() thread to configure the interface.
4. At the end of configuring thread, ixgbe_dev_setup_link_thread_handler() will clear the flag IXGBE_FLAG_NEED_LINK_CONFIG.
5. With IXGBE_FLAG_NEED_LINK_CONFIG being cleared, ixgbe_dev_link_update_share() can poll hardware link status in the next round.

But when the user is setting interface link up or down in the CLI, it will call ixgbe_dev_start() or ixgbe_dev_stop(). In both function, they will call ixgbe_dev_cancel_link_thread() to interrupt any running configuring thread (which is running in above step 3 and step 4), without clearing the flag IXGBE_FLAG_NEED_LINK_CONFIG. This will leave IXGBE_FLAG_NEED_LINK_CONFIG always set, and ixgbe_dev_link_update_share() cannot get hardware status. Thanks.

Quoted 84 lines

-----Original Message-----
From: Phil Yang <[email protected]>
Sent: 2020年3月19日 14:42
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; Lijian Zhang <[email protected]>; Gavin Hu <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>; [email protected]
Subject: [PATCH] net/ixgbe: fix link state timing issue on fiber ports

With some models of fiber ports (e.g. X520-2 device ID 0x10fb), it is possible when a port is started to experience a timing issue which prevents the link from ever being fully set up.

In ixgbe_dev_link_update_share(), if the media type is fiber and the link is down, a flag (IXGBE_FLAG_NEED_LINK_CONFIG) is set. A callback to ixgbe_dev_setup_link_thread_handler() is scheduled which should try to set up the link and clear the flag afterwards.

If the device is started before the flag is cleared, the scheduled callback is cancelled. This causes the flag to remain set and subsequent calls to ixgbe_dev_link_update_share() return without trying to retrieve the link state because the flag is set.

In ixgbe_dev_cancel_link_thread(), after cancelling the callback, unset the flag on the device to avoid this condition.

Fixes 819d0d1d57f1 ("net/ixgbe: fix blocking system events")

Bugzilla ID: 388

---
 drivers/net/ixgbe/ixgbe_ethdev.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
drivers/net/ixgbe/ixgbe_ethdev.c +9 -1
index 23b3f5b..2b65750 100644
@@ -4147,11 +4147,19 @@ static void
 ixgbe_dev_cancel_link_thread(struct rte_eth_dev *dev)  {
 	struct ixgbe_adapter *ad = dev->data->dev_private;
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	void *retval;
 
 	if (rte_atomic32_read(&ad->link_thread_running)) {
 		pthread_cancel(ad->link_thread_tid);
 		pthread_join(ad->link_thread_tid, &retval);
+		/* clear this flag once the thread has been
+		 * cancelled, to avoid link status error in
+		 * case unfinished threads cannot clean up
+		 * this flag.
+		 */
+		intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
 		rte_atomic32_clear(&ad->link_thread_running);
 	}
 }
@@ -4262,8 +4270,12 @@ ixgbe_dev_link_update_share(struct rte_eth_dev
 *dev,
 
 	if (link_up == 0) {
 		if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
-			intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
 			if (rte_atomic32_test_and_set(&ad-
 >link_thread_running)) {
+				/* To avoid race condition between threads,

set

drivers/net/ixgbe/ixgbe_ethdev.c +1 -0
+				 * the IXGBE_FLAG_NEED_LINK_CONFIG flag

only

drivers/net/ixgbe/ixgbe_ethdev.c +3 -0
+				 * when there is no link thread running.
+				 */
+				intr->flags |=

IXGBE_FLAG_NEED_LINK_CONFIG;
if (rte_ctrl_thread_create(&ad-

Quoted 1 line

link_thread_tid,

drivers/net/ixgbe/ixgbe_ethdev.c +0 -1
 					"ixgbe-link-handler",
 					NULL,
--

2.7.4

author
#2 lore raw
In reply to Phil Yang
Quoted 23 lines

Subject: [dpdk-dev] [PATCH] net/ixgbe: fix link state timing issue on fiber ports

With some models of fiber ports (e.g. X520-2 device ID 0x10fb), it is possible when a port is started to experience a timing issue which prevents the link from ever being fully set up.

In ixgbe_dev_link_update_share(), if the media type is fiber and the link is down, a flag (IXGBE_FLAG_NEED_LINK_CONFIG) is set. A callback to ixgbe_dev_setup_link_thread_handler() is scheduled which should try to set up the link and clear the flag afterwards.

If the device is started before the flag is cleared, the scheduled callback is cancelled. This causes the flag to remain set and subsequent calls to ixgbe_dev_link_update_share() return without trying to retrieve the link state because the flag is set.

In ixgbe_dev_cancel_link_thread(), after cancelling the callback, unset the flag on the device to avoid this condition.

Fixes 819d0d1d57f1 ("net/ixgbe: fix blocking system events")
Quoted 6 lines

Bugzilla ID: 388

---

Ping.

Thanks,
Phil

<Snip>

#3 lore raw
Quoted 23 lines · Phil Yang
Quoted 23 lines

Subject: [dpdk-dev] [PATCH] net/ixgbe: fix link state timing issue on fiber ports

With some models of fiber ports (e.g. X520-2 device ID 0x10fb), it is possible when a port is started to experience a timing issue which prevents the link from ever being fully set up.

In ixgbe_dev_link_update_share(), if the media type is fiber and the link is down, a flag (IXGBE_FLAG_NEED_LINK_CONFIG) is set. A callback to ixgbe_dev_setup_link_thread_handler() is scheduled which should try to set up the link and clear the flag afterwards.

If the device is started before the flag is cleared, the scheduled callback is cancelled. This causes the flag to remain set and subsequent calls to ixgbe_dev_link_update_share() return without trying to retrieve the link state because the flag is set.

In ixgbe_dev_cancel_link_thread(), after cancelling the callback, unset the flag on the device to avoid this condition.

Fixes 819d0d1d57f1 ("net/ixgbe: fix blocking system events")
Quoted 8 lines
Quoted 6 lines

Bugzilla ID: 388

---

Ping.

This fix makes sense to me, thanks for the work.
And it seems can't be applied to latest dpdk-next-net-intel cleanly, could you do a rebase?

Thanks,
Xiaolong

Quoted 5 lines

Thanks,
Phil

<Snip>

author
#4 lore raw
Quoted 42 lines

-----Original Message-----
From: Ye Xiaolong <[email protected]>
Sent: Friday, May 8, 2020 4:36 PM
To: Phil Yang <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Lijian Zhang <[email protected]>; Gavin Hu <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH] net/ixgbe: fix link state timing issue on fiber ports

Quoted 30 lines · Phil Yang
Quoted 29 lines

Subject: [dpdk-dev] [PATCH] net/ixgbe: fix link state timing issue on fiber ports

With some models of fiber ports (e.g. X520-2 device ID 0x10fb), it is possible when a port is started to experience a timing issue which prevents the link from ever being fully set up.

In ixgbe_dev_link_update_share(), if the media type is fiber and the link is down, a flag (IXGBE_FLAG_NEED_LINK_CONFIG) is set. A callback to ixgbe_dev_setup_link_thread_handler() is scheduled which should try to set up the link and clear the flag afterwards.

If the device is started before the flag is cleared, the scheduled callback is cancelled. This causes the flag to remain set and subsequent calls to ixgbe_dev_link_update_share() return without trying to retrieve the link state because the flag is set.

In ixgbe_dev_cancel_link_thread(), after cancelling the callback, unset the flag on the device to avoid this condition.

Fixes 819d0d1d57f1 ("net/ixgbe: fix blocking system events")

Bugzilla ID: 388

---

Quoted 6 lines
Quoted 1 line

Ping.

This fix makes sense to me, thanks for the work.
And it seems can't be applied to latest dpdk-next-net-intel cleanly, could you do a rebase?

Thank you Xiaolong.
I updated in V2, please review it.

Thanks,
Phil

author
#5 lore raw

PATCH v2 1/1 net/ixgbe: fix link state timing issue on fiber ports

In reply to Phil Yang

In ixgbe_dev_link_update_share(), if the media type is fiber and the link is down, a flag (IXGBE_FLAG_NEED_LINK_CONFIG) is set. A callback to ixgbe_dev_setup_link_thread_handler() is scheduled which should try to set up the link and clear the flag afterwards. This flag works as a guard variable between threads.

To avoid potential race condition between threads, set the IXGBE_FLAG_NEED_LINK_CONFIG flag only when there is no link thread running.

Fixes 819d0d1d57f1 ("net/ixgbe: fix blocking system events")

Bugzilla ID: 388

---
v2:
1. rebase code.

 drivers/net/ixgbe/ixgbe_ethdev.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/net/ixgbe/ixgbe_ethdev.c +5 -1
index aa1e8aa..dc19f0c 100644
@@ -4276,9 +4276,13 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 
 	if (link_up == 0) {
 		if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
-			intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
 			ixgbe_dev_wait_setup_link_complete(dev, 0);
 			if (rte_atomic32_test_and_set(&ad->link_thread_running)) {
+				/* To avoid race condition between threads, set
+				 * the IXGBE_FLAG_NEED_LINK_CONFIG flag only
+				 * when there is no link thread running.
+				 */
+				intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
 				if (rte_ctrl_thread_create(&ad->link_thread_tid,
 					"ixgbe-link-handler",
 					NULL,
Signature · 1 line
2.7.4
Acked-by on v2 1/1 Applied
#6 lore raw
Quoted 46 lines · Phil Yang

In ixgbe_dev_link_update_share(), if the media type is fiber and the link is down, a flag (IXGBE_FLAG_NEED_LINK_CONFIG) is set. A callback to ixgbe_dev_setup_link_thread_handler() is scheduled which should try to set up the link and clear the flag afterwards. This flag works as a guard variable between threads.

To avoid potential race condition between threads, set the IXGBE_FLAG_NEED_LINK_CONFIG flag only when there is no link thread running.

Fixes 819d0d1d57f1 ("net/ixgbe: fix blocking system events")

Bugzilla ID: 388

---
v2:
1. rebase code.

drivers/net/ixgbe/ixgbe_ethdev.c | 6 +++++-

1 file changed, 5 insertions(+), 1 deletion(-)
drivers/net/ixgbe/ixgbe_ethdev.c +5 -1
index aa1e8aa..dc19f0c 100644
@@ -4276,9 +4276,13 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 
 	if (link_up == 0) {
 		if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
-			intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
 			ixgbe_dev_wait_setup_link_complete(dev, 0);
 			if (rte_atomic32_test_and_set(&ad->link_thread_running)) {
+				/* To avoid race condition between threads, set
+				 * the IXGBE_FLAG_NEED_LINK_CONFIG flag only
+				 * when there is no link thread running.
+				 */
+				intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
 				if (rte_ctrl_thread_create(&ad->link_thread_tid,
 					"ixgbe-link-handler",
 					NULL,
Signature · 1 line
2.7.4

Applied to dpdk-next-net-intel, Thanks.