Skip to contentSkip to conversation

net/iavf: add lock for vf commands

PATCH Superseded Mike Pattrick 3 participants 8 messages Dec 19, 2022 – Jan 17, 2023 Network drivers

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net/iavf: add lock for vf commands

iavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently pend_cmd is over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.

Despite this limitation, vf commands may be executed from both iavf_dev_alarm_handler() in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.

Previously [1] was proposed as a potential solution, but this commit has been reverted. I propose adding locks until a more complete solution is available.

[1] commit cb5c1b91f76f ("net/iavf: add thread for event callbacks")

Fixes 48de41ca11f0 ("net/avf: enable link status update")
Fixes 84108425054a ("net/iavf: support asynchronous virtual channel message")
---
 drivers/net/iavf/iavf.h       |  1 +
 drivers/net/iavf/iavf_vchnl.c | 76 +++++++++++++++++++++++++++++++++++
 2 files changed, 77 insertions(+)
drivers/net/iavf/iavf.h +1 -0
index 1edebab8dc..aa18650ffa 100644
@@ -262,6 +262,7 @@ struct iavf_info {
 	struct iavf_qv_map *qv_map; /* queue vector mapping */
 	struct iavf_flow_list flow_list;
 	rte_spinlock_t flow_ops_lock;
+	rte_spinlock_t aq_lock;
 	struct iavf_parser_list rss_parser_list;
 	struct iavf_parser_list dist_parser_list;
 	struct iavf_parser_list ipsec_crypto_parser_list;
drivers/net/iavf/iavf_vchnl.c +76 -0

340 lines not on this page · raw message

Signature · 1 line
2.31.1
#1 lore raw

Hello Mike,

Quoted 12 lines · Mike Pattrick

iavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently pend_cmd is over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.

Despite this limitation, vf commands may be executed from both iavf_dev_alarm_handler() in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.

Quoted 3 lines

Previously [1] was proposed as a potential solution, but this commit has been reverted. I propose adding locks until a more complete solution is available.

- Hum, this commit is still in the main branch, and I did reproduce the issue with a E810 nic.
[1] is not enough for the race on pend_cmd (probably some alarm is still racing with the control thread).

So I think a fix is needed regardless of [1] fate.

- Your patch does make the race disappear, but then, about using a lock on pend_cmd, I wonder what the point to set it atomically is.

- The same issue probably affects the recently copy/pasted^Wintroduced driver idpf.

Quoted 9 lines

[1] commit cb5c1b91f76f ("net/iavf: add thread for event callbacks")

Fixes 48de41ca11f0 ("net/avf: enable link status update")
Fixes 84108425054a ("net/iavf: support asynchronous virtual channel message")

---

[snip]

Doing a git grep -nC 1 iavf_execute_vf_cmd, I see some unprotected calls. This is probably a rebase issue if you tested on 21.11.

Signature · 1 line
David Marchand
#2 lore raw

PATCH v2 1/1 net/iavf: add lock for vf commands

iavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently pend_cmd is over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.

Despite this limitation, vf commands may be executed from both iavf_dev_alarm_handler() in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.

Previously [1] was proposed as a potential solution, but this commit did not resolve all potential issues concerning the admin queue and has been reverted from the stable branch. I propose adding locks until a more complete solution is available.

[1] commit cb5c1b91f76f ("net/iavf: add thread for event callbacks")

Fixes 48de41ca11f0 ("net/avf: enable link status update")
Fixes 84108425054a ("net/iavf: support asynchronous virtual channel message")

---

v2:
- Added locks around some iavf_execute_vf_cmd calls which were missed in v1
- Changed description to indicate that [1] was only reverted out of the stable branch.

---
 drivers/net/iavf/iavf.h       |  1 +
 drivers/net/iavf/iavf_vchnl.c | 90 +++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+)
drivers/net/iavf/iavf.h +1 -0
index 1edebab8dc..aa18650ffa 100644
@@ -262,6 +262,7 @@ struct iavf_info {
 	struct iavf_qv_map *qv_map; /* queue vector mapping */
 	struct iavf_flow_list flow_list;
 	rte_spinlock_t flow_ops_lock;
+	rte_spinlock_t aq_lock;
 	struct iavf_parser_list rss_parser_list;
 	struct iavf_parser_list dist_parser_list;
 	struct iavf_parser_list ipsec_crypto_parser_list;
drivers/net/iavf/iavf_vchnl.c +90 -0

404 lines not on this page · raw message

Signature · 1 line
2.31.1
#3 lore raw
Quoted 85 lines

-----Original Message-----
From: Mike Pattrick <[email protected]>
Sent: Tuesday, December 20, 2022 11:54 PM
To: [email protected]
Cc: Zhang, Qi Z <[email protected]>; [email protected]; [email protected]; Zhou, YidingX <[email protected]>; [email protected]; Mike Pattrick <[email protected]>; [email protected]
Subject: [PATCH v2] net/iavf: add lock for vf commands

iavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently pend_cmd is over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.

Despite this limitation, vf commands may be executed from both iavf_dev_alarm_handler() in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.

Previously [1] was proposed as a potential solution, but this commit did not resolve all potential issues concerning the admin queue and has been reverted from the stable branch. I propose adding locks until a more complete solution is available.

[1] commit cb5c1b91f76f ("net/iavf: add thread for event callbacks")

Fixes 48de41ca11f0 ("net/avf: enable link status update")
Fixes 84108425054a ("net/iavf: support asynchronous virtual channel

message")
Cc: [email protected]

---

v2:
- Added locks around some iavf_execute_vf_cmd calls which were missed in
v1
- Changed description to indicate that [1] was only reverted out of the stable
branch.

---
 drivers/net/iavf/iavf.h       |  1 +
 drivers/net/iavf/iavf_vchnl.c | 90 +++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+)
drivers/net/iavf/iavf.h index renamed from drivers/net/iavf/iavf.h +0 -0

1edebab8dc..aa18650ffa 100644

drivers/net/iavf/iavf.h +1 -0
@@ -262,6 +262,7 @@ struct iavf_info {
 	struct iavf_qv_map *qv_map; /* queue vector mapping */
 	struct iavf_flow_list flow_list;
 	rte_spinlock_t flow_ops_lock;
+	rte_spinlock_t aq_lock;
 	struct iavf_parser_list rss_parser_list;
 	struct iavf_parser_list dist_parser_list;
 	struct iavf_parser_list ipsec_crypto_parser_list; diff --git

a/drivers/net/iavf/iavf_vchnl.c b/drivers/net/iavf/iavf_vchnl.c index f92daf97f2..edb74edcac 100644

drivers/net/iavf/iavf_vchnl.c +4 -0
@@ -554,7 +554,9 @@ iavf_enable_vlan_strip(struct iavf_adapter *adapter)
 	args.in_args_size = 0;
 	args.out_buffer = vf->aq_resp;
 	args.out_size = IAVF_AQ_BUF_SZ;
+	rte_spinlock_lock(&vf->aq_lock);
 	ret = iavf_execute_vf_cmd(adapter, &args, 0);
+	rte_spinlock_unlock(&vf->aq_lock);
 	if (ret)
 		PMD_DRV_LOG(ERR, "Failed to execute command of"
 			    " OP_ENABLE_VLAN_STRIPPING");
@@ -575,7 +577,9 @@ iavf_disable_vlan_strip(struct iavf_adapter *adapter)
 	args.in_args_size = 0;
 	args.out_buffer = vf->aq_resp;
 	args.out_size = IAVF_AQ_BUF_SZ;
+	rte_spinlock_lock(&vf->aq_lock);
 	ret = iavf_execute_vf_cmd(adapter, &args, 0);
+	rte_spinlock_unlock(&vf->aq_lock);
 	if (ret)
 		PMD_DRV_LOG(ERR, "Failed to execute command of"
 			    " OP_DISABLE_VLAN_STRIPPING");

drivers/net/iavf/iavf_vchnl.c iavf_check_api_version(struct iavf_adapter *adapter) · line 613

Quoted 6 lines
drivers/net/iavf/iavf_vchnl.c +2 -0
@@ -604,7 +608,9 @@ iavf_check_api_version(struct iavf_adapter *adapter)
 	args.out_buffer = vf->aq_resp;
 	args.out_size = IAVF_AQ_BUF_SZ;
 
+	rte_spinlock_lock(&vf->aq_lock);
 	err = iavf_execute_vf_cmd(adapter, &args, 0);
+	rte_spinlock_unlock(&vf->aq_lock);

Instead of wrapping lock and unlock every place, can we add a new wrap function e.g.: iavf_execute_vf_cmd_safe?

#4 lore raw

PATCH v3 1/1 net/iavf: add lock for vf commands

In reply to Mike Pattrick

iavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently pend_cmd is over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.

Despite this limitation, vf commands may be executed from both iavf_dev_alarm_handler() in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.

Previously [1] was proposed as a potential solution, but this commit did not resolve all potential issues concerning the admin queue and has been reverted from the stable branch. I propose adding locks until a more complete solution is available.

[1] commit cb5c1b91f76f ("net/iavf: add thread for event callbacks")

Fixes 48de41ca11f0 ("net/avf: enable link status update")
Fixes 84108425054a ("net/iavf: support asynchronous virtual channel message")

---

v2:
- Added locks around some iavf_execute_vf_cmd calls which were missed in v1
- Changed description to indicate that [1] was only reverted out of the stable branch.

v3:
- Refacted all locks into single function.

---
 drivers/net/iavf/iavf.h       |   1 +
 drivers/net/iavf/iavf_vchnl.c | 105 ++++++++++++++++++++--------------
 2 files changed, 62 insertions(+), 44 deletions(-)
drivers/net/iavf/iavf.h +1 -0
index 1edebab8dc..aa18650ffa 100644
@@ -262,6 +262,7 @@ struct iavf_info {
 	struct iavf_qv_map *qv_map; /* queue vector mapping */
 	struct iavf_flow_list flow_list;
 	rte_spinlock_t flow_ops_lock;
+	rte_spinlock_t aq_lock;
 	struct iavf_parser_list rss_parser_list;
 	struct iavf_parser_list dist_parser_list;
 	struct iavf_parser_list ipsec_crypto_parser_list;
drivers/net/iavf/iavf_vchnl.c +61 -44

381 lines not on this page · raw message

Signature · 1 line
2.31.1
#5 lore raw

PATCH v4 1/1 net/iavf: add lock for vf commands

In reply to Mike Pattrick

iavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently pend_cmd is over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.

Despite this limitation, vf commands may be executed from both iavf_dev_alarm_handler() in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.

Previously [1] was proposed as a potential solution, but this commit did not resolve all potential issues concerning the admin queue and has been reverted from the stable branch. I propose adding locks until a more complete solution is available.

[1] commit cb5c1b91f76f ("net/iavf: add thread for event callbacks")

Fixes 48de41ca11f0 ("net/avf: enable link status update")
Fixes 84108425054a ("net/iavf: support asynchronous virtual channel message")

---

v2:
- Added locks around some iavf_execute_vf_cmd calls which were missed in v1
- Changed description to indicate that [1] was only reverted out of the stable branch.

v3:
- Refacted all locks into single function.

v4:
- Previous patch got corrupted

---
 drivers/net/iavf/iavf.h       |   1 +
 drivers/net/iavf/iavf_vchnl.c | 106 ++++++++++++++++++++--------------
 2 files changed, 63 insertions(+), 44 deletions(-)
drivers/net/iavf/iavf.h +1 -0
index 1edebab8dc..aa18650ffa 100644
@@ -262,6 +262,7 @@ struct iavf_info {
 	struct iavf_qv_map *qv_map; /* queue vector mapping */
 	struct iavf_flow_list flow_list;
 	rte_spinlock_t flow_ops_lock;
+	rte_spinlock_t aq_lock;
 	struct iavf_parser_list rss_parser_list;
 	struct iavf_parser_list dist_parser_list;
 	struct iavf_parser_list ipsec_crypto_parser_list;
drivers/net/iavf/iavf_vchnl.c +62 -44

382 lines not on this page · raw message

Signature · 1 line
2.31.1
#6 lore raw

Hi iavf maintainers,

Quoted 17 lines · Mike Pattrick

iavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently pend_cmd is over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.

Despite this limitation, vf commands may be executed from both iavf_dev_alarm_handler() in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.

Previously [1] was proposed as a potential solution, but this commit did not resolve all potential issues concerning the admin queue and has been reverted from the stable branch. I propose adding locks until a more complete solution is available.

Quoted 7 lines

[1] commit cb5c1b91f76f ("net/iavf: add thread for event callbacks")

Fixes 48de41ca11f0 ("net/avf: enable link status update")
Fixes 84108425054a ("net/iavf: support asynchronous virtual channel message")

Review please.

Signature · 1 line
David Marchand
Acked-by on v4 1/1 Applied
#7 lore raw
In reply to Mike Pattrick
Quoted 27 lines

-----Original Message-----
From: Mike Pattrick <[email protected]>
Sent: Thursday, December 29, 2022 7:00 AM
To: [email protected]
Cc: Zhang, Qi Z <[email protected]>; [email protected]; [email protected]; Zhou, YidingX <[email protected]>; [email protected]; Mike Pattrick <[email protected]>; [email protected]
Subject: [PATCH v4] net/iavf: add lock for vf commands

iavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently pend_cmd is over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.

Despite this limitation, vf commands may be executed from both iavf_dev_alarm_handler() in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.

Previously [1] was proposed as a potential solution, but this commit did not resolve all potential issues concerning the admin queue and has been reverted from the stable branch. I propose adding locks until a more complete solution is available.

Quoted 8 lines

[1] commit cb5c1b91f76f ("net/iavf: add thread for event callbacks")

Fixes 48de41ca11f0 ("net/avf: enable link status update")
Fixes 84108425054a ("net/iavf: support asynchronous virtual channel

message")
Cc: [email protected]

Applied to dpdk-next-net-intel.

Thanks
Qi