net/iavf: add lock for vf commands
PATCH 1/1 net/iavf: add lock for vf commands
iavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently pend_cmd is over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.
Despite this limitation, vf commands may be executed from both iavf_dev_alarm_handler() in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.
Previously [1] was proposed as a potential solution, but this commit has been reverted. I propose adding locks until a more complete solution is available.
[1] commit cb5c1b91f76f ("net/iavf: add thread for event callbacks")
--- drivers/net/iavf/iavf.h | 1 + drivers/net/iavf/iavf_vchnl.c | 76 +++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+)
drivers/net/iavf/iavf.h +1 -0
@@ -262,6 +262,7 @@ struct iavf_info {struct iavf_qv_map *qv_map;/* queue vector mapping */ struct iavf_flow_list flow_list;rte_spinlock_t flow_ops_lock; +rte_spinlock_t aq_lock;struct iavf_parser_list rss_parser_list;struct iavf_parser_list dist_parser_list;struct iavf_parser_list ipsec_crypto_parser_list;
drivers/net/iavf/iavf_vchnl.c +76 -0
340 lines not on this page · raw message
Signature · 1 line
2.31.1
Hello Mike,
Quoted 12 lines · Mike Pattrick
iavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently
pend_cmdis over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.Despite this limitation, vf commands may be executed from both
iavf_dev_alarm_handler()in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.
Quoted 3 lines
Previously [1] was proposed as a potential solution, but this commit has been reverted. I propose adding locks until a more complete solution is available.
- Hum, this commit is still in the main branch, and I did reproduce the issue with a E810 nic.
[1] is not enough for the race on pend_cmd (probably some alarm is still racing with the control thread).
So I think a fix is needed regardless of [1] fate.
- Your patch does make the race disappear, but then, about using a lock on pend_cmd, I wonder what the point to set it atomically is.
- The same issue probably affects the recently copy/pasted^Wintroduced driver idpf.
Quoted 9 lines
[1] commit
cb5c1b91f76f("net/iavf: add thread for event callbacks")Fixes 48de41ca11f0 ("net/avf: enable link status update")Fixes 84108425054a ("net/iavf: support asynchronous virtual channel message")Signed-off-by Mike Pattrick <[email protected]>---
[snip]
Doing a git grep -nC 1 iavf_execute_vf_cmd, I see some unprotected calls. This is probably a rebase issue if you tested on 21.11.
Signature · 1 line
David Marchand
PATCH v2 1/1 net/iavf: add lock for vf commands
iavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently pend_cmd is over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.
Despite this limitation, vf commands may be executed from both iavf_dev_alarm_handler() in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.
Previously [1] was proposed as a potential solution, but this commit did not resolve all potential issues concerning the admin queue and has been reverted from the stable branch. I propose adding locks until a more complete solution is available.
[1] commit cb5c1b91f76f ("net/iavf: add thread for event callbacks")
---
v2:
- Added locks around some iavf_execute_vf_cmd calls which were missed in v1
- Changed description to indicate that [1] was only reverted out of the stable branch.
--- drivers/net/iavf/iavf.h | 1 + drivers/net/iavf/iavf_vchnl.c | 90 +++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+)
drivers/net/iavf/iavf.h +1 -0
@@ -262,6 +262,7 @@ struct iavf_info {struct iavf_qv_map *qv_map;/* queue vector mapping */ struct iavf_flow_list flow_list;rte_spinlock_t flow_ops_lock; +rte_spinlock_t aq_lock;struct iavf_parser_list rss_parser_list;struct iavf_parser_list dist_parser_list;struct iavf_parser_list ipsec_crypto_parser_list;
drivers/net/iavf/iavf_vchnl.c +90 -0
404 lines not on this page · raw message
Signature · 1 line
2.31.1
Quoted 85 lines
-----Original Message-----
From: Mike Pattrick <[email protected]>
Sent: Tuesday, December 20, 2022 11:54 PM
To: [email protected]
Cc: Zhang, Qi Z <[email protected]>; [email protected]; [email protected]; Zhou, YidingX <[email protected]>; [email protected]; Mike Pattrick <[email protected]>; [email protected]
Subject: [PATCH v2] net/iavf: add lock for vf commandsiavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently
pend_cmdis over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.Despite this limitation, vf commands may be executed from both
iavf_dev_alarm_handler()in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.Previously [1] was proposed as a potential solution, but this commit did not resolve all potential issues concerning the admin queue and has been reverted from the stable branch. I propose adding locks until a more complete solution is available.
[1] commit
cb5c1b91f76f("net/iavf: add thread for event callbacks")Fixes 48de41ca11f0 ("net/avf: enable link status update")Fixes 84108425054a ("net/iavf: support asynchronous virtual channelmessage")
Cc: [email protected]Signed-off-by Mike Pattrick <[email protected]>---
v2:
- Added locks around someiavf_execute_vf_cmdcalls which were missed in
v1
- Changed description to indicate that [1] was only reverted out of the stable
branch.--- drivers/net/iavf/iavf.h | 1 + drivers/net/iavf/iavf_vchnl.c | 90 +++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+)drivers/net/iavf/iavf.h index renamed from drivers/net/iavf/iavf.h +0
-0
1edebab8dc..aa18650ffa100644drivers/net/iavf/iavf.h +1
-0@@ -262,6 +262,7 @@ struct iavf_info {struct iavf_qv_map *qv_map;/* queue vector mapping */ struct iavf_flow_list flow_list;rte_spinlock_t flow_ops_lock; +rte_spinlock_t aq_lock;struct iavf_parser_list rss_parser_list;struct iavf_parser_list dist_parser_list; struct iavf_parser_list ipsec_crypto_parser_list; diff --git
a/drivers/net/iavf/iavf_vchnl.cb/drivers/net/iavf/iavf_vchnl.cindexf92daf97f2..edb74edcac100644drivers/net/iavf/iavf_vchnl.c +4
-0@@ -554,7 +554,9 @@ iavf_enable_vlan_strip(struct iavf_adapter *adapter) args.in_args_size =0 ; args.out_buffer = vf->aq_resp; args.out_size =IAVF_AQ_BUF_SZ ; +rte_spinlock_lock (&vf->aq_lock); ret =iavf_execute_vf_cmd (adapter, &args,0 ); +rte_spinlock_unlock (&vf->aq_lock);if (ret)PMD_DRV_LOG (ERR ,"Failed to execute command of" " OP_ENABLE_VLAN_STRIPPING" );@@ -575,7 +577,9 @@ iavf_disable_vlan_strip(struct iavf_adapter *adapter) args.in_args_size =0 ; args.out_buffer = vf->aq_resp; args.out_size =IAVF_AQ_BUF_SZ ; +rte_spinlock_lock (&vf->aq_lock); ret =iavf_execute_vf_cmd (adapter, &args,0 ); +rte_spinlock_unlock (&vf->aq_lock);if (ret)PMD_DRV_LOG (ERR ,"Failed to execute command of" " OP_DISABLE_VLAN_STRIPPING" );
drivers/net/iavf/iavf_vchnl.c iavf_check_api_version(struct iavf_adapter *adapter) · line 613
Quoted 6 lines
drivers/net/iavf/iavf_vchnl.c +2
-0@@ -604,7 +608,9 @@ iavf_check_api_version(struct iavf_adapter *adapter) args.out_buffer = vf->aq_resp; args.out_size =IAVF_AQ_BUF_SZ ; +rte_spinlock_lock (&vf->aq_lock); err =iavf_execute_vf_cmd (adapter, &args,0 ); +rte_spinlock_unlock (&vf->aq_lock);
Instead of wrapping lock and unlock every place, can we add a new wrap function e.g.: iavf_execute_vf_cmd_safe?
PATCH v3 1/1 net/iavf: add lock for vf commands
In reply to Mike Pattrickiavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently pend_cmd is over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.
Despite this limitation, vf commands may be executed from both iavf_dev_alarm_handler() in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.
Previously [1] was proposed as a potential solution, but this commit did not resolve all potential issues concerning the admin queue and has been reverted from the stable branch. I propose adding locks until a more complete solution is available.
[1] commit cb5c1b91f76f ("net/iavf: add thread for event callbacks")
---
v2:
- Added locks around some iavf_execute_vf_cmd calls which were missed in v1
- Changed description to indicate that [1] was only reverted out of the stable branch.
v3:
- Refacted all locks into single function.
--- drivers/net/iavf/iavf.h | 1 + drivers/net/iavf/iavf_vchnl.c | 105 ++++++++++++++++++++-------------- 2 files changed, 62 insertions(+), 44 deletions(-)
drivers/net/iavf/iavf.h +1 -0
@@ -262,6 +262,7 @@ struct iavf_info {struct iavf_qv_map *qv_map;/* queue vector mapping */ struct iavf_flow_list flow_list;rte_spinlock_t flow_ops_lock; +rte_spinlock_t aq_lock;struct iavf_parser_list rss_parser_list;struct iavf_parser_list dist_parser_list;struct iavf_parser_list ipsec_crypto_parser_list;
drivers/net/iavf/iavf_vchnl.c +61 -44
381 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v4 1/1 net/iavf: add lock for vf commands
In reply to Mike Pattrickiavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently pend_cmd is over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.
Despite this limitation, vf commands may be executed from both iavf_dev_alarm_handler() in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.
Previously [1] was proposed as a potential solution, but this commit did not resolve all potential issues concerning the admin queue and has been reverted from the stable branch. I propose adding locks until a more complete solution is available.
[1] commit cb5c1b91f76f ("net/iavf: add thread for event callbacks")
---
v2:
- Added locks around some iavf_execute_vf_cmd calls which were missed in v1
- Changed description to indicate that [1] was only reverted out of the stable branch.
v3:
- Refacted all locks into single function.
v4:
- Previous patch got corrupted
--- drivers/net/iavf/iavf.h | 1 + drivers/net/iavf/iavf_vchnl.c | 106 ++++++++++++++++++++-------------- 2 files changed, 63 insertions(+), 44 deletions(-)
drivers/net/iavf/iavf.h +1 -0
@@ -262,6 +262,7 @@ struct iavf_info {struct iavf_qv_map *qv_map;/* queue vector mapping */ struct iavf_flow_list flow_list;rte_spinlock_t flow_ops_lock; +rte_spinlock_t aq_lock;struct iavf_parser_list rss_parser_list;struct iavf_parser_list dist_parser_list;struct iavf_parser_list ipsec_crypto_parser_list;
drivers/net/iavf/iavf_vchnl.c +62 -44
382 lines not on this page · raw message
Signature · 1 line
2.31.1
Hi iavf maintainers,
Quoted 17 lines · Mike Pattrick
iavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently
pend_cmdis over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.Despite this limitation, vf commands may be executed from both
iavf_dev_alarm_handler()in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.Previously [1] was proposed as a potential solution, but this commit did not resolve all potential issues concerning the admin queue and has been reverted from the stable branch. I propose adding locks until a more complete solution is available.
Quoted 7 lines
[1] commit
cb5c1b91f76f("net/iavf: add thread for event callbacks")Fixes 48de41ca11f0 ("net/avf: enable link status update")Fixes 84108425054a ("net/iavf: support asynchronous virtual channel message")Signed-off-by Mike Pattrick <[email protected]>
Review please.
Signature · 1 line
David Marchand
Quoted 27 lines
-----Original Message-----
From: Mike Pattrick <[email protected]>
Sent: Thursday, December 29, 2022 7:00 AM
To: [email protected]
Cc: Zhang, Qi Z <[email protected]>; [email protected]; [email protected]; Zhou, YidingX <[email protected]>; [email protected]; Mike Pattrick <[email protected]>; [email protected]
Subject: [PATCH v4] net/iavf: add lock for vf commandsiavf admin queue commands aren't thread-safe. Bugs surrounding this issue can manifest in a variety of ways but frequently
pend_cmdis over written. Simultaneously executing commands can result in a misconfigured device or DPDK sleeping in a thread for 2 second.Despite this limitation, vf commands may be executed from both
iavf_dev_alarm_handler()in a control thread and the applications main thread. This is trivial to simulate in the testpmd application by creating a bond of vf's in active backup mode, and then turning the bond off and then on again repeatedly.Previously [1] was proposed as a potential solution, but this commit did not resolve all potential issues concerning the admin queue and has been reverted from the stable branch. I propose adding locks until a more complete solution is available.
Quoted 8 lines
[1] commit
cb5c1b91f76f("net/iavf: add thread for event callbacks")Fixes 48de41ca11f0 ("net/avf: enable link status update")Fixes 84108425054a ("net/iavf: support asynchronous virtual channelmessage")
Cc: [email protected]Signed-off-by Mike Pattrick <[email protected]>
Applied to dpdk-next-net-intel.
Thanks
Qi