v2 0/4]
Subsystems (3)
PATCH v2 1/1 v2 0/4]
IDPF Virtchnl: Enhance error reporting & fix locking/workqueue issues
This patch series addresses several IDPF virtchnl issues:
* Improved error reporting for better diagnostics.
* Fixed locking sequence in virtchnl message handling to avoid potential race conditions.
* Converted idpf workqueues to unbound to prevent virtchnl processing delays under heavy load.
Previously, CPU-bound kworkers for virtchnl processing could be starved,
leading to transaction timeouts and connection failures.
This was particularly problematic when IRQ traffic and user space processes contended for the same CPU.
By making the workqueues unbound, we ensure virtchnl processing is not tied to a specific CPU, improving responsiveness even under high system load.
---
V2:
- Dropped patch from Willem
- RCS/RCT variable naming
- Improved commit message on feddback
v1: https://lore.kernel.org/netdev/[email protected]/T/
David Decotigny (1):
idpf: address an rtnl lock splat in tx timeout recovery path
Manoj Vishwanathan (2):
idpf: Acquire the lock before accessing the xn->salt
idpf: add more info during virtchnl transaction time out
Marco Leogrande (1):
idpf: convert workqueues to unbound
drivers/net/ethernet/intel/idpf/idpf_main.c | 15 ++++++++++----- drivers/net/ethernet/intel/idpf/idpf_txrx.c | 14 +++++++++++++- drivers/net/ethernet/intel/idpf/idpf_virtchnl.c | 12 +++++++----- 3 files changed, 30 insertions(+), 11 deletions(-)
Signature · 1 line
2.46.0.295.g3b9ea8a38a-goog
PATCH v2 1/1 v2 1/4] idpf: address an rtnl lock splat in tx timeout recovery path
From: David Decotigny <[email protected]>
Adopt the same pattern as in other places in the code to take the rtnl lock during hard resets.
Tested the patch by injecting tx timeout in IDPF , observe that idpf recovers and IDPF comes back reachable
Without this patch causes there is a splat:
[ 270.145214] WARNING: CPU: PID: at net/sched/sch_generic.c:534 dev_watchdog
--- drivers/net/ethernet/intel/idpf/idpf_txrx.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/idpf/idpf_txrx.c +13 -1
@@ -4326,6 +4326,7 @@ int idpf_vport_intr_alloc(struct idpf_vport *vport)*/ int idpf_vport_intr_init (struct idpf_vport *vport) { +bool hr_reset_in_prog;char *int_name;int err;
@@ -4334,8 +4335,19 @@ int idpf_vport_intr_init(struct idpf_vport *vport)return err;idpf_vport_intr_map_vector_to_qs (vport); +/** +* If we're in normal up path, the stack already takes the +* rtnl_lock for us, however, if we're doing up as a part of a +* hard reset, we'll need to take the lock ourself before +* touching the netdev. +*/ + hr_reset_in_prog =test_bit (IDPF_HR_RESET_IN_PROG , + vport->adapter->flags); +if (hr_reset_in_prog) +rtnl_lock ();idpf_vport_intr_napi_add_all (vport); - +if (hr_reset_in_prog) +rtnl_unlock (); err = vport->adapter->dev_ops.reg_ops.intr_reg_init (vport);if (err) goto unroll_vectors_alloc;
Signature · 1 line
2.46.0.295.g3b9ea8a38a-goog
Re: [Intel-wired-lan] [[PATCH v2 iwl-next] v2 1/4] idpf: address an rtnl lock splat in tx timeout recovery path
Quoted 36 lines · Manoj Vishwanathan
From: David Decotigny <[email protected]>
Adopt the same pattern as in other places in the code to take the rtnl lock during hard resets.
Tested the patch by injecting tx timeout in IDPF , observe that idpf recovers and IDPF comes back reachableWithout this patch causes there is a splat:
[ 270.145214] WARNING: CPU: PID: at net/sched/sch_generic.c:534 dev_watchdogFixes d4d5587182664 (idpf: initialize interrupts and enable vport)Signed-off-by Manoj Vishwanathan <[email protected]>--- drivers/net/ethernet/intel/idpf/idpf_txrx.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)drivers/net/ethernet/intel/idpf/idpf_txrx.c +5
-0@@ -4326,6 +4326,7 @@ int idpf_vport_intr_alloc(struct idpf_vport *vport)*/ int idpf_vport_intr_init (struct idpf_vport *vport) { +bool hr_reset_in_prog;char *int_name;int err;@@ -4334,8 +4335,19 @@ int idpf_vport_intr_init(struct idpf_vport *vport)return err;idpf_vport_intr_map_vector_to_qs (vport); +/** +* If we're in normal up path, the stack already takes the +* rtnl_lock for us, however, if we're doing up as a part of a +* hard reset, we'll need to take the lock ourself before
drivers/net/ethernet/intel/idpf/idpf_txrx.c int idpf_vport_intr_init(struct idpf_vport *vport) · line 4350
Quoted 10 lines
drivers/net/ethernet/intel/idpf/idpf_txrx.c +8
-1+* touching the netdev. +*/ + hr_reset_in_prog =test_bit (IDPF_HR_RESET_IN_PROG , + vport->adapter->flags); +if (hr_reset_in_prog) +rtnl_lock ();idpf_vport_intr_napi_add_all (vport); - +if (hr_reset_in_prog) +rtnl_unlock ();
This feels a little fragile. Why not pass the reset in progress as a flag from the caller? Surely the caller knows whether this is happening due to an interface up or due to a reset?
PATCH v2 1/1 v2 2/4] idpf: Acquire the lock before accessing the xn->salt
In reply to Manoj VishwanathanThe transaction salt was being accessed before acquiring the idpf_vc_xn_lock when idpf has to forward the virtchnl reply.
--- drivers/net/ethernet/intel/idpf/idpf_virtchnl.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +2 -1
@@ -612,14 +612,15 @@ idpf_vc_xn_forward_reply(struct idpf_adapter *adapter,return -EINVAL ; } xn = &adapter->vcxn_mngr->ring[xn_idx]; +idpf_vc_xn_lock (xn); salt =FIELD_GET (IDPF_VC_XN_SALT_M , msg_info);if (xn->salt != salt) {dev_err_ratelimited (&adapter->pdev->dev,"Transaction salt does not match (%02x != %02x)\n" , xn->salt, salt); +idpf_vc_xn_unlock (xn);return -EINVAL ; } -idpf_vc_xn_lock (xn);switch (xn->state) {case IDPF_VC_XN_WAITING :/* success */
Signature · 1 line
2.46.0.295.g3b9ea8a38a-goog
Re: [Intel-wired-lan] [[PATCH v2 iwl-next] v2 2/4] idpf: Acquire the lock before accessing the xn->salt
Quoted 6 lines · Manoj Vishwanathan
The transaction salt was being accessed before acquiring the
idpf_vc_xn_lockwhen idpf has to forward the virtchnl reply.Fixes 34c21fa894a1a (“idpf: implement virtchnl transaction manager”)Signed-off-by Manoj Vishwanathan <[email protected]>---
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c idpf_vc_xn_forward_reply(struct idpf_adapter *adapter, · line 615
Quoted 12 lines
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +1
-0@@ -612,14 +612,15 @@ idpf_vc_xn_forward_reply(struct idpf_adapter *adapter,return -EINVAL ; } xn = &adapter->vcxn_mngr->ring[xn_idx]; +idpf_vc_xn_lock (xn);
Could look at implementing cleanup.h based locking here so that we could use guard or scope_guard and not have to litter the exit paths with unlocks.
I don't think that needs to be done in this patch, though.
Quoted 12 lines
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +1
-1salt =FIELD_GET (IDPF_VC_XN_SALT_M , msg_info);if (xn->salt != salt) {dev_err_ratelimited (&adapter->pdev->dev,"Transaction salt does not match (%02x != %02x)\n" , xn->salt, salt); +idpf_vc_xn_unlock (xn);return -EINVAL ; } -idpf_vc_xn_lock (xn);switch (xn->state) {case IDPF_VC_XN_WAITING :/* success */
Przemek Kitszel
maintainerRe: [Intel-wired-lan] [[PATCH v2 iwl-next] v2 2/4] idpf: Acquire the lock before accessing the xn->salt
Quoted 15 lines · Jacob Keller
Quoted 6 lines · Manoj Vishwanathan
The transaction salt was being accessed before acquiring the
idpf_vc_xn_lockwhen idpf has to forward the virtchnl reply.Fixes 34c21fa894a1a (“idpf: implement virtchnl transaction manager”)Signed-off-by Manoj Vishwanathan <[email protected]>---
Reviewed-by Jacob Keller <[email protected]>Quoted 3 lines
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 12 lines
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c idpf_vc_xn_forward_reply(struct idpf_adapter *adapter, · line 615
Quoted 9 lines
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +1
-0@@ -612,14 +612,15 @@ idpf_vc_xn_forward_reply(struct idpf_adapter *adapter,return -EINVAL ; } xn = &adapter->vcxn_mngr->ring[xn_idx]; +idpf_vc_xn_lock (xn);Could look at implementing
cleanup.hbased locking here so that we could use guard orscope_guardand not have to litter the exit paths with unlocks.
only scope_guard() for networking code
Quoted 2 lines
I don't think that needs to be done in this patch, though.
+1
Quoted 13 lines
Quoted 13 lines
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +1
-1salt =FIELD_GET (IDPF_VC_XN_SALT_M , msg_info);if (xn->salt != salt) {dev_err_ratelimited (&adapter->pdev->dev,"Transaction salt does not match (%02x != %02x)\n" , xn->salt, salt); +idpf_vc_xn_unlock (xn);return -EINVAL ; } -idpf_vc_xn_lock (xn);switch (xn->state) {case IDPF_VC_XN_WAITING :/* success */
RE: [Intel-wired-lan] [[PATCH v2 iwl-next] v2 2/4] idpf: Acquire the lock before accessing the xn->salt
Quoted 45 lines
-----Original Message-----
From: Kitszel, Przemyslaw <[email protected]>
Sent: Thursday, August 29, 2024 11:05 PM
To: Keller, Jacob E <[email protected]>; Nguyen, Anthony L <[email protected]>
Cc: [email protected]; [email protected]; google-lan- [email protected]; Manoj Vishwanathan <[email protected]>; David S. Miller <[email protected]>; Eric Dumazet <[email protected]>; [email protected]
Subject: Re: [Intel-wired-lan] [[PATCH v2 iwl-next] v2 2/4] idpf: Acquire the lock before accessing thexn->saltQuoted 10 lines · Jacob Keller
Quoted 6 lines · Manoj Vishwanathan
The transaction salt was being accessed before acquiring the
idpf_vc_xn_lockwhen idpf has to forward the virtchnl reply.Fixes 34c21fa894a1a (“idpf: implement virtchnl transaction manager”)Signed-off-by Manoj Vishwanathan <[email protected]>---
Quoted 6 lines
Reviewed-by Jacob Keller <[email protected]>Review
Quoted 4 lines
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)(quoted diff) +0
-0
b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.cQuoted 4 lines
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c idpf_vc_xn_forward_reply(struct idpf_adapter
Quoted 4 lines
index
70986e12da28..30eec674d594100644drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +0
-0@@ -612,14 +612,15 @@ idpf_vc_xn_forward_reply(struct idpf_adapter*adapter,
Quoted 7 lines
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c idpf_vc_xn_forward_reply(struct idpf_adapter *adapter, · line 615
Quoted 4 lines
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +1
-0return -EINVAL ; } xn = &adapter->vcxn_mngr->ring[xn_idx]; +idpf_vc_xn_lock (xn);Could look at implementing
cleanup.hbased locking here so that we could use guard orscope_guardand not have to litter the exit paths with unlocks.
Quoted 2 lines
only
scope_guard()for networking code
Yea, leaving it as-is is fine. I personally find cleanup-based locking better, but it appears the maintainers and majority feel otherwise.
Re: [[PATCH v2 iwl-next] v2 2/4] idpf: Acquire the lock before accessing the xn->salt
In reply to Manoj VishwanathanQuoted 5 lines · Manoj Vishwanathan
The transaction salt was being accessed before acquiring the
idpf_vc_xn_lockwhen idpf has to forward the virtchnl reply.Fixes 34c21fa894a1a (“idpf: implement virtchnl transaction manager”)Signed-off-by Manoj Vishwanathan <[email protected]>
Quoted 25 lines
--- drivers/net/ethernet/intel/idpf/idpf_virtchnl.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +2
-1@@ -612,14 +612,15 @@ idpf_vc_xn_forward_reply(struct idpf_adapter *adapter,return -EINVAL ; } xn = &adapter->vcxn_mngr->ring[xn_idx]; +idpf_vc_xn_lock (xn); salt =FIELD_GET (IDPF_VC_XN_SALT_M , msg_info);if (xn->salt != salt) {dev_err_ratelimited (&adapter->pdev->dev,"Transaction salt does not match (%02x != %02x)\n" , xn->salt, salt); +idpf_vc_xn_unlock (xn);return -EINVAL ; } -idpf_vc_xn_lock (xn);switch (xn->state) {case IDPF_VC_XN_WAITING :/* success */
PATCH v2 1/1 v2 3/4] idpf: convert workqueues to unbound
In reply to Manoj VishwanathanFrom: Marco Leogrande <[email protected]>
When a workqueue is created with WQ_UNBOUND, its work items are served by special worker-pools, whose host workers are not bound to any specific CPU. In the default configuration (i.e. when queue_delayed_work and friends do not specify which CPU to run the work item on), WQ_UNBOUND allows the work item to be executed on any CPU in the same node of the CPU it was enqueued on. While this solution potentially sacrifices locality, it avoids contention with other processes that might dominate the CPU time of the processor the work item was scheduled on.
This is not just a theoretical problem: in a praticular scenario misconfigured process was hogging most of the time from CPU0, leaving less than 0.5% of its CPU time to the kworker. The IDPF workqueues that were using the kworker on CPU0 suffered large completion delays as a result, causing performance degradation, timeouts and eventual system crash.
Tested:
* I have also run a manual test to gauge the performance improvement. The test consists of an antagonist process (./stress --cpu 2) consuming as much of CPU 0 as possible. This process is run under taskset 01 to bind it to CPU0, and its priority is changed with chrt -pQ 9900 10000 ${pid} and renice -n -20 ${pid} after start.
Then, the IDPF driver is forced to prefer CPU0 by editing all calls to queue_delayed_work, mod_delayed_work, etc... to use CPU 0.
Finally, ktraces for the workqueue events are collected.
Without the current patch, the antagonist process can force arbitrary delays between workqueue_queue_work and workqueue_execute_start, that in my tests were as high as 30ms. With the current patch applied, the workqueue can be migrated to another unloaded CPU in the same node, and, keeping everything else equal, the maximum delay I could see was 6us.
--- drivers/net/ethernet/intel/idpf/idpf_main.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
drivers/net/ethernet/intel/idpf/idpf_main.c +10 -5
@@ -174,7 +174,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)pci_set_master (pdev);pci_set_drvdata (pdev, adapter); - adapter->init_wq =alloc_workqueue ("%s-%s-init" ,0 ,0 , + adapter->init_wq =alloc_workqueue ("%s-%s-init" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->init_wq) {
@@ -183,7 +184,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_free; } - adapter->serv_wq =alloc_workqueue ("%s-%s-service" ,0 ,0 , + adapter->serv_wq =alloc_workqueue ("%s-%s-service" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->serv_wq) {
@@ -192,7 +194,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_serv_wq_alloc; } - adapter->mbx_wq =alloc_workqueue ("%s-%s-mbx" ,0 ,0 , + adapter->mbx_wq =alloc_workqueue ("%s-%s-mbx" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->mbx_wq) {
@@ -201,7 +204,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_mbx_wq_alloc; } - adapter->stats_wq =alloc_workqueue ("%s-%s-stats" ,0 ,0 , + adapter->stats_wq =alloc_workqueue ("%s-%s-stats" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->stats_wq) {
@@ -210,7 +214,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_stats_wq_alloc; } - adapter->vc_event_wq =alloc_workqueue ("%s-%s-vc_event" ,0 ,0 , + adapter->vc_event_wq =alloc_workqueue ("%s-%s-vc_event" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->vc_event_wq) {
Signature · 1 line
2.46.0.295.g3b9ea8a38a-goog
Re: [Intel-wired-lan] [[PATCH v2 iwl-next] v2 3/4] idpf: convert workqueues to unbound
Quoted 12 lines · Manoj Vishwanathan
From: Marco Leogrande <[email protected]>
When a workqueue is created with
WQ_UNBOUND, its work items are served by special worker-pools, whose host workers are not bound to any specific CPU. In the default configuration (i.e. whenqueue_delayed_workand friends do not specify which CPU to run the work item on),WQ_UNBOUNDallows the work item to be executed on any CPU in the same node of the CPU it was enqueued on. While this solution potentially sacrifices locality, it avoids contention with other processes that might dominate the CPU time of the processor the work item was scheduled on.
Quoted 1 line
This is not just a theoretical problem: in a praticular scenario
Nit: s/praticular/particular/
Quoted 6 lines
misconfigured process was hogging most of the time from CPU0, leaving less than 0.5% of its CPU time to the kworker. The IDPF workqueues that were using the kworker on CPU0 suffered large completion delays as a result, causing performance degradation, timeouts and eventual system crash.
Curious how the delay could result in a full system crash. That seems like some other concurrency issue. I guess something like a Tx timeout could happen though.
Quoted 12 lines
Tested:
* I have also run a manual test to gauge the performance improvement. The test consists of an antagonist process (
./stress --cpu 2) consuming as much of CPU 0 as possible. This process is run undertaskset 01to bind it to CPU0, and its priority is changed withchrt -pQ 9900 10000 ${pid}andrenice -n -20 ${pid}after start.Then, the IDPF driver is forced to prefer CPU0 by editing all calls to
queue_delayed_work,mod_delayed_work, etc... to use CPU 0.
Quoted 9 lines
Finally,
ktracesfor the workqueue events are collected.Without the current patch, the antagonist process can force arbitrary delays between
workqueue_queue_workandworkqueue_execute_start, that in my tests were as high as30ms. With the current patch applied, the workqueue can be migrated to another unloaded CPU in the same node, and, keeping everything else equal, the maximum delay I could see was6us.
Hmm. I don't have a direct issue with using WQ_UNBOUND, and I can't think of any reason these work queue tasks *need* to be CPU bound.
I do feel like there may be other solutions to managing the tasks on the system such that this isn't necessary.
However, if using WQ_UNBOUND solves these problems and is simpler in that system administrators are less likely to screw things up, I think its a net positive.
I do not know if there are any other side effects of WQ_UNBOUND, so take this with a grain of salt:
Quoted 52 lines
Fixes 0fe45467a1041 (idpf: add create vport and netdev configuration)Signed-off-by Marco Leogrande <[email protected]>Signed-off-by Manoj Vishwanathan <[email protected]>--- drivers/net/ethernet/intel/idpf/idpf_main.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)drivers/net/ethernet/intel/idpf/idpf_main.c +8
-4@@ -174,7 +174,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)pci_set_master (pdev);pci_set_drvdata (pdev, adapter); - adapter->init_wq =alloc_workqueue ("%s-%s-init" ,0 ,0 , + adapter->init_wq =alloc_workqueue ("%s-%s-init" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->init_wq) {@@ -183,7 +184,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_free; } - adapter->serv_wq =alloc_workqueue ("%s-%s-service" ,0 ,0 , + adapter->serv_wq =alloc_workqueue ("%s-%s-service" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->serv_wq) {@@ -192,7 +194,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_serv_wq_alloc; } - adapter->mbx_wq =alloc_workqueue ("%s-%s-mbx" ,0 ,0 , + adapter->mbx_wq =alloc_workqueue ("%s-%s-mbx" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->mbx_wq) {@@ -201,7 +204,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_mbx_wq_alloc; } - adapter->stats_wq =alloc_workqueue ("%s-%s-stats" ,0 ,0 , + adapter->stats_wq =alloc_workqueue ("%s-%s-stats" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->stats_wq) {
drivers/net/ethernet/intel/idpf/idpf_main.c static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) · line 221
Quoted 9 lines
drivers/net/ethernet/intel/idpf/idpf_main.c +2
-1@@ -210,7 +214,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_stats_wq_alloc; } - adapter->vc_event_wq =alloc_workqueue ("%s-%s-vc_event" ,0 ,0 , + adapter->vc_event_wq =alloc_workqueue ("%s-%s-vc_event" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->vc_event_wq) {
This seems like quite a lot of work queues for a driver :D
Re: [Intel-wired-lan] [[PATCH v2 iwl-next] v2 3/4] idpf: convert workqueues to unbound
In reply to Manoj VishwanathanQuoted 39 lines · Manoj Vishwanathan
From: Marco Leogrande <[email protected]>
When a workqueue is created with
WQ_UNBOUND, its work items are served by special worker-pools, whose host workers are not bound to any specific CPU. In the default configuration (i.e. whenqueue_delayed_workand friends do not specify which CPU to run the work item on),WQ_UNBOUNDallows the work item to be executed on any CPU in the same node of the CPU it was enqueued on. While this solution potentially sacrifices locality, it avoids contention with other processes that might dominate the CPU time of the processor the work item was scheduled on.This is not just a theoretical problem: in a praticular scenario > misconfigured process was hogging most of the time from CPU0, leaving less than 0.5% of its CPU time to the kworker. The IDPF workqueues
that were using the kworker on CPU0 suffered large completion delays
as a result, causing performance degradation, timeouts and eventual
system crash.Tested:
* I have also run a manual test to gauge the performance improvement. The test consists of an antagonist process (
./stress --cpu 2) consuming as much of CPU 0 as possible. This process is run undertaskset 01to bind it to CPU0, and its priority is changed withchrt -pQ 9900 10000 ${pid}andrenice -n -20 ${pid}after start.Then, the IDPF driver is forced to prefer CPU0 by editing all calls to
queue_delayed_work,mod_delayed_work, etc... to use CPU 0.Finally,
ktracesfor the workqueue events are collected.Without the current patch, the antagonist process can force arbitrary delays between
workqueue_queue_workandworkqueue_execute_start, that in my tests were as high as30ms. With the current patch applied, the workqueue can be migrated to another unloaded CPU in the same node, and, keeping everything else equal, the maximum delay I could see was6us.
Quoted 3 lines
Fixes 0fe45467a1041 (idpf: add create vport and netdev configuration)Signed-off-by Marco Leogrande <[email protected]>Signed-off-by Manoj Vishwanathan <[email protected]>
Except the nit (s/praticular/particular) what Jake mentioned, changes look good to me.
Quoted 58 lines
--- drivers/net/ethernet/intel/idpf/idpf_main.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)drivers/net/ethernet/intel/idpf/idpf_main.c +10
-5@@ -174,7 +174,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)pci_set_master (pdev);pci_set_drvdata (pdev, adapter); - adapter->init_wq =alloc_workqueue ("%s-%s-init" ,0 ,0 , + adapter->init_wq =alloc_workqueue ("%s-%s-init" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->init_wq) {@@ -183,7 +184,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_free; } - adapter->serv_wq =alloc_workqueue ("%s-%s-service" ,0 ,0 , + adapter->serv_wq =alloc_workqueue ("%s-%s-service" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->serv_wq) {@@ -192,7 +194,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_serv_wq_alloc; } - adapter->mbx_wq =alloc_workqueue ("%s-%s-mbx" ,0 ,0 , + adapter->mbx_wq =alloc_workqueue ("%s-%s-mbx" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->mbx_wq) {@@ -201,7 +204,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_mbx_wq_alloc; } - adapter->stats_wq =alloc_workqueue ("%s-%s-stats" ,0 ,0 , + adapter->stats_wq =alloc_workqueue ("%s-%s-stats" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->stats_wq) {@@ -210,7 +214,8 @@ static int idpf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_stats_wq_alloc; } - adapter->vc_event_wq =alloc_workqueue ("%s-%s-vc_event" ,0 ,0 , + adapter->vc_event_wq =alloc_workqueue ("%s-%s-vc_event" , +WQ_UNBOUND |WQ_MEM_RECLAIM ,0 ,dev_driver_string (dev),dev_name (dev));if (!adapter->vc_event_wq) {
PATCH v2 1/1 v2 4/4] idpf: add more info during virtchnl transaction time out
In reply to Manoj VishwanathanAdd more information related to the transaction like cookie, vc_op, salt when transaction times out and include similar information when transaction salt does not match.
Info output for transaction timeout:
-------------------
(op:5015 cookie:45fe vc_op:5015 salt:45 timeout:60000ms)
-------------------
--- drivers/net/ethernet/intel/idpf/idpf_virtchnl.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +5 -4
@@ -517,8 +517,9 @@ static ssize_t idpf_vc_xn_exec(struct idpf_adapter *adapter, retval = -ENXIO ; goto only_unlock;case IDPF_VC_XN_WAITING : -dev_notice_ratelimited (&adapter->pdev->dev,"Transaction timed-out (op %d, %dms)\n" , - params->vc_op, params->timeout_ms); +dev_notice_ratelimited (&adapter->pdev->dev, +"Transaction timed-out (op:%d cookie:%04x vc_op:%d salt:%02x timeout:%dms)\n" , + params->vc_op, cookie, xn->vc_op, xn->salt, params->timeout_ms); retval = -ETIME ;break ;case IDPF_VC_XN_COMPLETED_SUCCESS :
@@ -615,8 +616,8 @@ idpf_vc_xn_forward_reply(struct idpf_adapter *adapter,idpf_vc_xn_lock (xn); salt =FIELD_GET (IDPF_VC_XN_SALT_M , msg_info);if (xn->salt != salt) { -dev_err_ratelimited (&adapter->pdev->dev,"Transaction salt does not match (%02x != %02x)\n" , - xn->salt, salt); +dev_err_ratelimited (&adapter->pdev->dev,"Transaction salt does not match (exp:%d@%02x(%d) != got:%d@%02x)\n" , + xn->vc_op, xn->salt, xn->state, ctlq_msg->cookie.mbx.chnl_opcode, salt);idpf_vc_xn_unlock (xn);return -EINVAL ; }
Signature · 1 line
2.46.0.295.g3b9ea8a38a-goog
Re: [Intel-wired-lan] [[PATCH v2 iwl-next] v2 4/4] idpf: add more info during virtchnl transaction time out
Quoted 11 lines · Manoj Vishwanathan
Add more information related to the transaction like cookie,
vc_op, salt when transaction times out and include similar information when transaction salt does not match.Info output for transaction timeout:
-------------------
(op:5015 cookie:45fe vc_op:5015 salt:45 timeout:60000ms)
-------------------Signed-off-by Manoj Vishwanathan <[email protected]>---
Re: [Intel-wired-lan] [[PATCH v2 iwl-next] v2 4/4] idpf: add more info during virtchnl transaction time out
In reply to Manoj VishwanathanQuoted 10 lines · Manoj Vishwanathan
Add more information related to the transaction like cookie,
vc_op, salt when transaction times out and include similar information when transaction salt does not match.Info output for transaction timeout:
-------------------
(op:5015 cookie:45fe vc_op:5015 salt:45 timeout:60000ms)
-------------------Signed-off-by Manoj Vishwanathan <[email protected]>
Quoted 31 lines
--- drivers/net/ethernet/intel/idpf/idpf_virtchnl.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +5
-4@@ -517,8 +517,9 @@ static ssize_t idpf_vc_xn_exec(struct idpf_adapter *adapter, retval = -ENXIO ; goto only_unlock;case IDPF_VC_XN_WAITING : -dev_notice_ratelimited (&adapter->pdev->dev,"Transaction timed-out (op %d, %dms)\n" , - params->vc_op, params->timeout_ms); +dev_notice_ratelimited (&adapter->pdev->dev, +"Transaction timed-out (op:%d cookie:%04x vc_op:%d salt:%02x timeout:%dms)\n" , + params->vc_op, cookie, xn->vc_op, xn->salt, params->timeout_ms); retval = -ETIME ;break ;case IDPF_VC_XN_COMPLETED_SUCCESS :@@ -615,8 +616,8 @@ idpf_vc_xn_forward_reply(struct idpf_adapter *adapter,idpf_vc_xn_lock (xn); salt =FIELD_GET (IDPF_VC_XN_SALT_M , msg_info);if (xn->salt != salt) { -dev_err_ratelimited (&adapter->pdev->dev,"Transaction salt does not match (%02x != %02x)\n" , - xn->salt, salt); +dev_err_ratelimited (&adapter->pdev->dev,"Transaction salt does not match (exp:%d@%02x(%d) != got:%d@%02x)\n" , + xn->vc_op, xn->salt, xn->state, ctlq_msg->cookie.mbx.chnl_opcode, salt);idpf_vc_xn_unlock (xn);return -EINVAL ; }
Tony Nguyen
maintainerQuoted 3 lines · Manoj Vishwanathan
IDPF Virtchnl: Enhance error reporting & fix locking/workqueue issues
This patch series addresses several IDPF virtchnl issues:
How are you sending these patches? Are you sending it all in one command/send? The threading is not showing up correctly in Patchwork, though lore does show it correctly.
Quoted 10 lines
* Improved error reporting for better diagnostics.
* Fixed locking sequence in virtchnl message handling to avoid potential race conditions.
* Converted idpf workqueues to unbound to prevent virtchnl processing delays under heavy load.Previously, CPU-bound kworkers for virtchnl processing could be starved,
leading to transaction timeouts and connection failures.
This was particularly problematic when IRQ traffic and user space processes contended for the same CPU.By making the workqueues unbound, we ensure virtchnl processing is not tied to a specific CPU, improving responsiveness even under high system load.
This should be broken up between bugs and improvements, one for iwl-net (fixes) and one for iwl-next (improvements). Patches 1-3 should go to iwl-net and 4 to iwl-next.
All the Fixes: are not in the correct format
e.g.
WARNING: Please use correct Fixes: style 'Fixes: <12 chars of sha1> ("<title line>")' - ie: 'Fixes: 34c21fa894a1 ("idpf: implement virtchnl transaction manager")'
They all have 1 extra char. Patches 1 & 3 are missing the '"' and patch is using the wrong '"' char.
Finally, what tree are you using for your patches? Some of the patches don't apply to the iwl tree or the netdev tree.
For IWL patches, please use these trees (dev-queue branch)
https://git.kernel.org/pub/scm/linux/kernel/git/tnguy/net-queue.git/
https://git.kernel.org/pub/scm/linux/kernel/git/tnguy/next-queue.git/
Thanks,
Tony
Quoted 22 lines
---
V2:
- Dropped patch from Willem
- RCS/RCT variable naming
- Improved commit message on feddback
v1: https://lore.kernel.org/netdev/[email protected]/T/David Decotigny (1):
idpf: address an rtnl lock splat in tx timeout recovery pathManoj Vishwanathan (2):
idpf: Acquire the lock before accessing thexn->salt
idpf: add more info during virtchnl transaction time outMarco Leogrande (1):
idpf: convert workqueues to unbounddrivers/net/ethernet/intel/idpf/idpf_main.c | 15 ++++++++++----- drivers/net/ethernet/intel/idpf/idpf_txrx.c | 14 +++++++++++++- drivers/net/ethernet/intel/idpf/idpf_virtchnl.c | 12 +++++++----- 3 files changed, 30 insertions(+), 11 deletions(-)