rcu: Use system_unbound_wq to avoid disturbing isolated CPUs
PATCH 1/1 rcu: Use system_unbound_wq to avoid disturbing isolated CPUs
It was discovered that isolated CPUs could sometimes be disturbed by kworkers processing kfree_rcu() works causing higher than expected latency. It is because the RCU core uses "system_wq" which doesn't have the WQ_UNBOUND flag to handle all its work items. Fix this violation of latency limits by using "system_unbound_wq" in the RCU core instead. This will ensure that those work items will not be run on CPUs marked as isolated.
Beside the WQ_UNBOUND flag, the other major difference between system_wq and system_unbound_wq is their max_active count. The system_unbound_wq has a max_active of WQ_MAX_ACTIVE (512) while system_wq's max_active is WQ_DFL_ACTIVE (256) which is half of WQ_MAX_ACTIVE.
--- kernel/rcu/tasks.h | 4 ++-- kernel/rcu/tree.c | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-)
kernel/rcu/tree.c +4 -4
@@ -3539,10 +3539,10 @@ schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp)if (delayed_work_pending (&krcp->monitor_work)) { delay_left = krcp->monitor_work.timer.expires - jiffies;if (delay < delay_left) -mod_delayed_work (system_wq, &krcp->monitor_work, delay); +mod_delayed_work (system_unbound_wq, &krcp->monitor_work, delay);return ; } -queue_delayed_work (system_wq, &krcp->monitor_work, delay); +queue_delayed_work (system_unbound_wq, &krcp->monitor_work, delay); } static void
@@ -3634,7 +3634,7 @@ static void kfree_rcu_monitor(struct work_struct *work)// be that the work is in the pending state when // channels have been detached following by each // other. -queue_rcu_work (system_wq, &krwp->rcu_work); +queue_rcu_work (system_unbound_wq, &krwp->rcu_work); } }
@@ -3704,7 +3704,7 @@ run_page_cache_worker(struct kfree_rcu_cpu *krcp)if (rcu_scheduler_active ==RCU_SCHEDULER_RUNNING && !atomic_xchg (&krcp->work_in_progress,1 )) {if (atomic_read (&krcp->backoff_page_cache_fill)) { -queue_delayed_work (system_wq, +queue_delayed_work (system_unbound_wq, &krcp->page_cache_work,msecs_to_jiffies (rcu_delay_page_cache_fill_msec)); }else {
Signature · 1 line
2.43.5
Quoted 59 lines · Waiman Long
It was discovered that isolated CPUs could sometimes be disturbed by kworkers processing
kfree_rcu()works causing higher than expected latency. It is because the RCU core uses "system_wq" which doesn't have theWQ_UNBOUNDflag to handle all its work items. Fix this violation of latency limits by using "system_unbound_wq" in the RCU core instead. This will ensure that those work items will not be run on CPUs marked as isolated.Beside the
WQ_UNBOUNDflag, the other major difference betweensystem_wqandsystem_unbound_wqis theirmax_activecount. Thesystem_unbound_wqhas amax_activeofWQ_MAX_ACTIVE(512) while system_wq'smax_activeisWQ_DFL_ACTIVE(256) which is half ofWQ_MAX_ACTIVE.Reported-by Vratislav Bendel <[email protected]>Signed-off-by Waiman Long <[email protected]>--- kernel/rcu/tasks.h | 4 ++-- kernel/rcu/tree.c | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-)kernel/rcu/tree.c +4
-4@@ -3539,10 +3539,10 @@ schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp)if (delayed_work_pending (&krcp->monitor_work)) { delay_left = krcp->monitor_work.timer.expires - jiffies;if (delay < delay_left) -mod_delayed_work (system_wq, &krcp->monitor_work, delay); +mod_delayed_work (system_unbound_wq, &krcp->monitor_work, delay);return ; } -queue_delayed_work (system_wq, &krcp->monitor_work, delay); +queue_delayed_work (system_unbound_wq, &krcp->monitor_work, delay); } static void@@ -3634,7 +3634,7 @@ static void kfree_rcu_monitor(struct work_struct *work)// be that the work is in the pending state when // channels have been detached following by each // other. -queue_rcu_work (system_wq, &krwp->rcu_work); +queue_rcu_work (system_unbound_wq, &krwp->rcu_work); } }@@ -3704,7 +3704,7 @@ run_page_cache_worker(struct kfree_rcu_cpu *krcp)if (rcu_scheduler_active ==RCU_SCHEDULER_RUNNING && !atomic_xchg (&krcp->work_in_progress,1 )) {if (atomic_read (&krcp->backoff_page_cache_fill)) { -queue_delayed_work (system_wq, +queue_delayed_work (system_unbound_wq, &krcp->page_cache_work,msecs_to_jiffies (rcu_delay_page_cache_fill_msec)); }else {Signature · 1 line
2.43.5
Thanks!
--
Uladzislau Rezki
Quoted 8 lines · Waiman Long
It was discovered that isolated CPUs could sometimes be disturbed by kworkers processing
kfree_rcu()works causing higher than expected latency. It is because the RCU core uses "system_wq" which doesn't have theWQ_UNBOUNDflag to handle all its work items. Fix this violation of latency limits by using "system_unbound_wq" in the RCU core instead. This will ensure that those work items will not be run on CPUs marked as isolated.
Quoted 6 lines
Beside the
WQ_UNBOUNDflag, the other major difference betweensystem_wqandsystem_unbound_wqis theirmax_activecount. Thesystem_unbound_wqhas amax_activeofWQ_MAX_ACTIVE(512) while system_wq'smax_activeisWQ_DFL_ACTIVE(256) which is half ofWQ_MAX_ACTIVE.Reported-by Vratislav Bendel <[email protected]>
I've seen this problem a while ago and reported to the list:
https://lore.kernel.org/all/[email protected]/
I've just applied this test, and run my workload for 2 hours without hitting this issue. Thanks for solving it.
Uladzislau Rezki
maintainerQuoted 16 lines · Breno Leitao
Quoted 8 lines · Waiman Long
It was discovered that isolated CPUs could sometimes be disturbed by kworkers processing
kfree_rcu()works causing higher than expected latency. It is because the RCU core uses "system_wq" which doesn't have theWQ_UNBOUNDflag to handle all its work items. Fix this violation of latency limits by using "system_unbound_wq" in the RCU core instead. This will ensure that those work items will not be run on CPUs marked as isolated.Quoted 6 lines
Beside the
WQ_UNBOUNDflag, the other major difference betweensystem_wqandsystem_unbound_wqis theirmax_activecount. Thesystem_unbound_wqhas amax_activeofWQ_MAX_ACTIVE(512) while system_wq'smax_activeisWQ_DFL_ACTIVE(256) which is half ofWQ_MAX_ACTIVE.Reported-by Vratislav Bendel <[email protected]>
Quoted 9 lines
I've seen this problem a while ago and reported to the list:
https://lore.kernel.org/all/[email protected]/I've just applied this test, and run my workload for 2 hours without hitting this issue. Thanks for solving it.
Tested-by Breno Leitao <[email protected]>
Thank you for testing this! I saw your recent email about that :)
--
Uladzislau Rezki
Quoted 15 lines · Breno Leitao
Quoted 8 lines · Waiman Long
It was discovered that isolated CPUs could sometimes be disturbed by kworkers processing
kfree_rcu()works causing higher than expected latency. It is because the RCU core uses "system_wq" which doesn't have theWQ_UNBOUNDflag to handle all its work items. Fix this violation of latency limits by using "system_unbound_wq" in the RCU core instead. This will ensure that those work items will not be run on CPUs marked as isolated.Quoted 6 lines
Beside the
WQ_UNBOUNDflag, the other major difference betweensystem_wqandsystem_unbound_wqis theirmax_activecount. Thesystem_unbound_wqhas amax_activeofWQ_MAX_ACTIVE(512) while system_wq'smax_activeisWQ_DFL_ACTIVE(256) which is half ofWQ_MAX_ACTIVE.Reported-by Vratislav Bendel <[email protected]>
Quoted 8 lines
I've seen this problem a while ago and reported to the list:
https://lore.kernel.org/all/[email protected]/I've just applied this test, and run my workload for 2 hours without hitting this issue. Thanks for solving it.
Tested-by Breno Leitao <[email protected]>
Thank for testing this patch. So it is just us that saw this problem.
Cheers,
Longman
Neeraj Upadhyay
maintainerQuoted 8 lines · Waiman Long
It was discovered that isolated CPUs could sometimes be disturbed by kworkers processing
kfree_rcu()works causing higher than expected latency. It is because the RCU core uses "system_wq" which doesn't have theWQ_UNBOUNDflag to handle all its work items. Fix this violation of latency limits by using "system_unbound_wq" in the RCU core instead. This will ensure that those work items will not be run on CPUs marked as isolated.
Alternative approach here could be, in case we want to keep per CPU worker pools, define a wq with WQ_CPU_INTENSIVE flag. Are there cases where WQ_CPU_INTENSIVE wq won't be sufficient for the problem this patch is fixing?
- Neeraj
Quoted 51 lines
Beside the
WQ_UNBOUNDflag, the other major difference betweensystem_wqandsystem_unbound_wqis theirmax_activecount. Thesystem_unbound_wqhas amax_activeofWQ_MAX_ACTIVE(512) while system_wq'smax_activeisWQ_DFL_ACTIVE(256) which is half ofWQ_MAX_ACTIVE.Reported-by Vratislav Bendel <[email protected]>Signed-off-by Waiman Long <[email protected]>--- kernel/rcu/tasks.h | 4 ++-- kernel/rcu/tree.c | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-)kernel/rcu/tree.c +4
-4@@ -3539,10 +3539,10 @@ schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp)if (delayed_work_pending (&krcp->monitor_work)) { delay_left = krcp->monitor_work.timer.expires - jiffies;if (delay < delay_left) -mod_delayed_work (system_wq, &krcp->monitor_work, delay); +mod_delayed_work (system_unbound_wq, &krcp->monitor_work, delay);return ; } -queue_delayed_work (system_wq, &krcp->monitor_work, delay); +queue_delayed_work (system_unbound_wq, &krcp->monitor_work, delay); } static void@@ -3634,7 +3634,7 @@ static void kfree_rcu_monitor(struct work_struct *work)// be that the work is in the pending state when // channels have been detached following by each // other. -queue_rcu_work (system_wq, &krwp->rcu_work); +queue_rcu_work (system_unbound_wq, &krwp->rcu_work); } }@@ -3704,7 +3704,7 @@ run_page_cache_worker(struct kfree_rcu_cpu *krcp)if (rcu_scheduler_active ==RCU_SCHEDULER_RUNNING && !atomic_xchg (&krcp->work_in_progress,1 )) {if (atomic_read (&krcp->backoff_page_cache_fill)) { -queue_delayed_work (system_wq, +queue_delayed_work (system_unbound_wq, &krcp->page_cache_work,msecs_to_jiffies (rcu_delay_page_cache_fill_msec)); }else {Signature · 1 line
2.43.5
Quoted 9 lines · Neeraj Upadhyay
Quoted 8 lines · Waiman Long
It was discovered that isolated CPUs could sometimes be disturbed by kworkers processing
kfree_rcu()works causing higher than expected latency. It is because the RCU core uses "system_wq" which doesn't have theWQ_UNBOUNDflag to handle all its work items. Fix this violation of latency limits by using "system_unbound_wq" in the RCU core instead. This will ensure that those work items will not be run on CPUs marked as isolated.
Quoted 4 lines
Alternative approach here could be, in case we want to keep per CPU worker pools, define a wq with
WQ_CPU_INTENSIVEflag. Are there cases whereWQ_CPU_INTENSIVEwq won't be sufficient for the problem this patch is fixing?
What exactly will we gain by defining a WQ_CPU_INTENSIVE workqueue? Or what will we lose by using system_unbound_wq? All the calls that are modified to use system_unbound_wq are using WORK_CPU_UNBOUND as their cpu. IOW, they doesn't care which CPUs are used to run the work items. The only downside I can see is the possible loss of some cache locality.
In fact, WQ_CPU_INTENSIVE can be considered a subset of WQ_UNBOUND. An WQ_UNBOUND workqueue will avoid using isolated CPUs, but not a WQ_CPU_INTENSIVE workqueue.
Cheers,
Longman
Neeraj Upadhyay
maintainerQuoted 15 lines · Waiman Long
Quoted 9 lines · Neeraj Upadhyay
Quoted 8 lines · Waiman Long
It was discovered that isolated CPUs could sometimes be disturbed by kworkers processing
kfree_rcu()works causing higher than expected latency. It is because the RCU core uses "system_wq" which doesn't have theWQ_UNBOUNDflag to handle all its work items. Fix this violation of latency limits by using "system_unbound_wq" in the RCU core instead. This will ensure that those work items will not be run on CPUs marked as isolated.Quoted 4 lines
Alternative approach here could be, in case we want to keep per CPU worker pools, define a wq with
WQ_CPU_INTENSIVEflag. Are there cases whereWQ_CPU_INTENSIVEwq won't be sufficient for the problem this patch is fixing?
Quoted 6 lines
What exactly will we gain by defining a
WQ_CPU_INTENSIVEworkqueue? Or what will we lose by usingsystem_unbound_wq? All the calls that are modified to usesystem_unbound_wqare usingWORK_CPU_UNBOUNDas their cpu. IOW, they doesn't care which CPUs are used to run the work items. The only downside I can see is the possible loss of some cache locality.
For the nohz_full case, where unbounded pool workers run only on housekeeping CPU (cpu0), if multiple other CPUs are queuing work, the execution of those works could get delayed. However, this should not generally happen as other CPUs would be mostly running in user mode.
Quoted 3 lines
In fact,
WQ_CPU_INTENSIVEcan be considered a subset ofWQ_UNBOUND. AnWQ_UNBOUNDworkqueue will avoid using isolated CPUs, but not aWQ_CPU_INTENSIVEworkqueue.
Got it, thanks!
I have picked the patch for further review and testing [1]
[1] https://git.kernel.org/pub/scm/linux/kernel/git/neeraj.upadhyay/linux-rcu.git/log/?h=next
- Neeraj
Quoted 5 lines
Cheers,
Longman
Quoted 21 lines · Neeraj Upadhyay
Quoted 14 lines · Waiman Long
Quoted 9 lines · Neeraj Upadhyay
Quoted 8 lines · Waiman Long
It was discovered that isolated CPUs could sometimes be disturbed by kworkers processing
kfree_rcu()works causing higher than expected latency. It is because the RCU core uses "system_wq" which doesn't have theWQ_UNBOUNDflag to handle all its work items. Fix this violation of latency limits by using "system_unbound_wq" in the RCU core instead. This will ensure that those work items will not be run on CPUs marked as isolated.Quoted 4 lines
Alternative approach here could be, in case we want to keep per CPU worker pools, define a wq with
WQ_CPU_INTENSIVEflag. Are there cases whereWQ_CPU_INTENSIVEwq won't be sufficient for the problem this patch is fixing?Quoted 6 lines
What exactly will we gain by defining a
WQ_CPU_INTENSIVEworkqueue? Or what will we lose by usingsystem_unbound_wq? All the calls that are modified to usesystem_unbound_wqare usingWORK_CPU_UNBOUNDas their cpu. IOW, they doesn't care which CPUs are used to run the work items. The only downside I can see is the possible loss of some cache locality.
Quoted 4 lines
For the
nohz_fullcase, where unbounded pool workers run only on housekeeping CPU (cpu0), if multiple other CPUs are queuing work, the execution of those works could get delayed. However, this should not generally happen as other CPUs would be mostly running in user mode.
Well, it there is only one housekeeping CPU, a lot of background kernel tasks will be slowed down. Users should be careful about the proper balance between the number of housekeeping and nohz-full CPUs.
Quoted 11 lines
Quoted 5 lines
In fact,
WQ_CPU_INTENSIVEcan be considered a subset ofWQ_UNBOUND. AnWQ_UNBOUNDworkqueue will avoid using isolated CPUs, but not aWQ_CPU_INTENSIVEworkqueue.Got it, thanks!
I have picked the patch for further review and testing [1]
[1] https://git.kernel.org/pub/scm/linux/kernel/git/neeraj.upadhyay/linux-rcu.git/log/?h=next
Thanks, let me know if you see any problem.
Cheers,
Longman