wq: Avoid using isolated cpus' timers on unbounded queue_delayed_work
RFC PATCH 1/1 wq: Avoid using isolated cpus' timers on unbounded queue_delayed_work
When __queue_delayed_work() is called with WORK_CPU_UNBOUND, it means any cpu is able to run the work, as well as any cpu timer is able to be used.
This is not good if a system does use CPU isolation, because it can take away some valuable cpu time to:
1 - deal with the timer interrupt,
2 - schedule-out the desired task,
3 - queue work on a random workqueue, and
4 - schedule the desired task back to the cpu.
So to fix this, during __queue_delayed_work(), if both:
- Work is not cpu-bounded,
- CPU isolation is in place,
then pick a random non-isolated cpu to use both the timer and the system per-cpu workqueue.
--- kernel/workqueue.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/workqueue.c +8 -0
@@ -1954,6 +1954,14 @@ static void __queue_delayed_work(int cpu, struct workqueue_struct *wq,return ; } +/* +* If the work is cpu-unbound, and cpu isolation is in place, only +* schedule use timers from housekeeping cpus. In favor of avoiding +* cacheline bouncing, run the WQ in the same cpu as the timer. +*/ +if (cpu ==WORK_CPU_UNBOUND &&housekeeping_enabled (HK_TYPE_TIMER )) + cpu =housekeeping_any_cpu (HK_TYPE_TIMER ); + dwork->wq = wq; dwork->cpu = cpu; timer->expires = jiffies + delay;
Signature · 1 line
2.43.0
Tejun Heo
maintainerkernel/workqueue.c static void __queue_delayed_work(int cpu, struct workqueue_struct *wq, · line 1963
Quoted 7 lines · Leonardo Bras
kernel/workqueue.c +7
-0+/* +* If the work is cpu-unbound, and cpu isolation is in place, only +* schedule use timers from housekeeping cpus. In favor of avoiding +* cacheline bouncing, run the WQ in the same cpu as the timer. +*/ +if (cpu ==WORK_CPU_UNBOUND &&housekeeping_enabled (HK_TYPE_TIMER )) + cpu =housekeeping_any_cpu (HK_TYPE_TIMER );
Would it make more sense to use wq_unbound_cpumask?
Thanks.
Signature · 1 line
tejun
Quoted 10 lines · Tejun Heo
kernel/workqueue.c static void __queue_delayed_work(int cpu, struct workqueue_struct *wq, · line 1963
Quoted 7 lines · Leonardo Bras
kernel/workqueue.c +7
-0+/* +* If the work is cpu-unbound, and cpu isolation is in place, only +* schedule use timers from housekeeping cpus. In favor of avoiding +* cacheline bouncing, run the WQ in the same cpu as the timer. +*/ +if (cpu ==WORK_CPU_UNBOUND &&housekeeping_enabled (HK_TYPE_TIMER )) + cpu =housekeeping_any_cpu (HK_TYPE_TIMER );Would it make more sense to use
wq_unbound_cpumask?
Hello Tejun, thank you for this reply!
That's a good suggestion, but looking at workqueue_init_early() I see that, in short:wq_unbound_cpumask = cpu_possible_mask &
housekeeping_cpumask(HK_TYPE_WQ) & housekeeping_cpumask(HK_TYPE_DOMAIN) & wq_cmdline_cpumask
So wq_unbound_cpumask relates to domain and workqueue cpu isolation.
In our case, we are using this to choose in which cpu is the timer we want to use, so it makes sense to use timer-related cpu isolation, instead.
As of today, your suggestion would work the same, as the only way to enable WQ cpu isolation is to use nohz_full, which also enables TIMER cpu isolation. But since that can change in the future, for any reason, I would suggest that we stick to using the HK_TYPE_TIMER cpumask.
I can now notice that this can end up introducing an issue: possibly running on a workqueue on a cpu outside of a valid wq_cmdline_cpumask.
I would suggest fixing this in a couple ways:
1 - We introduce a new cpumask which is basically
housekeeping_cpumask(HK_TYPE_DOMAIN) & wq_cmdline_cpumask, allowing us to keep the timer interrupt in the same cpu as the scheduled function,
2- We use the resulting cpu only to pick the right timer.
What are your thouhts on that?
Thank you!
Leo
Tejun Heo
maintainerHello,
Quoted 11 lines · Leonardo Bras
That's a good suggestion, but looking at
workqueue_init_early()I see that, in short:wq_unbound_cpumask=cpu_possible_mask&
housekeeping_cpumask(HK_TYPE_WQ)&housekeeping_cpumask(HK_TYPE_DOMAIN)&wq_cmdline_cpumaskSo
wq_unbound_cpumaskrelates to domain and workqueue cpu isolation.In our case, we are using this to choose in which cpu is the timer we want to use, so it makes sense to use timer-related cpu isolation, instead.
- In the proposed code, when cpu == WORK_CPU_UNBOUND, it's always setting cpu to housekeeping_any_cpu(HK_TYPE_TIMER). This may unnecessarily move the timer and task away from local CPU. Preferring the local CPU would likely make sense.
- If HK_TYPE_TIMER and workqueue masks may not agree, setting dwork->cpu to the one returned from HK_TYPE_TIMER is likely problematic. That would force __queue_work() to use that CPU instead of picking one from wq_unbound_cpumask.
Quoted 7 lines
As of today, your suggestion would work the same, as the only way to enable WQ cpu isolation is to use
nohz_full, which also enables TIMER cpu isolation. But since that can change in the future, for any reason, I would suggest that we stick to using theHK_TYPE_TIMERcpumask.I can now notice that this can end up introducing an issue: possibly running on a workqueue on a cpu outside of a valid
wq_cmdline_cpumask.
Yeap.
Quoted 7 lines
I would suggest fixing this in a couple ways:
1 - We introduce a new cpumask which is basically
housekeeping_cpumask(HK_TYPE_DOMAIN)&wq_cmdline_cpumask, allowing us to keep the timer interrupt in the same cpu as the scheduled function,
2- We use the resulting cpu only to pick the right timer.What are your thouhts on that?
How about something like the following instead?
- If current CPU is in HK_TYPE_TIMER, pick that CPU.
- If not, pick a CPU from HK_TYPE_TIMER.
- Do add_timer_on() on the selected CPU but leave dwork->cpu as WORK_CPU_UNBOUND and leave that part to __queue_work().
Thanks.
Signature · 1 line
tejun
Quoted 45 lines · Tejun Heo
Hello,
Quoted 11 lines · Leonardo Bras
That's a good suggestion, but looking at
workqueue_init_early()I see that, in short:wq_unbound_cpumask=cpu_possible_mask&
housekeeping_cpumask(HK_TYPE_WQ)&housekeeping_cpumask(HK_TYPE_DOMAIN)&wq_cmdline_cpumaskSo
wq_unbound_cpumaskrelates to domain and workqueue cpu isolation.In our case, we are using this to choose in which cpu is the timer we want to use, so it makes sense to use timer-related cpu isolation, instead.
- In the proposed code, when cpu ==
WORK_CPU_UNBOUND, it's always setting cpu tohousekeeping_any_cpu(HK_TYPE_TIMER). This may unnecessarily move the timer and task away from local CPU. Preferring the local CPU would likely make sense.- If
HK_TYPE_TIMERand workqueue masks may not agree, settingdwork->cputo the one returned fromHK_TYPE_TIMERis likely problematic. That would force__queue_work()to use that CPU instead of picking one fromwq_unbound_cpumask.Quoted 7 lines
As of today, your suggestion would work the same, as the only way to enable WQ cpu isolation is to use
nohz_full, which also enables TIMER cpu isolation. But since that can change in the future, for any reason, I would suggest that we stick to using theHK_TYPE_TIMERcpumask.I can now notice that this can end up introducing an issue: possibly running on a workqueue on a cpu outside of a valid
wq_cmdline_cpumask.Yeap.
Quoted 7 lines
I would suggest fixing this in a couple ways:
1 - We introduce a new cpumask which is basically
housekeeping_cpumask(HK_TYPE_DOMAIN)&wq_cmdline_cpumask, allowing us to keep the timer interrupt in the same cpu as the scheduled function,
2- We use the resulting cpu only to pick the right timer.What are your thouhts on that?
How about something like the following instead?
Quoted 8 lines
- If current CPU is in
HK_TYPE_TIMER, pick that CPU.- If not, pick a CPU from
HK_TYPE_TIMER.- Do
add_timer_on()on the selected CPU but leavedwork->cpuasWORK_CPU_UNBOUNDand leave that part to__queue_work().Thanks.
It looks like a good idea to me.
It's basicaly (2) with "keep the timer in this cpu if it's not isolated", which seems the right thing to do.
Thanks!
Leo
Quoted 4 lines
Signature · 2 lines
-- tejun