workqueue: Automatic affinity scope fallback for single-pod topologies
RFC PATCH 1/1 workqueue: Automatic affinity scope fallback for single-pod topologies
From: Chuck Lever <[email protected]>
The default affinity scope WQ_AFFN_CACHE assumes systems have multiple last-level caches. On systems where all CPUs share a single LLC (common with Intel monolithic dies), this scope degenerates to a single worker pool. All queue_work() calls then contend on that pool's single lock, causing severe performance degradation under high-throughput workloads.
For example, on a 12-core system with a single shared L3 cache running NFS over RDMA with 12 fio jobs, perf shows approximately 39% of CPU cycles spent in native_queued_spin_lock_slowpath, nearly all from __queue_work() contending on the single pool lock.
On such systems WQ_AFFN_CACHE, WQ_AFFN_SMT, and WQ_AFFN_NUMA scopes all collapse to a single pod.
Add wq_effective_affn_scope() to detect when a selected affinity scope provides only one pod despite having multiple CPUs, and automatically fall back to a finer-grained scope. This ensures reasonable lock distribution without requiring manual configuration via the workqueue.default_affinity_scope parameter or per-workqueue sysfs tuning.
The fallback is conservative: it triggers only when a scope degenerates to exactly one pod, and respects explicitly configured (non-default) scopes.
Also update wq_affn_scope_show() to display the effective scope when fallback occurs, making the behavior transparent to administrators via sysfs (e.g., "default (cache -> smt)").
--- include/linux/workqueue.h | 7 ++++- kernel/workqueue.c | 60 +++++++++++++++++++++++++++++++++++---- 2 files changed, 60 insertions(+), 7 deletions(-)
include/linux/workqueue.h +6 -1
@@ -128,10 +128,15 @@ struct rcu_work {struct workqueue_struct *wq; }; +/* +* Affinity scopes are ordered from finest to coarsest granularity. This +* ordering is used by the automatic fallback logic in wq_effective_affn_scope() +* which walks from coarse toward fine when a scope degenerates to a single pod. +*/ enum wq_affn_scope {WQ_AFFN_DFL ,/* use system default */ WQ_AFFN_CPU ,/* one pod per CPU */ -WQ_AFFN_SMT ,/* one pod poer SMT */ +WQ_AFFN_SMT ,/* one pod per SMT */ WQ_AFFN_CACHE ,/* one pod per LLC */ WQ_AFFN_NUMA ,/* one pod per NUMA node */ WQ_AFFN_SYSTEM ,/* one pod across the whole system */
kernel/workqueue.c +54 -6
84 lines not on this page · raw message
Signature · 1 line
2.52.0
Tejun Heo
maintainerHello,
Quoted 2 lines · Chuck Lever
On such systems
WQ_AFFN_CACHE,WQ_AFFN_SMT, andWQ_AFFN_NUMAscopes all collapse to a single pod.
WQ_AFFN_SMT should be on CPU core boundaries, right?
Quoted 10 lines
Add
wq_effective_affn_scope()to detect when a selected affinity scope provides only one pod despite having multiple CPUs, and automatically fall back to a finer-grained scope. This ensures reasonable lock distribution without requiring manual configuration via the workqueue.default_affinity_scope parameter or per-workqueue sysfs tuning.The fallback is conservative: it triggers only when a scope degenerates to exactly one pod, and respects explicitly configured (non-default) scopes.
While I understand the problem, I don't think dropping down to core boundary for unbound workqueues by default makes sense. That may help with some use cases but cause problem with others. Given that WQ_AFFN_CACHE is the same as WQ_AFFN_NUMA on these machines, maybe we can shard it automatically according to some heuristics or maybe we can introduce another affinity level between CACHE and SMT which is sharded on machines with too many CPUs in a single cache domain.
Thanks.
Signature · 1 line
tejun
Quoted 15 lines · Tejun Heo
Hello,
Quoted 2 lines · Chuck Lever
On such systems
WQ_AFFN_CACHE,WQ_AFFN_SMT, andWQ_AFFN_NUMAscopes all collapse to a single pod.
WQ_AFFN_SMTshould be on CPU core boundaries, right?Quoted 7 lines
Add
wq_effective_affn_scope()to detect when a selected affinity scope provides only one pod despite having multiple CPUs, and automatically fall back to a finer-grained scope. This ensures reasonable lock distribution without requiring manual configuration via the workqueue.default_affinity_scope parameter or per-workqueue sysfs tuning.
Quoted 7 lines
Quoted 3 lines
The fallback is conservative: it triggers only when a scope degenerates to exactly one pod, and respects explicitly configured (non-default) scopes.
While I understand the problem, I don't think dropping down to core boundary for unbound workqueues by default makes sense. That may help with some use cases but cause problem with others.
I've never seen a case where it doesn't help. In order to craft an alternative, I'll need to have some examples to avoid. Is it only the SMT case that is concerning?
Quoted 5 lines
Given that
WQ_AFFN_CACHEis the same asWQ_AFFN_NUMAon these machines, maybe we can shard it automatically according to some heuristics or maybe we can introduce another affinity level between CACHE and SMT which is sharded on machines with too many CPUs in a single cache domain.
Signature · 1 line
Chuck Lever
Tejun Heo
maintainerQuoted 7 lines · Chuck Lever
Quoted 3 lines
While I understand the problem, I don't think dropping down to core boundary for unbound workqueues by default makes sense. That may help with some use cases but cause problem with others.
I've never seen a case where it doesn't help. In order to craft an alternative, I'll need to have some examples to avoid. Is it only the SMT case that is concerning?
It's just a lot of separate pools on large machines. If you have relatively high concurrency, the number of workers can go pretty high. They'd also migrate back and forth more depending on usage pattern and have worse cache locality. Imagine you have a bursty workload wandering through the system, if you have nr_cores pools, it can easily end up with kworkers > nr_cores * max_concurrency.
Thanks.
Signature · 1 line
tejun
Quoted 9 lines · Tejun Heo
Quoted 7 lines · Chuck Lever
Quoted 3 lines
While I understand the problem, I don't think dropping down to core boundary for unbound workqueues by default makes sense. That may help with some use cases but cause problem with others.
I've never seen a case where it doesn't help. In order to craft an alternative, I'll need to have some examples to avoid. Is it only the SMT case that is concerning?
Quoted 6 lines
It's just a lot of separate pools on large machines. If you have relatively high concurrency, the number of workers can go pretty high. They'd also migrate back and forth more depending on usage pattern and have worse cache locality. Imagine you have a bursty workload wandering through the system, if you have
nr_corespools, it can easily end up with kworkers >nr_cores*max_concurrency.
The patch addresses that, I'd hope, by only switching to per-CPU on single pod (ie, simple) systems. Larger, more complicated, topologies should be left unchanged. I imagine that on a single pod machine with a large number of cores, having per-CPU locking will nearly always be a win.
Signature · 1 line
Chuck Lever
Tejun Heo
maintainerHello,
Quoted 5 lines · Chuck Lever
The patch addresses that, I'd hope, by only switching to per-CPU on single pod (ie, simple) systems. Larger, more complicated, topologies should be left unchanged. I imagine that on a single pod machine with a large number of cores, having per-CPU locking will nearly always be a win.
Oh, I mean, unfortunately, intel produces chips with a lot of CPUs on a single L3 cache. e.g. Recent intel chips have upto 128 cores per socket and each socket still presents as a single L3 cache domain, so falling back to AFFN_SMT would mean that all unbound workqueues by default would be backed by 128 pools per socket. Create some hundreds of threads per pool and then now you end up with hundreds of thousands of kworkers.
Thanks.
Signature · 1 line
tejun