Skip to contentSkip to conversation

workqueue: Automatic affinity scope fallback for single-pod topologies

RFC PATCH Superseded Chuck Lever 2 participants 6 messages Feb 3 Workqueue

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 workqueue: Automatic affinity scope fallback for single-pod topologies

From: Chuck Lever <[email protected]>

The default affinity scope WQ_AFFN_CACHE assumes systems have multiple last-level caches. On systems where all CPUs share a single LLC (common with Intel monolithic dies), this scope degenerates to a single worker pool. All queue_work() calls then contend on that pool's single lock, causing severe performance degradation under high-throughput workloads.

For example, on a 12-core system with a single shared L3 cache running NFS over RDMA with 12 fio jobs, perf shows approximately 39% of CPU cycles spent in native_queued_spin_lock_slowpath, nearly all from __queue_work() contending on the single pool lock.

On such systems WQ_AFFN_CACHE, WQ_AFFN_SMT, and WQ_AFFN_NUMA scopes all collapse to a single pod.

Add wq_effective_affn_scope() to detect when a selected affinity scope provides only one pod despite having multiple CPUs, and automatically fall back to a finer-grained scope. This ensures reasonable lock distribution without requiring manual configuration via the workqueue.default_affinity_scope parameter or per-workqueue sysfs tuning.

The fallback is conservative: it triggers only when a scope degenerates to exactly one pod, and respects explicitly configured (non-default) scopes.

Also update wq_affn_scope_show() to display the effective scope when fallback occurs, making the behavior transparent to administrators via sysfs (e.g., "default (cache -> smt)").

---
 include/linux/workqueue.h |  7 ++++-
 kernel/workqueue.c        | 60 +++++++++++++++++++++++++++++++++++----
 2 files changed, 60 insertions(+), 7 deletions(-)
include/linux/workqueue.h +6 -1
index dabc351cc127..130c452fcecf 100644
@@ -128,10 +128,15 @@ struct rcu_work {
 	struct workqueue_struct *wq;
 };
 
+/*
+ * Affinity scopes are ordered from finest to coarsest granularity. This
+ * ordering is used by the automatic fallback logic in wq_effective_affn_scope()
+ * which walks from coarse toward fine when a scope degenerates to a single pod.
+ */
 enum wq_affn_scope {
 	WQ_AFFN_DFL,			/* use system default */
 	WQ_AFFN_CPU,			/* one pod per CPU */
-	WQ_AFFN_SMT,			/* one pod poer SMT */
+	WQ_AFFN_SMT,			/* one pod per SMT */
 	WQ_AFFN_CACHE,			/* one pod per LLC */
 	WQ_AFFN_NUMA,			/* one pod per NUMA node */
 	WQ_AFFN_SYSTEM,			/* one pod across the whole system */
kernel/workqueue.c +54 -6

84 lines not on this page · raw message

Signature · 1 line
2.52.0
maintainer
#1 lore raw

Hello,

Quoted 2 lines · Chuck Lever

On such systems WQ_AFFN_CACHE, WQ_AFFN_SMT, and WQ_AFFN_NUMA scopes all collapse to a single pod.

WQ_AFFN_SMT should be on CPU core boundaries, right?

Quoted 10 lines

Add wq_effective_affn_scope() to detect when a selected affinity scope provides only one pod despite having multiple CPUs, and automatically fall back to a finer-grained scope. This ensures reasonable lock distribution without requiring manual configuration via the workqueue.default_affinity_scope parameter or per-workqueue sysfs tuning.

The fallback is conservative: it triggers only when a scope degenerates to exactly one pod, and respects explicitly configured (non-default) scopes.

While I understand the problem, I don't think dropping down to core boundary for unbound workqueues by default makes sense. That may help with some use cases but cause problem with others. Given that WQ_AFFN_CACHE is the same as WQ_AFFN_NUMA on these machines, maybe we can shard it automatically according to some heuristics or maybe we can introduce another affinity level between CACHE and SMT which is sharded on machines with too many CPUs in a single cache domain.

Thanks.

Signature · 1 line
tejun
#2 lore raw
Quoted 15 lines · Tejun Heo

Hello,

Quoted 2 lines · Chuck Lever

On such systems WQ_AFFN_CACHE, WQ_AFFN_SMT, and WQ_AFFN_NUMA scopes all collapse to a single pod.

WQ_AFFN_SMT should be on CPU core boundaries, right?

Quoted 7 lines

Add wq_effective_affn_scope() to detect when a selected affinity scope provides only one pod despite having multiple CPUs, and automatically fall back to a finer-grained scope. This ensures reasonable lock distribution without requiring manual configuration via the workqueue.default_affinity_scope parameter or per-workqueue sysfs tuning.

Quoted 7 lines
Quoted 3 lines

The fallback is conservative: it triggers only when a scope degenerates to exactly one pod, and respects explicitly configured (non-default) scopes.

While I understand the problem, I don't think dropping down to core boundary for unbound workqueues by default makes sense. That may help with some use cases but cause problem with others.

I've never seen a case where it doesn't help. In order to craft an alternative, I'll need to have some examples to avoid. Is it only the SMT case that is concerning?

Quoted 5 lines

Given that WQ_AFFN_CACHE is the same as
WQ_AFFN_NUMA on these machines, maybe we can shard it automatically according to some heuristics or maybe we can introduce another affinity level between CACHE and SMT which is sharded on machines with too many CPUs in a single cache domain.

Signature · 1 line
Chuck Lever
maintainer
#3 lore raw
Quoted 7 lines · Chuck Lever
Quoted 3 lines

While I understand the problem, I don't think dropping down to core boundary for unbound workqueues by default makes sense. That may help with some use cases but cause problem with others.

I've never seen a case where it doesn't help. In order to craft an alternative, I'll need to have some examples to avoid. Is it only the SMT case that is concerning?

It's just a lot of separate pools on large machines. If you have relatively high concurrency, the number of workers can go pretty high. They'd also migrate back and forth more depending on usage pattern and have worse cache locality. Imagine you have a bursty workload wandering through the system, if you have nr_cores pools, it can easily end up with kworkers > nr_cores * max_concurrency.

Thanks.

Signature · 1 line
tejun
#4 lore raw
Quoted 9 lines · Tejun Heo
Quoted 7 lines · Chuck Lever
Quoted 3 lines

While I understand the problem, I don't think dropping down to core boundary for unbound workqueues by default makes sense. That may help with some use cases but cause problem with others.

I've never seen a case where it doesn't help. In order to craft an alternative, I'll need to have some examples to avoid. Is it only the SMT case that is concerning?

Quoted 6 lines

It's just a lot of separate pools on large machines. If you have relatively high concurrency, the number of workers can go pretty high. They'd also migrate back and forth more depending on usage pattern and have worse cache locality. Imagine you have a bursty workload wandering through the system, if you have nr_cores pools, it can easily end up with kworkers > nr_cores * max_concurrency.

The patch addresses that, I'd hope, by only switching to per-CPU on single pod (ie, simple) systems. Larger, more complicated, topologies should be left unchanged. I imagine that on a single pod machine with a large number of cores, having per-CPU locking will nearly always be a win.

Signature · 1 line
Chuck Lever
maintainer
#5 lore raw

Hello,

Quoted 5 lines · Chuck Lever

The patch addresses that, I'd hope, by only switching to per-CPU on single pod (ie, simple) systems. Larger, more complicated, topologies should be left unchanged. I imagine that on a single pod machine with a large number of cores, having per-CPU locking will nearly always be a win.

Oh, I mean, unfortunately, intel produces chips with a lot of CPUs on a single L3 cache. e.g. Recent intel chips have upto 128 cores per socket and each socket still presents as a single L3 cache domain, so falling back to AFFN_SMT would mean that all unbound workqueues by default would be backed by 128 pools per socket. Create some hundreds of threads per pool and then now you end up with hundreds of thousands of kworkers.

Thanks.

Signature · 1 line
tejun