Skip to contentSkip to conversation

workqueue: Automatic affinity scope fallback for single-pod topologies

PATCH In review Chuck Lever 2 participants 3 messages Feb 5 Workqueue

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 workqueue: Automatic affinity scope fallback for single-pod topologies

From: Chuck Lever <[email protected]>

The default affinity scope WQ_AFFN_CACHE assumes systems have multiple last-level caches. On systems where all CPUs share a single LLC (common with Intel monolithic dies), this scope degenerates to a single worker pool. All queue_work() calls then contend on that pool's single lock, causing severe performance degradation under high-throughput workloads.

For example, on a 12-core system with a single shared L3 cache running NFS over RDMA with 12 fio jobs, perf shows approximately 39% of CPU cycles spent in native_queued_spin_lock_slowpath, nearly all from __queue_work() contending on the single pool lock.

On such systems WQ_AFFN_CACHE, WQ_AFFN_SMT, and WQ_AFFN_NUMA scopes all collapse to a single pod.

Add wq_effective_affn_scope() to detect when a selected affinity scope provides only one pod despite having multiple CPUs, and automatically fall back to a finer-grained scope. This enables lock distribution to scale with CPU count without requiring manual configuration via the workqueue.default_affinity_scope parameter or per-workqueue sysfs tuning.

The fallback is conservative: it triggers only when a scope degenerates to exactly one pod, and respects explicitly configured (non-default) scopes.

Also update wq_affn_scope_show() to display the effective scope when fallback occurs, making the behavior transparent to administrators via sysfs (e.g., "default (cache -> smt)").

---

Changes since RFC:
- Add a new affinity scope between CPU and CACHE

 include/linux/workqueue.h |  8 ++++-
 kernel/workqueue.c        | 68 +++++++++++++++++++++++++++++++++++----
 2 files changed, 69 insertions(+), 7 deletions(-)
include/linux/workqueue.h +7 -1
index dabc351cc127..1fca5791337d 100644
@@ -128,10 +128,16 @@ struct rcu_work {
 	struct workqueue_struct *wq;
 };
 
+/*
+ * Affinity scopes are ordered from finest to coarsest granularity. This
+ * ordering is used by the automatic fallback logic in wq_effective_affn_scope()
+ * which walks from coarse toward fine when a scope degenerates to a single pod.
+ */
 enum wq_affn_scope {
 	WQ_AFFN_DFL,			/* use system default */
 	WQ_AFFN_CPU,			/* one pod per CPU */
-	WQ_AFFN_SMT,			/* one pod poer SMT */
+	WQ_AFFN_SMT,			/* one pod per SMT */
+	WQ_AFFN_CLUSTER,		/* one pod per cluster */
 	WQ_AFFN_CACHE,			/* one pod per LLC */
 	WQ_AFFN_NUMA,			/* one pod per NUMA node */
 	WQ_AFFN_SYSTEM,			/* one pod across the whole system */
kernel/workqueue.c +62 -6

110 lines not on this page · raw message

Signature · 1 line
2.52.0
maintainer
#1 lore raw

Hello, Chuck.

kernel/workqueue.c static bool __init cpus_share_smt(int cpu0, int cpu1) · line 8079

Quoted 4 lines · Chuck Lever
kernel/workqueue.c +4 -0
+static bool __init cpus_share_cluster(int cpu0, int cpu1)
+{
+	return cpumask_test_cpu(cpu0, topology_cluster_cpumask(cpu1));
+}

Cluster boundary == core boundary for a lot of CPUs. I don't think this is going to work. Here are a couple options:

- Introduce an affinity level which splits CACHE according to some adjustable heuristics.

- Make the NFS workqueue default to WQ_AFFN_CORE (or maybe switch based on some heuristics) or switch to a per-cpu workqueue.

Thanks.

Signature · 1 line
tejun
#2 lore raw
Quoted 10 lines · Tejun Heo

Hello, Chuck.

kernel/workqueue.c static bool __init cpus_share_smt(int cpu0, int cpu1) · line 8079

Quoted 4 lines · Chuck Lever
kernel/workqueue.c +4 -0
+static bool __init cpus_share_cluster(int cpu0, int cpu1)
+{
+	return cpumask_test_cpu(cpu0, topology_cluster_cpumask(cpu1));
+}

Cluster boundary == core boundary for a lot of CPUs. I don't think this is going to work.

Fair enough; WQ_AFFN_CLUSTER is not a reliable intermediate level. On x86 cpu_clustergroup_mask() returns cpu_l2c_shared_mask(), which is per-core on many chips. The arm64 cpu_clustergroup_mask() has a similar collapse: when cluster_sibling spans the coregroup, it falls back to SMT siblings. And the generic fallback in topology.h is cpumask_of(cpu).

I was hoping it would be a proper intermediate sharding scope.

Quoted 6 lines

Here are a couple options:>
- Introduce an affinity level which splits CACHE according to some adjustable heuristics.

- Make the NFS workqueue default to WQ_AFFN_CORE (or maybe switch based on some heuristics) or switch to a per-cpu workqueue.

The issue I see is that the contention isn't confined to a single workqueue. In the NFS-over-RDMA I/O paths, at least four unbound workqueues are in the hot path:

- rpciod (WQ_UNBOUND) in net/sunrpc/sched.c -- core RPC task wake-up on every completion
- xprtiod (WQ_UNBOUND) in net/sunrpc/xprt.c -- transport cleanup and receive processing
- nfsiod (WQ_UNBOUND) in fs/nfs/inode.c -- direct write and local I/O completion
- svcrdma_wq (WQ_UNBOUND) in svc_rdma.c -- send context and write info release on every RDMA completion

These span three subsystems and maintainers. Other RDMA ULPs (iSER, SRP target, kSMBd) have their own unbound workqueues with the same exposure. Tuning each one individually is fragile, and any new WQ_UNBOUND workqueue added to these paths inherits the degenerate default behavior.

Even on platforms with a large core-to-pod ratio, pool lock contention is going to be a significant problem when WQ pools are shared by more than a handful of cores.

I don't have access to the kind of hardware needed to deeply test sharding ideas, so I'll drop this patch for now and simply set boot command line options on all my systems.

Signature · 1 line
Chuck Lever