workqueue: Introduce a sharded cache affinity scope
Subsystems (6)
v3 cover workqueue: Introduce a sharded cache affinity scope
TL;DR: Some modern processors have many CPUs per LLC (L3 cache), and unbound workqueues using the default affinity (WQ_AFFN_CACHE) collapse to a single worker pool, causing heavy spinlock (pool->lock) contention. Create a new affinity (WQ_AFFN_CACHE_SHARD) that caps each pool at wq_cache_shard_size CPUs (default 8).
Changes from RFC:
* wq_cache_shard_size is in terms of cores (not vCPU). So, wq_cache_shard_size=8 means the pool will have 8 cores and their siblings, like 16 threads/CPUs if SMT=1
* Got more data:
- AMD EPYC: All means are within ~1 stdev of zero. The deltas are indistinguishable from noise. Shard scoping has no measurable effect regardless of shard size. This is justified due to AMD EPYC having 11 L3 domains and lock contention not being a problem:
- ARM: A strong, consistent signal. At shard sizes 8 and 16 the mean improvement is ~7% with relatively tight stdev (~1–2%) at write, meaning the gain is real and reproducible across all IO engines. Even shard size 4 shows a solid +3.5% with the tightest stdev (0.97%).
Reads: Small shard sizes (2, 4) show a slight regression of ~1.3–1.7% (low stdev, so consistent). Larger shard sizes (8, 16) flip to a modest +1.4% gain, though shard_size=8 reads have high variance (stdev 2.79%) driven by a single outliers (which seems noise)
- Sweet spot: Shard size 8 to 16 offers the best overall profile — highest write gain (6.95%) with the lowest write stdev (1.18%), plus a consistent read gain (1.42%, stdev 0.70%), no impact on AMD/x86.
* ARM (NVIDIA Grace - Neoverse V2 - single L3 domain: CPUs 0-71)
┌────────────┬────────────┬─────────────┬───────────┬────────────┐ │ Shard Size │ Write Mean │ Write StDev │ Read Mean │ Read StDev │ ├────────────┼────────────┼─────────────┼───────────┼────────────┤ │ 2 │ +0.75% │ 1.32% │ -1.28% │ 0.45% │ ├────────────┼────────────┼─────────────┼───────────┼────────────┤ │ 4 │ +3.45% │ 0.97% │ -1.73% │ 0.52% │ ├────────────┼────────────┼─────────────┼───────────┼────────────┤ │ 8 │ +6.72% │ 1.97% │ +1.38% │ 2.79% │ ├────────────┼────────────┼─────────────┼───────────┼────────────┤ │ 16 │ +6.95% │ 1.18% │ +1.42% │ 0.70% │ └────────────┴────────────┴─────────────┴───────────┴────────────┘
* x86 (AMD EPYC 9D64 88-Core Processor - 11 L3 domains, 8 Cores / 16 vCPUs each)
┌────────────┬────────────┬─────────────┬───────────┬────────────┐ │ Shard Size │ Write Mean │ Write StDev │ Read Mean │ Read StDev │ ├────────────┼────────────┼─────────────┼───────────┼────────────┤ │ 2 │ +3.22% │ 1.90% │ -0.08% │ 0.72% │ ├────────────┼────────────┼─────────────┼───────────┼────────────┤ │ 4 │ +0.92% │ 1.59% │ +0.67% │ 2.33% │ ├────────────┼────────────┼─────────────┼───────────┼────────────┤ │ 8 │ +1.75% │ 1.47% │ -0.42% │ 0.72% │ ├────────────┼────────────┼─────────────┼───────────┼────────────┤ │ 16 │ +1.22% │ 1.72% │ +0.43% │ 1.32% │ └────────────┴────────────┴─────────────┴───────────┴────────────┘
---
Changes in v3:
- Precomputed the shards to avoid exponential time when creating the
pool. (Tejun)
- Add documentation about the new cache sharding affinity.
- Fixed use-after-free on module unload (on the selftest)
- Link to v2: https://patch.msgid.link/[email protected]
Changes in v2:
- wq_cache_shard_size is in terms of cores (not vCPU)
- Link to v1: https://patch.msgid.link/[email protected]
---
Breno Leitao (6):
workqueue: fix typo in WQ_AFFN_SMT comment
workqueue: add WQ_AFFN_CACHE_SHARD affinity scope
workqueue: set WQ_AFFN_CACHE_SHARD as the default affinity scope tools/workqueue: add CACHE_SHARD support to wq_dump.py
workqueue: add test_workqueue benchmark module
docs: workqueue: document WQ_AFFN_CACHE_SHARD affinity scope
Documentation/admin-guide/kernel-parameters.txt | 3 +- Documentation/core-api/workqueue.rst | 14 +- include/linux/workqueue.h | 3 +- kernel/workqueue.c | 185 ++++++++++++++- lib/Kconfig.debug | 10 + lib/Makefile | 1 + lib/test_workqueue.c | 294 ++++++++++++++++++++++++ tools/workqueue/wq_dump.py | 3 +- 8 files changed, 505 insertions(+), 8 deletions(-)
---
base-commit: 0e4f8f1a3d081e834be5fd0a62bdb2554fadd307
change-id: 20260309-workqueue_sharded-2327956e889b
Best regards,
Signature · 1 line
Breno Leitao <[email protected]>
PATCH v3 1/6 workqueue: fix typo in WQ_AFFN_SMT comment
Fix "poer" -> "per" in the WQ_AFFN_SMT enum comment.
--- include/linux/workqueue.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/workqueue.h +1 -1
@@ -131,7 +131,7 @@ struct rcu_work {enum wq_affn_scope {WQ_AFFN_DFL ,/* use system default */ WQ_AFFN_CPU ,/* one pod per CPU */ -WQ_AFFN_SMT ,/* one pod poer SMT */ +WQ_AFFN_SMT ,/* one pod per SMT */ WQ_AFFN_CACHE ,/* one pod per LLC */ WQ_AFFN_NUMA ,/* one pod per NUMA node */ WQ_AFFN_SYSTEM ,/* one pod across the whole system */
Signature · 1 line
2.52.0
PATCH v3 2/6 workqueue: add WQ_AFFN_CACHE_SHARD affinity scope
In reply to Breno LeitaoOn systems where many CPUs share one LLC, unbound workqueues using WQ_AFFN_CACHE collapse to a single worker pool, causing heavy spinlock contention on pool->lock. For example, Chuck Lever measured 39% of cycles lost to native_queued_spin_lock_slowpath on a 12-core shared-L3 NFS-over-RDMA system.
The existing affinity hierarchy (cpu, smt, cache, numa, system) offers no intermediate option between per-LLC and per-SMT-core granularity.
Add WQ_AFFN_CACHE_SHARD, which subdivides each LLC into groups of at most wq_cache_shard_size cores (default 8, tunable via boot parameter). Shards are always split on core (SMT group) boundaries so that Hyper-Threading siblings are never placed in different pods. Cores are distributed across shards as evenly as possible -- for example, 36 cores in a single LLC with max shard size 8 produces 5 shards of 8+7+7+7+7 cores.
The implementation follows the same comparator pattern as other affinity
scopes: precompute_cache_shard_ids() pre-fills the cpu_shard_id[] array from the already-initialized WQ_AFFN_CACHE and WQ_AFFN_SMT topology, and cpus_share_cache_shard() is passed to init_pod_type().
Benchmark on NVIDIA Grace (72 CPUs, single LLC, 50k items/thread), show cache_shard delivers ~5x the throughput and ~6.5x lower p50 latency compared to cache scope on this 72-core single-LLC system.
--- include/linux/workqueue.h | 1 + kernel/workqueue.c | 183 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 184 insertions(+)
include/linux/workqueue.h +1 -0
@@ -133,6 +133,7 @@ enum wq_affn_scope {WQ_AFFN_CPU ,/* one pod per CPU */ WQ_AFFN_SMT ,/* one pod per SMT */ WQ_AFFN_CACHE ,/* one pod per LLC */ +WQ_AFFN_CACHE_SHARD ,/* synthetic sub-LLC shards */ WQ_AFFN_NUMA ,/* one pod per NUMA node */ WQ_AFFN_SYSTEM, /* one pod across the whole system */
kernel/workqueue.c +183 -0
213 lines not on this page · raw message
Signature · 1 line
2.52.0
PATCH v3 3/6 workqueue: set WQ_AFFN_CACHE_SHARD as the default affinity scope
In reply to Breno LeitaoSet WQ_AFFN_CACHE_SHARD as the default affinity scope for unbound workqueues. On systems where many CPUs share one LLC, the previous default (WQ_AFFN_CACHE) collapses all CPUs to a single worker pool, causing heavy spinlock contention on pool->lock.
WQ_AFFN_CACHE_SHARD subdivides each LLC into smaller groups, providing a better balance between locality and contention. Users can revert to the previous behavior with workqueue.default_affinity_scope=cache.
On systems with 8 or fewer cores per LLC, CACHE_SHARD produces a single shard covering the entire LLC, making it functionally identical to the previous CACHE default. The sharding only activates when an LLC has more than 8 cores.
--- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/workqueue.c +1 -1
@@ -449,7 +449,7 @@ static bool wq_topo_initialized __read_mostly = false;static struct kmem_cache *pwq_cache;static struct wq_pod_type wq_pod_types[WQ_AFFN_NR_TYPES ]; -static enum wq_affn_scope wq_affn_dfl =WQ_AFFN_CACHE ; +static enum wq_affn_scope wq_affn_dfl =WQ_AFFN_CACHE_SHARD ;/* buf for wq_update_unbound_pod_attrs(), protected by CPU hotplug exclusion */ static struct workqueue_attrs *unbound_wq_update_pwq_attrs_buf;
Signature · 1 line
2.52.0
PATCH v3 4/6 tools/workqueue: add CACHE_SHARD support to wq_dump.py
In reply to Breno LeitaoThe WQ_AFFN_CACHE_SHARD affinity scope was added to the kernel but wq_dump.py was not updated to enumerate it. Add the missing constant lookup and include it in the affinity scopes iteration so that drgn output shows the CACHE_SHARD pod topology alongside the other scopes.
--- tools/workqueue/wq_dump.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tools/workqueue/wq_dump.py +2 -1
@@ -107,6 +107,7 @@ WQ_MEM_RECLAIM = prog['WQ_MEM_RECLAIM'] WQ_AFFN_CPU = prog['WQ_AFFN_CPU'] WQ_AFFN_SMT = prog['WQ_AFFN_SMT'] WQ_AFFN_CACHE = prog['WQ_AFFN_CACHE'] +WQ_AFFN_CACHE_SHARD = prog['WQ_AFFN_CACHE_SHARD'] WQ_AFFN_NUMA = prog['WQ_AFFN_NUMA'] WQ_AFFN_SYSTEM = prog['WQ_AFFN_SYSTEM']
@@ -138,7 +139,7 @@ def print_pod_type(pt): print(f' [{cpu}]={pt.cpu_pod[cpu].value_()}', end='') print('') -for affn in [WQ_AFFN_CPU, WQ_AFFN_SMT, WQ_AFFN_CACHE, WQ_AFFN_NUMA, WQ_AFFN_SYSTEM]: +for affn in [WQ_AFFN_CPU, WQ_AFFN_SMT, WQ_AFFN_CACHE, WQ_AFFN_CACHE_SHARD, WQ_AFFN_NUMA, WQ_AFFN_SYSTEM]: print('') print(f'{wq_affn_names[affn].string_().decode().upper()}{" (default)" if affn == wq_affn_dfl else ""}') print_pod_type(wq_pod_types[affn])
Signature · 1 line
2.52.0
PATCH v3 5/6 workqueue: add test_workqueue benchmark module
In reply to Breno LeitaoAdd a kernel module that benchmarks queue_work() throughput on an unbound workqueue to measure pool->lock contention under different affinity scope configurations (cache vs cache_shard).
The module spawns N kthreads (default: num_online_cpus()), each bound to a different CPU. All threads start simultaneously and queue work items, measuring the latency of each queue_work() call. Results are reported as p50/p90/p95 latencies for each affinity scope.
The affinity scope is switched between runs via the workqueue's sysfs affinity_scope attribute (WQ_SYSFS), avoiding the need for any new exported symbols.
The module runs as __init-only, returning -EAGAIN to auto-unload, and can be re-run via insmod.
Example of the output:
running 50 threads, 50000 items/thread
cpu 6806017 items/sec p50=2574 p90=5068 p95=5818 ns smt 6821040 items/sec p50=2624 p90=5168 p95=5949 ns cache_shard 1633653 items/sec p50=5337 p90=9694 p95=11207 ns cache 286069 items/sec p50=72509 p90=82304 p95=85009 ns numa 319403 items/sec p50=63745 p90=73480 p95=76505 ns system 308461 items/sec p50=66561 p90=75714 p95=78048 ns
--- lib/Kconfig.debug | 10 ++ lib/Makefile | 1 + lib/test_workqueue.c | 294 +++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 305 insertions(+)
lib/Kconfig.debug +10 -0
@@ -2628,6 +2628,16 @@ config TEST_VMALLOC If unsure, say N. +config TEST_WORKQUEUE + tristate "Test module for stress/performance analysis of workqueue" + default n + help + This builds the "test_workqueue" module for benchmarking + workqueue throughput under contention. Useful for evaluating + affinity scope changes (e.g., cache_shard vs cache). + + If unsure, say N. + config TEST_BPF tristate "Test BPF filter functionality" depends on m && NET
lib/Makefile +1 -0
@@ -79,6 +79,7 @@ UBSAN_SANITIZE_test_ubsan.o := y obj-$(CONFIG_TEST_KSTRTOX) += test-kstrtox.o obj-$(CONFIG_TEST_LKM) += test_module.o obj-$(CONFIG_TEST_VMALLOC) += test_vmalloc.o +obj-$(CONFIG_TEST_WORKQUEUE) += test_workqueue.o obj-$(CONFIG_TEST_RHASHTABLE) += test_rhashtable.o obj-$(CONFIG_TEST_STATIC_KEYS) += test_static_keys.o obj-$(CONFIG_TEST_STATIC_KEYS) += test_static_key_base.o
lib/test_workqueue.c new file +294 -0
294 lines not on this page · raw message
Signature · 1 line
2.52.0
PATCH v3 6/6 docs: workqueue: document WQ_AFFN_CACHE_SHARD affinity scope
In reply to Breno LeitaoUpdate kernel-parameters.txt and workqueue.rst to reflect the new cache_shard affinity scope and the default change from cache to cache_shard.
--- Documentation/admin-guide/kernel-parameters.txt | 3 ++- Documentation/core-api/workqueue.rst | 14 ++++++++++---- 2 files changed, 12 insertions(+), 5 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +2 -1
@@ -8535,7 +8535,8 @@ Kernel parameters workqueue.default_affinity_scope= Select the default affinity scope to use for unbound workqueues. Can be one of "cpu", "smt", "cache", - "numa" and "system". Default is "cache". For more + "cache_shard", "numa" and "system". Default is + "cache_shard". For more information, see the Affinity Scopes section in Documentation/core-api/workqueue.rst.
Documentation/core-api/workqueue.rst +10 -4
@@ -378,9 +378,9 @@ Affinity Scopes An unbound workqueue groups CPUs according to its affinity scope to improve cache locality. For example, if a workqueue is using the default affinity -scope of "cache", it will group CPUs according to last level cache -boundaries. A work item queued on the workqueue will be assigned to a worker -on one of the CPUs which share the last level cache with the issuing CPU. +scope of "cache_shard", it will group CPUs into sub-LLC shards. A work item +queued on the workqueue will be assigned to a worker on one of the CPUs +within the same shard as the issuing CPU. Once started, the worker may or may not be allowed to move outside the scope depending on the ``affinity_strict`` setting of the scope.
@@ -402,7 +402,13 @@ Workqueue currently supports the following affinity scopes. ``cache`` CPUs are grouped according to cache boundaries. Which specific cache boundary is used is determined by the arch code. L3 is used in a lot of - cases. This is the default affinity scope. + cases. + +``cache_shard`` + CPUs are grouped into sub-LLC shards of at most ``wq_cache_shard_size`` + cores (default 8, tunable via the ``workqueue.cache_shard_size`` boot + parameter). Shards are always split on core (SMT group) boundaries. + This is the default affinity scope. ``numa`` CPUs are grouped according to NUMA boundaries.
Signature · 1 line
2.52.0
Tejun Heo
maintainerHello,
Applied 1-6 to wq/for-7.1.
Thanks.
Signature · 1 line
tejun