Hierarchical Percpu Counters for RSS
Subsystems (9)
Mathieu Desnoyers
reviewerv23 cover Hierarchical Percpu Counters for RSS
Hi David,
Here is the hierarchical per-cpu counters series for the RSS counters, based on v7.4-rc1. It is an RFC because of the questions at the end, and because of the significant changes since v22 [1], notably the per-cpu counter size (long -> byte), the configuration heuristics, and the lazy activation introduced by patches 6 and 7.
The problem
-----------
Since commit f1a7941243c1 ("mm: convert mm's rss stats into percpu_counter"), the RSS counters are struct percpu_counter with a batch size of max(32, 2 * nr_cpus): their approximation error grows with O(nr_cpus^2), up to 1.1GB per counter at 384 logical CPUs. /proc/PID/status, /proc/PID/statm and the OOM killer moved to precise sums (commits 82241a83cd15 and 5898aa8f9a0b), but the other readers still use the approximation: the RSS high watermark (VmHWM, the ru_maxrss reported as "Maximum resident set size" by /usr/bin/time -v, taskstats), /proc/PID/stat (ps, top, monitoring agents), the taskstats RSS-time integral (coremem, sampled at each tick), NUMA balancing and the mark_victim tracepoint. On the test machine below, the /proc/PID/stat RSS of a test process was off by 17MB while single-threaded, and by up to 134MB with 64 threads.
Precise sums cannot fix the high watermark or the RSS-time integral:update_hiwater_rss() samples the RSS on every unmap path, and the integral at each tick, while a precise sum reads every possible CPU, about 4us per counter at 384 logical CPUs. A smaller struct percpu_counter batch takes the counter spinlock on every carry: at batch 32, an update of the counters of a shared mm costs 4.4 to 69us with one thread per CPU, compared to 26ns to 1.2us with this series.
The series
----------
Patches 1 to 5 replace struct percpu_counter for the RSS counters with hierarchical per-cpu counters: per-cpu bytes carry into a tree of atomic counters shaped at boot from nr_cpu_ids, with a fixed batch of 32, an error bound in O(n*log(n)) and no global lock. Each level carries only once its residual reaches the batch in either direction, so updates going back and forth around a boundary do not climb the tree. The comparison functions have no user in this series; a follow-up will use them for a faster OOM killer task selection.
Patches 6 and 7 keep the RSS counters of single-threaded processes exact and without per-cpu memory: until a second thread shares the mm, the task running on the mm adds to the bias of each counter without atomic instructions, and other tasks add to its atomic bias, both read in O(1). Creating the second thread allocates the per-cpu counters and tree items, and activates the tree. struct percpu_counter cannot do this as
is: its count is an s64 under a raw spinlock, which the remote updates (reclaim, khugepaged, remote GUP) would then take.
Patches 8 to 10 are the tools to tune the tree shape on other hardware:
the percpu_counter_tree.arity= and rss_stat_batch= boot parameters, and a KUnit benchmark. The default shapes were measured on one machine, with up to 384 logical CPUs, so results from other hardware are welcome.
Results
-------
2-socket AMD EPYC 9654, 192 cores, 384 logical CPUs with SMT (the CPU counts in this letter are logical CPUs, as in nr_cpu_ids), tree 8,8,8, batch 32, whole series; details in patches 3 and 7. The bare-metal results were measured on v7.3: the rebase onto v7.4-rc1 had no conflict, and the hpcc update costs of the series as sent, measured in a pinned 384-vCPU guest, match the bare-metal ones within 2%.
Accuracy per counter: bound, and largest |approximate - precise| seen by get_mm_counter() while a process hops across all CPUs doing mmap/munmap of 1 to 16 pages (384-vCPU KVM guest pinned to the host):
hpcc upstream
bound 158.5MB 1.1GB
measured, single-threaded 0 (exact) 841MB
measured, with a second thread 96MB 845MB
Update cost, ns per update per thread, one thread per CPU, bare metal (mm: the 4 counters of one mm; private: one counter per thread; batch 32: local change of the benchmark, bound 46.5MB):
upstream upstream
hpcc batch 768 batch 32
mm, +1 26 288 4370
mm, random +-32 135 385 69171
mm, random +-512 1174 20229 69186
private, random +-32 31 20 -
/proc/PID/statm read with 1000 to 16000 processes, bare metal:
5.3-6.5us vs 16.5-19.0us for single-threaded processes, on par for multi-threaded ones (14.4-16.0us vs 14.5-17.1us, guest). fork + exit of a single-threaded child: 116us vs 139us. will-it-scale page_fault1, 2 and 3: no regression (-0.6% to +5.1%, one boot each, the upstream one on the previous day).
Memory per mm, counting the slab objects of mm_struct and of the tree items, and the per-cpu memory of the RSS counters, in bytes (guest, v7.4-rc1):
single-threaded threaded upstream
16 CPUs 1820 1948 2076
384 CPUs 1928 8145 8072
1024 CPUs 2185 17204 18569 The tree items are allocated with the per-cpu counters when the second thread is created, and the mm_struct slab object keeps its upstream size.
Behavior changes
----------------
- The allocation of the per-cpu counters and tree items moves from fork() to the second thread (clone() with CLONE_VM, without CLONE_VFORK), which can then fail with -ENOMEM.
- mm_struct gains rss_owner, the task running on the mm, compared with current and never dereferenced, and rss_stat_mutex, which serializes the activations of the tree, as mm_cid.mutex serializes the mode changes of mm_cid at fork and exit.
- The precise sums (/proc/PID/status, /proc/PID/statm, the OOM killer) take no lock: under concurrent updates of a multi-threaded process, they can miss a carry on its way up the tree, where upstream counts each concurrent update either whole or not at all. They are exact for a process at rest.
- The rss_stat tracepoint reports the approximation (patch 5): exact for single-threaded processes, within the bound otherwise.
- Like struct percpu_counter, the tree is sized at boot and not reshaped on hotplug; its shape and bound depend on nr_cpu_ids, while the default upstream batch follows the online CPUs.
Caveats
-------
- From 33 to 39 CPUs, up to 1.37 times less accurate than upstream.
- The tree is shaped for the number of CPUs rounded up to a power of 2:
just above a power of 2, the RSS counters of a threaded process, with rss_owner and the mutex, use up to 1.50 times the memory of upstream (129 CPUs, 1.28 times at 65 and 1.38 times at 257), compared to at most 0.90 times at a power of 2, and 1.02 times at 384 logical CPUs.
- Private counters with random +-32 updates: 31ns vs 20ns, the price of a 7 times tighter bound.
- A multi-threaded process which ran on few CPUs can hold more error than upstream: 2047 pages per CPU path vs 767 at 384 logical CPUs. A follow-up could index the counters by concurrency ID (mm_cid) instead of CPU, so the accuracy bound and the precise sums would scale with the concurrency the process uses.
- Before activation, concurrent remote updates of a single-threaded process contend on one atomic per counter.
- The second thread allocates per-cpu memory, which the per-cpu allocator serializes, as it does upstream at every fork: with 384 threads at once, init + destroy of 4 counters takes 874us per operation (1004us upstream, 6.0us alone). Tree item slab objects owning their per-cpu counters, with slab destructors, could avoid it.
- The accuracy was measured in a guest, with a debug patch and a test program which are not part of the series. The bound is analytic and tested in both directions.
Changes since v22 [1]
---------------------
- Per-cpu byte counters with hysteresis, updated with this_cpu_try_cmpxchg(); the tree items of counters initialized together share cache lines; the library allocates the per-cpu counters and tree items, so struct percpu_counter_tree has a fixed size of 40 bytes, like struct percpu_counter, and mm_struct no longer embeds tree items.
- Flat tree up to 32 CPUs, 16,4 up to 64, fan-out 8 beyond, with a root fan-out of 16 when one order remains (8,16 at 128 CPUs, 8,8,16 at 1024).
- New patches 6 and 7, for single-threaded processes.
- Fixed approximation margins in the comparison functions; hysteresis on the intermediate levels, so the accuracy range is exact and symmetric.
- Userspace test of every tree configuration, KUnit tests on per-CPU workqueues, Sashiko review comments, MAINTAINERS entry.
Testing: every commit builds without warnings on x86_64 (SMP and UP), arm64, ppc64le and 32-bit powerpc, sparse adds no warnings, KUnit passes on bare metal (v7.3) and in guests with 1, 16 and 384 vCPUs, the userspace test passes on simulated topologies up to 65536 CPUs, and the RSS counters match a page table walk from 1 to 64 threads. KUnit and the RSS check also pass with lockdep (PROVE_LOCKING) in guests with 16 and 64 vCPUs, without report.
Starting from v22, which I wrote, an LLM coding assistant drove the implementation of this revision from my design decisions: it wrote the code, the tests, the benchmark and the measurement scripts, ran the measurements in guests and prepared the bare-metal ones, and drafted the comments, the changelogs and this letter. I made the design decisions, reviewed every change and directed the rework through my review comments.
Questions
---------
1. Is mm->rss_owner acceptable, or would you prefer current->mm == mm (without PF_KTHREAD) and an "activated" flag?
2. Is moving the -ENOMEM from fork() to the second thread acceptable?
3. Would you rather take the single-threaded mode first, on its own?
4. Should the rss_stat tracepoint report the approximation, stay precise, or report both?
5. Should the tuning tools be merged, built only with the benchmark config, or dropped later? Who could run them on non-AMD, 4-socket, arm64 or POWER machines?
6. Is tools/testing/hpcc the right home for the topology test?
Thanks,
Mathieu
[1] https://lore.kernel.org/lkml/[email protected]/
Mathieu Desnoyers (10):
lib: introduce hierarchical per-cpu counters
lib: test hierarchical per-cpu counters
mm: use hierarchical per-cpu counters for the RSS counters tools/testing/hpcc: add userspace test of hierarchical per-CPU
counters
mm: report the approximate RSS in the rss_stat tracepoint
lib: percpu_counter_tree: add lazy tree activation
mm: keep the RSS counters of single-threaded processes exact
lib: percpu_counter_tree: add the arity boot parameter
mm: add the rss_stat_batch boot parameter
lib: add a benchmark of hierarchical per-cpu counters
.../admin-guide/kernel-parameters.txt | 24 + Documentation/core-api/index.rst | 1 + .../core-api/percpu-counter-tree.rst | 145 ++ MAINTAINERS | 12 + fs/exec.c | 5 + include/linux/mm.h | 28 +- include/linux/mm_types.h | 19 +- include/linux/percpu_counter_tree.h | 365 +++++ include/trace/events/kmem.h | 3 +- init/main.c | 3 + kernel/exit.c | 3 + kernel/fork.c | 61 +- lib/Kconfig | 27 + lib/Makefile | 1 + lib/percpu_counter_tree.c | 1450 +++++++++++++++++ lib/tests/Makefile | 3 + lib/tests/percpu_counter_tree_bench.c | 604 +++++++ lib/tests/percpu_counter_tree_kunit.c | 1050 ++++++++++++ tools/testing/hpcc/.gitignore | 2 + tools/testing/hpcc/Makefile | 18 + tools/testing/hpcc/include/hpcc_shim.h | 203 +++ tools/testing/hpcc/include/kunit/visibility.h | 2 + tools/testing/hpcc/include/linux/args.h | 4 + tools/testing/hpcc/include/linux/atomic.h | 2 + tools/testing/hpcc/include/linux/bug.h | 2 + tools/testing/hpcc/include/linux/cache.h | 2 + tools/testing/hpcc/include/linux/cpumask.h | 2 + tools/testing/hpcc/include/linux/errno.h | 3 + tools/testing/hpcc/include/linux/limits.h | 2 + tools/testing/hpcc/include/linux/math.h | 2 + .../testing/hpcc/include/linux/moduleparam.h | 12 + tools/testing/hpcc/include/linux/percpu.h | 2 + .../hpcc/include/linux/percpu_counter_tree.h | 2 + tools/testing/hpcc/include/linux/preempt.h | 2 + tools/testing/hpcc/include/linux/slab.h | 2 + tools/testing/hpcc/main.c | 536 ++++++ 36 files changed, 4592 insertions(+), 12 deletions(-) create mode 100644 Documentation/core-api/percpu-counter-tree.rst create mode 100644 include/linux/percpu_counter_tree.h create mode 100644 lib/percpu_counter_tree.c create mode 100644 lib/tests/percpu_counter_tree_bench.c create mode 100644 lib/tests/percpu_counter_tree_kunit.c create mode 100644 tools/testing/hpcc/.gitignore create mode 100644 tools/testing/hpcc/Makefile create mode 100644 tools/testing/hpcc/include/hpcc_shim.h create mode 100644 tools/testing/hpcc/include/kunit/visibility.h create mode 100644 tools/testing/hpcc/include/linux/args.h create mode 100644 tools/testing/hpcc/include/linux/atomic.h create mode 100644 tools/testing/hpcc/include/linux/bug.h create mode 100644 tools/testing/hpcc/include/linux/cache.h create mode 100644 tools/testing/hpcc/include/linux/cpumask.h create mode 100644 tools/testing/hpcc/include/linux/errno.h create mode 100644 tools/testing/hpcc/include/linux/limits.h create mode 100644 tools/testing/hpcc/include/linux/math.h create mode 100644 tools/testing/hpcc/include/linux/moduleparam.h create mode 100644 tools/testing/hpcc/include/linux/percpu.h create mode 100644 tools/testing/hpcc/include/linux/percpu_counter_tree.h create mode 100644 tools/testing/hpcc/include/linux/preempt.h create mode 100644 tools/testing/hpcc/include/linux/slab.h create mode 100644 tools/testing/hpcc/main.c
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
Signature · 1 line
2.47.3
Mathieu Desnoyers
reviewerRFC PATCH v23 1/10 lib: introduce hierarchical per-cpu counters
Introduce hierarchical per-cpu counters, struct percpu_counter_tree: a split counter whose approximate sum stays within a fixed bound of its precise sum, without taking a global lock on updates. They are meant for counters updated from many CPUs whose approximation is read often, such as the RSS counters of an mm. A following patch uses them for the RSS counters, to bound their inaccuracy on large many-core systems [1].
* Motivation
The purpose of this hierarchical split-counter scheme is to:
- Minimize contention when incrementing and decrementing counters,
- Provide fast access to a sum approximation,
- Provide a sum approximation with an acceptable accuracy level when scaling to many-core systems.
- Provide approximate and precise comparison of two counters, and between a counter and a value.
- Provide possible precise sum ranges for a given sum approximation.
* Design
The hierarchical per-CPU counters propagate a sum approximation through an N-way tree. When reaching the batch size, the carry is propagated up the tree, whose fan-out at each level is selected at boot from nr_cpu_ids. The batch size of each level is the batch size of the prior level multiplied by the prior level's fan-out.
The following diagram illustrates the propagation with 8 CPUs through a binary tree. This is not an actual configuration: at 8 CPUs, the default tree is flat (see the default shapes below). CPU c propagates into item (c & (L - 1)) of a level holding L items, so the CPUs are listed in bit-reversed order:
Level 3: 0 (root)
| \
| \
| \
Level 2: 0 1
| \ | \
| \ | \
| \ | \
Level 1: 0 2 1 3
| \ | \ | \ | \
| \ | \ | \ | \
| \ | \ | \ | \
Level 0: 0 4 2 6 1 5 3 7Each counter below the root keeps a residual: once it reaches its batch size in either direction, it propagates its multiples of the batch size and subtracts them, which leaves a residual within (-batch, +batch). This hysteresis absorbs updates going back and forth around a batch size boundary, instead of propagating them up to the root on each update. The residual moves in steps of the counter's input: 1 for the per-cpu counters, which receive the updates, and BATCH(N - 1), the batch size of the level below, for the counters of level N, which only receive its carries. The largest residual is therefore batch - 1 at level 0, and BATCH(N) - BATCH(N - 1) at level N. The approximate sum is the precise sum minus the residuals, so it is within the same distance of the precise sum in both directions:
accuracy = nr_cpu_ids * (batch - 1) +
sum(N >= 1) (BATCH(N) - BATCH(N - 1)) * nr_items(N)
The level 0 per-cpu counters are signed bytes, updated with a this_cpu_try_cmpxchg() loop, which limits the batch size to 128.
Up to 32 CPUs, the tree is flat: the per-cpu counters propagate directly to the root, like struct percpu_counter. From 33 to 64 CPUs, the per-cpu counters propagate to 4 intermediate counters, each shared by up to 16 CPUs, so the carries of all CPUs do not contend on the root. Beyond, each level has a fan-out of 8: a remaining order of 1 gives the root a fan-out of 16, and a remaining order of 2 adds a level with a fan-out of 4 below the root. With a fixed batch size, the maximum inaccuracy grows with O(n*log(n)) as the number of CPUs increases, compared to O(n^2) for struct percpu_counter, whose batch size grows with the number of CPUs. For example, at 384 logical CPUs with a batch size of 32, the bound is 40576, compared to 294528 for a struct percpu_counter with its default batch size of 768.
* CPU hotplug
Like the per-cpu data of struct percpu_counter, the tree is sized at boot, and is not reshaped on CPU hotplug. The level 0 residual of an offline CPU stays in place: the precise sum still reads it, and the bound, computed from nr_cpu_ids, still holds. Unlike the default batch size of struct percpu_counter, which follows the number of online CPUs, the bound does not shrink when fewer CPUs are online. When the possible CPUs are sparse, the tree is still shaped for nr_cpu_ids, the highest possible CPU number plus one, and the bound also counts the level 0 residuals of the missing CPUs: it holds, but cannot be reached.
* Memory Use
Each counter uses a 32-byte struct percpu_counter_tree, one byte of per-cpu data per possible CPU, and its tree items. percpu_counter_tree_init_many() allocates the per-cpu data and the tree items, whose sizes depend on the number of possible CPUs and on nr_cpu_ids, so struct percpu_counter_tree has a fixed size. The items at the same tree position of all the counters initialized together are packed in a cache line aligned block, so a set of counters updated together, such as the 4 RSS counters of an mm, shares cache lines without false sharing between CPUs updating different branches of the tree. When the items of a tree position fit in a cache line, the tree items come from a slab cache created by percpu_counter_tree_subsystem_init(), which runs once the slab allocator is up.
The memory use of nr_counters initialized together, up to 8, on a 64-bit arch with 64-byte cache lines is:
nr_counters * 32 + ALIGN(nr_counters, 4) * nr_possible_cpus
+ nr_items * 64
as the per-cpu allocator rounds sizes up to 4 bytes. The tree is shaped for nr_cpu_ids rounded up to a power of 2: nr_items is 1 (the root) up to 32 CPUs, 5 up to 64, 17 up to 128, 41 up to 256, 73 up to 512, 145 up to 1024 and 585 up to 4096. In comparison, a struct percpu_counter uses 40 bytes and 4 bytes of per-cpu data per possible CPU.
For a set of 4 counters, the hierarchical counters use less memory than as many struct percpu_counter from 3 to 64 CPUs, and beyond when the number of CPUs is a power of 2 (at most 0.89 times as much). Between powers of 2, fewer CPUs share the cost of the tree items: 0.94 to 1.09 times at 3 * 2^n CPUs (96, 192, 384, ...), and up to 1.47 times just above a power of 2 (129 CPUs). A lone counter uses one item of each cache line: 3.0 to 3.5 times the memory of a struct percpu_counter when the number of CPUs is a power of 2 beyond 64, and up to 5.7 times just above.
---
Changes since v22:
- Refactor configuration table, remove duplicated definitions, document heuristic, add table content validation, run by the KUnit and userspace tests.
- Allocate the per-cpu counters and the tree items in percpu_counter_tree_init_many(), instead of taking tree items from the caller: remove percpu_counter_tree_items_size(), PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE and the items argument of the init functions. struct percpu_counter_tree has a fixed size of 32 bytes, down from 64.
- Call percpu_counter_tree_subsystem_init() after mm_core_init(), once its boot parameters are parsed and slab is up, before the first counter is used, and warn about counters initialized before it.
- Use signed bytes for the level 0 per-cpu counters, with the same hysteresis as the intermediate levels, updated with a this_cpu_try_cmpxchg() loop. Limit the batch size to 128.
- Pack the tree items of the counters initialized together by percpu_counter_tree_init_many() in cache lines, allocated from a slab cache when the items of a tree position fit in one cache line.
- Use a flat tree up to 32 CPUs, a fan-out of 16 then 4 from 33 to 64 CPUs to keep the carries of all CPUs from contending on the root, and a fan-out of 8 beyond, with a root fan-out of 16 for a remaining order of 1, to match the memory footprint and accuracy of struct percpu_counter at low CPU counts and bound the inaccuracy with O(n*log(n)) at high CPU counts.
- Print the tree shape at boot.
- Document that the accuracy range is a property of settled counters. (Reported by Sashiko.)
- Add a diagram of the approximation ranges above compare_delta(). (Reported by Sashiko.)
- Introduce wrapped_delta(), and document that comparisons are valid only for counters differing by less than LONG_MAX. (Reported by Sashiko.)
- Fix the direction of the approximation margins in the comparison functions. They were swapped, which returned wrong definitive answers when 2^arity of a level exceeds the batch size (batch size 2 from 9 CPUs, batch size 4 from 65 CPUs).
- Add hysteresis to the intermediate tree levels: they keep a residual within (-batch, +batch) of their level, and only propagate once it reaches their batch size, so updates going back and forth around a batch size boundary no longer propagate up to the root on each update.
- Replace percpu_counter_tree_approximate_accuracy_range() and its "under" and "over" ranges by a single accuracy, the exact bound of the residuals, which is the same in both directions:
percpu_counter_tree_approximate_accuracy().
- Add percpu_counter_tree_check_residuals() for the tests: it checks the residuals of the intermediate counters, and that no carry was lost or duplicated.
- Extract the carries of the intermediate counters with a cmpxchg on the value returned by the addition, so concurrent updaters never subtract a carry computed from a stale residual, and explain why relaxed ordering is sufficient.
- Read the level 0 counters with READ_ONCE() in the precise sum.
- Document that percpu_counter_tree_set() must not run concurrently with updates.
- Add the documentation to the core-api toctree.
- Fix the propagation diagram to match the CPU to item mapping.
- Mark boot-time topology state __ro_after_init, use pr_err().
- Validate the batch size on UP as on SMP.
- Fix the sign of the !CONFIG_SMP value comparisons, and compare on the wrapped difference as on SMP.
- Add a MAINTAINERS entry.
- Fix checkpatch warnings.
- Fix documentation typos.
--- Documentation/core-api/index.rst | 1 + .../core-api/percpu-counter-tree.rst | 109 ++ MAINTAINERS | 9 + include/linux/percpu_counter_tree.h | 292 +++++ init/main.c | 3 + lib/Makefile | 1 + lib/percpu_counter_tree.c | 1119 +++++++++++++++++ 7 files changed, 1534 insertions(+) create mode 100644 Documentation/core-api/percpu-counter-tree.rst create mode 100644 include/linux/percpu_counter_tree.h create mode 100644 lib/percpu_counter_tree.c
Documentation/core-api/index.rst +1 -0
@@ -48,6 +48,7 @@ Library functionality that is used throughout the kernel. generic-radix-tree packing this_cpu_ops + percpu-counter-tree timekeeping errseq wrappers/atomic_t
Documentation/core-api/percpu-counter-tree.rst new file +109 -0
109 lines not on this page · raw message
MAINTAINERS +9 -0
@@ -11947,6 +11947,15 @@ L: [email protected] S: Maintained F: drivers/hid/hid-logitech-hidpp.c +HIERARCHICAL PER-CPU COUNTERS (HPCC) +M: Mathieu Desnoyers <[email protected]> +L: [email protected] +L: [email protected] +S: Supported +F: Documentation/core-api/percpu-counter-tree.rst +F: include/linux/percpu_counter_tree.h +F: lib/percpu_counter_tree.c + HIGH-RESOLUTION TIMERS, TIMER WHEEL, CLOCKEVENTS M: Anna-Maria Behnsen <[email protected]> M: Frederic Weisbecker <[email protected]>
include/linux/percpu_counter_tree.h new file +292 -0
292 lines not on this page · raw message
init/main.c +3 -0
@@ -108,6 +108,7 @@#include <linux/time_namespace.h> #include <linux/unaligned.h> #include <linux/vdso_datastore.h> +#include <linux/percpu_counter_tree.h> #include <net/net_namespace.h> #include <asm/io.h>
@@ -1104,6 +1105,8 @@ void start_kernel(void)sort_main_extable ();trap_init ();mm_core_init (); +/* After parsing its boot parameters and setting up slab, before the first counter. */ +percpu_counter_tree_subsystem_init ();maple_tree_init ();poking_init ();ftrace_init ();
lib/Makefile +1 -0
@@ -182,6 +182,7 @@ obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o obj-$(CONFIG_TEXTSEARCH_BM) += ts_bm.o obj-$(CONFIG_TEXTSEARCH_FSM) += ts_fsm.o obj-$(CONFIG_SMP) += percpu_counter.o +obj-$(CONFIG_SMP) += percpu_counter_tree.o obj-$(CONFIG_AUDIT_GENERIC) += audit.o obj-$(CONFIG_AUDIT_COMPAT_GENERIC) += compat_audit.o
lib/percpu_counter_tree.c new file +1119 -0
1,119 lines not on this page · raw message
Signature · 1 line
2.47.3
Mathieu Desnoyers
reviewerRFC PATCH v23 2/10 lib: test hierarchical per-cpu counters
In reply to Mathieu DesnoyersIntroduce Kunit tests for hierarchical per-cpu counters.
Keep track of two sets of hierarchical counters, each meant to have the same precise sum at any time, but distributed differently across the topology.
Keep track of an atomic counter along with each hierarchical counter, for sum validation.
Those tests cover:
- Single-threaded (no concurrency) updates.
- Concurrent updates of counters from various CPUs.
- Contended carries: rounds of concurrent carries from many CPUs into the tree items they share.
The multi-threaded tests run their workers on at most 64 CPUs, spread evenly across the online CPUs, so their run time does not grow with the size of the machine.
The single-threaded and saturation tests run each update on its target CPU with work_on_cpu(), and the multi-threaded tests queue their updates on a CPU intensive workqueue, whose workers are reused across rounds and can preempt each other on a CPU. The single and multi-threaded tests stop early after a time bound, so they complete on large virtual machines where cross-CPU wakeups are slow.
Perform the following validations:
- Compare the precise sum of counters with the sum tracked by an atomic counter.
- Compare the precise sum of two sets of hierarchical counters.
- Approximated comparison of hierarchical counter with atomic counter.
- Approximated comparison of two sets of hierarchical counters.
- Validate the bounds of approximation ranges.
- Check the residuals of the tree items, and that no carry was lost or duplicated, whenever the counters are settled, including between the rounds of contended carries.
- Saturate a counter upward and downward to its worst-case approximation errors, check that it reaches the edges of its accuracy range exactly when all possible CPUs are online, and validate the comparisons at the edges of its precise range.
Run with the following .kunit/.kunitconfig:
CONFIG_KUNIT=yCONFIG_SMP=yCONFIG_PREEMPT=yCONFIG_NR_CPUS=32CONFIG_HOTPLUG_CPU=yCONFIG_PERCPU_COUNTER_TREE_KUNIT_TEST=y
With the following execution (to use SMP):
./tools/testing/kunit/kunit.py run --arch=x86_64 --qemu_args="-smp 12"
---
Changes since v22:
- Fix CPU hotplug locking, module unload, and error handling: ERR_PTR handling, a test data leak, allocation sizing, the cpu & ~1 skip, a lost worker when kthread_stop() runs before the thread function, and a task_struct reaping race. (Reported by Sashiko.)
- Merge the four multi-thread increment tests into one function taking a CPU selector.
- Add a saturated counter comparison test, upward and downward.
- Add a contended carries test.
- Bound the number of CPUs running workers in the multi-threaded tests.
- Check the residuals and carries of settled counters.
- Add a configuration table validation test.
- Run the updates with work_on_cpu() and a workqueue rather than creating a kthread for each update, and bound the run time of the single and multi-threaded tests, which timed out with 192 CPUs in a virtual machine.
- Fix checkpatch warnings.
Changes since v19:
- Rename config option to PERCPU_COUNTER_TREE_KUNIT_TEST.
--- MAINTAINERS | 1 + lib/Kconfig | 12 + lib/tests/Makefile | 2 + lib/tests/percpu_counter_tree_kunit.c | 832 ++++++++++++++++++++++++++ 4 files changed, 847 insertions(+) create mode 100644 lib/tests/percpu_counter_tree_kunit.c
MAINTAINERS +1 -0
@@ -11955,6 +11955,7 @@ S: Supported F: Documentation/core-api/percpu-counter-tree.rst F: include/linux/percpu_counter_tree.h F: lib/percpu_counter_tree.c +F: lib/tests/percpu_counter_tree_kunit.c HIGH-RESOLUTION TIMERS, TIMER WHEEL, CLOCKEVENTS M: Anna-Maria Behnsen <[email protected]>
lib/Kconfig +12 -0
@@ -41,6 +41,18 @@ config PACKING_KUNIT_TEST When in doubt, say N. +config PERCPU_COUNTER_TREE_KUNIT_TEST + tristate "KUnit tests for hierarchical per-cpu counters" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + This builds Kunit tests for the hierarchical per-cpu counters. + + For more information on KUnit and unit tests in general, + please refer to the KUnit documentation in Documentation/dev-tools/kunit/. + + When in doubt, say N. + config BITREVERSE tristate select GENERIC_BITREVERSE if !HAVE_ARCH_BITREVERSE
lib/tests/Makefile +2 -0
@@ -59,4 +59,6 @@ obj-$(CONFIG_UTIL_MACROS_KUNIT) += util_macros_kunit.o obj-$(CONFIG_RATELIMIT_KUNIT_TEST) += test_ratelimit.o obj-$(CONFIG_UUID_KUNIT_TEST) += uuid_kunit.o +obj-$(CONFIG_PERCPU_COUNTER_TREE_KUNIT_TEST) += percpu_counter_tree_kunit.o + obj-$(CONFIG_TEST_RUNTIME_MODULE) += module/
lib/tests/percpu_counter_tree_kunit.c new file +832 -0
832 lines not on this page · raw message
Signature · 1 line
2.47.3
[email protected]
AI reviewRe: [RFC PATCH v23 02/10] lib: test hierarchical per-cpu counters
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Medium] lib/tests: unlocked cpu_online_mask read passed to work_on_cpu() - [Medium] lib/tests: missing…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Medium] lib/tests: unlocked cpu_online_mask read passed to work_on_cpu()
- [Medium] lib/tests: missing release/acquire ordering on sh.started in contention test
- [Low] lib/tests: unlocked num_online_cpus() check in do_hpcc_test_saturated_compare() --
commit 48b3ce088047ee1ccff5c20155903c1a143e78df
Author: Mathieu Desnoyers <[email protected]>
lib: test hierarchical per-cpu counters
Introduce KUnit tests for hierarchical per-CPU counters covering single-threaded updates, concurrent multi-threaded updates, contended carries, and saturated counter comparisons.
Quoted 5 lines
lib/tests/percpu_counter_tree_kunit.c new file +0
-0
[ ... ]
Quoted 54 lines
lib/tests/percpu_counter_tree_kunit.c new file +54
-0+static void do_hpcc_test_single_thread (struct kunit *test,int _cpu0,int _cpu1, +enum up_test_inc_type type) +{ +unsigned long batch_size_order =5 ; +int cpu0 = _cpu0; +int cpu1 = _cpu1; +unsigned long deadline = jiffies +SINGLE_THREAD_MAX_SECONDS *HZ ; +int ret, i; + + ret =init_counters (1UL << batch_size_order); +KUNIT_EXPECT_EQ (test, ret,0 ); +if (ret) +return ; +cpus_read_lock (); +for (i =0 ; i <10000 &&time_before (jiffies, deadline); i++) { +struct single_thread_update u0 = { .counter_index =0 }; +struct single_thread_update u1 = { .counter_index =1 }; +long increment; + +switch (type) { +case INC_ONE : + increment =1 ; +break ; +case INC_MINUS_ONE : + increment =-1 ; +break ; +case INC_RANDOM : + increment = (long )get_random_long () %50000 ; +break ; +default : +WARN_ON_ONCE (1 ); + increment =0 ; + } +if (_cpu0 <0 ) + cpu0 =cpumask_any_distribute (cpu_online_mask); +if (_cpu1 <0 ) + cpu1 =cpumask_any_distribute (cpu_online_mask); + u0.increment = increment; + u1.increment = increment; +work_on_cpu (cpu0, single_thread_update_fn, &u0); +work_on_cpu (cpu1, single_thread_update_fn, &u1); +check_counters (test); + } +cpus_read_unlock (); +fini_counters (); +} + +static void hpcc_test_single_thread_first (struct kunit *test) +{ +int cpu =cpumask_first (cpu_online_mask); + +do_hpcc_test_single_thread (test, cpu, cpu,INC_ONE ); +do_hpcc_test_single_thread (test, cpu, cpu,INC_MINUS_ONE ); +do_hpcc_test_single_thread (test, cpu, cpu,INC_RANDOM );
lib/tests/percpu_counter_tree_kunit.c line 269
Quoted 10 lines
lib/tests/percpu_counter_tree_kunit.c new file +10
-0+} + +static void hpcc_test_single_thread_first_random (struct kunit *test) +{ +int cpu =cpumask_first (cpu_online_mask); + +do_hpcc_test_single_thread (test, cpu,-1 ,INC_ONE ); +do_hpcc_test_single_thread (test, cpu,-1 ,INC_MINUS_ONE ); +do_hpcc_test_single_thread (test, cpu,-1 ,INC_RANDOM ); +}
[Severity: Medium]
Can cpu go offline before or between the calls todo_hpcc_test_single_thread()?
Both hpcc_test_single_thread_first() and hpcc_test_single_thread_first_random() sample cpumask_first(cpu_online_mask) without holding cpus_read_lock(), and then pass that cpu across three separate calls to do_hpcc_test_single_thread(), each of which acquires and releases cpus_read_lock() on its own.
Because _cpu0 >= 0 (and _cpu1 >= 0 in hpcc_test_single_thread_first()), do_hpcc_test_single_thread() never re-reads or validates cpu0 or cpu1 under cpus_read_lock() before passing them to work_on_cpu(), which requires the caller to ensure the target CPU cannot go offline.
If CPU hot-unplug runs concurrently under CONFIG_HOTPLUG_CPU=y, could work_on_cpu() be called on an offline CPU?
[ ... ]
Quoted 31 lines
lib/tests/percpu_counter_tree_kunit.c new file +31
-0+static int contention_worker_fn (void *data) +{ +struct contention_data *cd = data; +struct contention_shared *sh = cd->sh; +int round; + +wait_event (contention_wq,READ_ONCE (sh->started)); +for (round =1 ; round <=CONTENTION_NR_ROUNDS ; round++) { +if (cd->coordinator) { +atomic_set_release (&sh->round, round); +contention_add (cd); +while (atomic_read_acquire (&sh->nr_done) < round * (sh->nr_workers -1 )) +cond_resched (); +#ifdef CONFIG_SMP +if (percpu_counter_tree_check_residuals (&counter[0 ])) + sh->nr_errors++; +#endif + }else { +while (atomic_read_acquire (&sh->round) < round) +cond_resched (); +contention_add (cd); +/* Order the additions before the coordinator check. */ +atomic_inc_return_release (&sh->nr_done); + } + } +complete_work (); +return 0 ; +} + +static void do_hpcc_test_contended_carries (struct kunit *test,unsigned long batch_size) +{
[ ... ]
Quoted 8 lines
lib/tests/percpu_counter_tree_kunit.c new file +8
-0+for_each_cpu (cpu, cpus) { + cd[cpu].sh = &sh; + cd[cpu].coordinator = first; + cd[cpu].seed =get_random_u32 () |1 ; + task[cpu] =kthread_create_on_cpu (contention_worker_fn, &cd[cpu], cpu, +"kunit_hpcc_contention/%u" ); +KUNIT_EXPECT_FALSE (test,IS_ERR (task[cpu])); +if (IS_ERR (task[cpu]))
lib/tests/percpu_counter_tree_kunit.c line 565
Quoted 10 lines
lib/tests/percpu_counter_tree_kunit.c new file +10
-0+break ; +atomic_inc (&kernel_threads_to_run); +get_task_struct (task[cpu]); +wake_up_process (task[cpu]); + sh.nr_workers++; + first = false; + } +/* Start the workers once their number is known. */ +WRITE_ONCE (sh.started, true); +wake_up_all (&contention_wq);
[Severity: Medium]
On weakly-ordered architectures, can the coordinator thread in contention_worker_fn() observe sh->started == true before seeing all increments to sh.nr_workers?
In do_hpcc_test_contended_carries(), each worker kthread (starting with the coordinator on the first iteration) is woken via wake_up_process(task[cpu]) inside the loop while sh.nr_workers++ is still being incremented on subsequent iterations, and completion is published with WRITE_ONCE(sh.started, true).
In contention_worker_fn(), if the already-running coordinator hits the wait_event(contention_wq, READ_ONCE(sh->started)) fastpath before wake_up_all() is called, it does not go through the waitqueue spinlock and there is no release/acquire pairing ordering the updates to sh.nr_workers before the coordinator reads sh->nr_workers in:
while (atomic_read_acquire(&sh->nr_done) < round * (sh->nr_workers - 1))
Could this cause the coordinator to wait for fewer workers than were created and call percpu_counter_tree_check_residuals(&counter[0]) while other workers are still updating counter[0]? Should sh.started use smp_store_release() and smp_load_acquire()?
[ ... ]
Quoted 3 lines
lib/tests/percpu_counter_tree_kunit.c new file +3
-0+static void do_hpcc_test_saturated_compare (struct kunit *test,unsigned long batch_size, +long inc) +{
[ ... ]
Quoted 13 lines
lib/tests/percpu_counter_tree_kunit.c new file +13
-0+cpus_read_lock (); +for_each_online_cpu (cpu) { +work_on_cpu (cpu, saturate_worker_fn, &sd); +KUNIT_EXPECT_FALSE (test, sd.overrun); +if (sd.overrun) +break ; + } +cpus_read_unlock (); + + precise =percpu_counter_tree_precise_sum (c); + approx =percpu_counter_tree_approximate_sum (c); +kunit_info (test,"batch_size=%lu inc=%ld precise=%ld approx=%ld accuracy=%lu\n" , + batch_size, inc, precise, approx, accuracy);
lib/tests/percpu_counter_tree_kunit.c line 742
Quoted 10 lines
lib/tests/percpu_counter_tree_kunit.c new file +10
-0+ +check_residuals (test, c); +check_residuals (test, &probe); + +/* Accuracy contract. */ +KUNIT_EXPECT_GE (test, approx, precise - (long )accuracy); +KUNIT_EXPECT_LE (test, approx, precise + (long )accuracy); +/* Saturating every possible CPU reaches the accuracy exactly. */ +if (num_online_cpus () == nr_cpu_ids) +KUNIT_EXPECT_EQ (test, (precise - approx) * inc, (long )accuracy);
[Severity: Low]
In do_hpcc_test_saturated_compare(), what happens if a CPU is offline during the for_each_online_cpu() saturation loop and comes online after cpus_read_unlock()?
Because num_online_cpus() == nr_cpu_ids is checked after cpus_read_unlock() is dropped, a CPU brought online in that window would make num_online_cpus() == nr_cpu_ids true even though saturate_worker_fn() did not run on it, causing the KUNIT_EXPECT_EQ(test, (precise - approx) * inc, (long)accuracy) assertion to fail spuriously.
Should num_online_cpus() == nr_cpu_ids be recorded while cpus_read_lock() is still held?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Mathieu Desnoyers
reviewerRFC PATCH v23 3/10 mm: use hierarchical per-cpu counters for the RSS counters
In reply to Mathieu DesnoyersUse hierarchical per-cpu counters for RSS tracking to bound the inaccuracy of the per-mm RSS approximation on large many-core systems [1].
The approximation is read by /proc/PID/stat (rss), by the RSS high watermark (VmHWM in /proc/PID/status, getrusage() ru_maxrss, taskstats hiwater_rss), by the taskstats integral RSS, by NUMA balancing to scale its scan period, and by the mark_victim OOM tracepoint. /proc/PID/status, /proc/PID/statm, the OOM killer and the rss_stat tracepoint use precise sums, which iterate over all CPUs for each read:
the tracepoint since commit f1a7941243c1 ("mm: convert mm's rss stats into percpu_counter"), and the others since commit 82241a83cd15 ("mm:
fix the inaccurate memory statistics issue for users") and commit 5898aa8f9a0b ("mm: fix OOM killer inaccuracy on large many-core systems").
Here is a (possibly incomplete) list of the prior approaches that were used or proposed, along with their downside:
1) Per-thread rss tracking: large error on many-thread processes.
2) Per-CPU counters: up to 12% slower for short-lived processes and 9% increased system time in make test workloads [1]. Moreover, the inaccuracy increases with O(n^2) with the number of CPUs.
3) Per-NUMA-node counters: requires atomics on fast-path (overhead), error is high with systems that have lots of NUMA nodes (32 times the number of NUMA nodes).
4) Use a precise per-cpu counter sum for each counter value query:
Requires iteration on each possible CPU for each sum, which adds overhead on large many-core systems running many processes.
The approach proposed here is to replace the per-cpu counters by the hierarchical per-cpu counters, which bounds the inaccuracy based on the system topology with O(N*logN).
* Testing results:
Test hardware: 2 sockets AMD EPYC 9654 96-Core Processor (384 logical CPUs)
Methodology:
Comparing the current upstream implementation with the hierarchical counters is done with a debug patch, which keeps both implementations wired up in parallel, and a test program, neither part of this series. The single-process, single-threaded program hops randomly across CPUs in the system, calling mmap(2) and munmap(2) on random CPUs, keeping track of an array of 100000 mapping slots, randomly choosing slots to either map 1 to 16 pages or unmap them.
get_mm_counter() is instrumented to compare both counter approximations to the precise value, and to record the largest delta of each. The test runs in a KVM guest with 384 vCPUs, each pinned to its own CPU of the test hardware, with the NUMA topology of the host.
With the counters of this patch, before the single-threaded mode added later in the series, the upstream implementation counter approximation reaches an 830MB delta from the precise value within the first seconds, while the mappings accumulate, compared to a 106MB delta with the hierarchical counter. The hierarchical counter approximation is guaranteed to be within 158.5MB of the precise value in both directions on that hardware topology, compared to 1.1GB for the upstream implementation (384 CPUs * 767 pages).
* Fast path implementation comparison
The new percpu_counter_tree_add() updates a per-cpu byte with a this_cpu_try_cmpxchg() loop on the fast path, like the upstream percpu_counter_add_batch(). When the per-cpu residual reaches the batch size, it calls a slow path which "trickles up" the carry to upper level counters with atomic_add_return(), each of them keeping its own residual.
In comparison, the upstream percpu_counter_add_batch() takes a global raw_spin_lock_irqsave() when reaching the batch size, and flushes the whole per-cpu count into the global count.
The hierarchical implementation is therefore expected to have less contention on mid-sized allocations than the upstream counters because the atomic counters tracking those bits are each shared by a fraction of the CPUs.
* Benchmarks
Using will-it-scale page_fault1 benchmarks to compare the upstream counters to the hierarchical counters of this patch. This is done with hyperthreading disabled at runtime, so the counter tree keeps its shape for 384 logical CPUs, while the upstream batch size follows the 192 online ones (384 instead of 768). The differences are within the boot-to-boot spread of up to 3%, and the multi-threaded runs, bound by the mmap_lock, have a standard deviation of 5% to 11%: no regression is measured, and no gain is claimed. The numbers with the whole series are in patch 7 and the cover letter.
upstream hierarchical delta page_fault1_processes -s 100 -t 1 544932 557092 +2.2% page_fault1_threads -s 100 -t 1 547501 557742 +1.9% page_fault1_processes -s 100 -t 96 37374404 37580199 +0.6% page_fault1_threads -s 100 -t 96 2311884 2299998 -0.5% page_fault1_processes -s 100 -t 192 69491758 70590309 +1.6% page_fault1_threads -s 100 -t 192 2284108 2271423 -0.6%
* Memory Use
Each mm_struct holds NR_MM_COUNTERS (4) counters, initialized together by percpu_counter_tree_init_many() with a batch size of 32, so their tree items share cache lines. Per mm, on a 64-bit arch with 64-byte cache lines, compared to the upstream percpu counters, along with the maximum inaccuracy of each counter in pages:
Bytes per mm Accuracy (pages)
nr_cpus fan-out hpcc upstream hpcc upstream
2 2 200 192 +-62 +-62
16 16 256 416 +-496 +-496
64 16,4 704 1184 +-3904 +-8128
128 8,16 1728 2208 +-7552 +-32640
256 8,4,8 3776 4256 +-21248 +-130816
384 8,8,8 6336 6304 +-40576 +-294528
512 8,8,8 6848 8352 +-44544 +-523776
1024 8,8,16 13504 16544 +-89088 +-2096128
4096 8,8,8,8 53952 65696 +-471040 +-33550336This counts the 4 counter structures, their per-cpu data, and the tree items, without the rounding of the slab cache holding them. The fan-out is listed from the per-cpu counters to the root. Up to 32 CPUs, the tree is flat. Up to 16 CPUs, its accuracy matches the upstream counters, whose batch size of max(32, 2 * nr_cpus) grows beyond, making them less accurate than the hierarchical counters, except from 33 to 39 CPUs, where the intermediate counters of the 16,4 tree make the hierarchical counters up to 1.37 times less accurate.
The hierarchical counters use less memory than the upstream counters from 3 to 64 CPUs. The tree is shaped for nr_cpus rounded up to a power of 2: when nr_cpus is a power of 2, they also use less memory beyond 64 CPUs (at most 0.89 times as much). Between powers of 2, fewer CPUs share the cost of the tree items: 1.01 times at 384 logical CPUs, 1.09 at 192 and 0.94 at 96, and up to 1.47 times just above a power of 2 (at 129 CPUs, 1.23 at 65 and 1.36 at 257).
* mm_struct layout
percpu_counter_tree_init_many() allocates the per-cpu data and the tree items of the RSS counters, so their size in mm_struct does not depend on the number of CPUs: struct mm_struct shrinks by 32 bytes, as a struct percpu_counter_tree is 32 bytes, compared to 40 for a struct percpu_counter.
* rss_stat tracepoint
The rss_stat tracepoint keeps reporting the precise sum, now computed from the hierarchical per-cpu counter.
---
Changes since v22:
- The library allocates the per-cpu counters and the tree items of the RSS counters: mm_struct holds no tree items, and the static init_mm and efi_mm need no change.
- check_mm(): sum each RSS counter once.
- Keep reporting the precise sum in the rss_stat tracepoint. Switching it to the approximation is done by a separate patch.
- Initialize the 4 RSS counters together so their tree items share cache lines.
- Fix checkpatch warnings.
--- include/linux/mm.h | 10 +++++----- include/linux/mm_types.h | 3 ++- include/trace/events/kmem.h | 3 +-- kernel/fork.c | 13 +++++++++---- 4 files changed, 17 insertions(+), 12 deletions(-)
include/linux/mm.h +5 -5
@@ -3607,33 +3607,33 @@ static inline bool get_user_page_fast_only(unsigned long addr,*/ static inline unsigned long get_mm_counter (struct mm_struct *mm,int member) { -return percpu_counter_read_positive (&mm->rss_stat[member]); +return percpu_counter_tree_approximate_sum_positive (&mm->rss_stat[member]); }static inline unsigned long get_mm_counter_sum (struct mm_struct *mm,int member) { -return percpu_counter_sum_positive (&mm->rss_stat[member]); +return percpu_counter_tree_precise_sum_positive (&mm->rss_stat[member]); }void mm_trace_rss_stat (struct mm_struct *mm,int member);static inline void add_mm_counter (struct mm_struct *mm,int member,long value) { -percpu_counter_add (&mm->rss_stat[member], value); +percpu_counter_tree_add (&mm->rss_stat[member], value);mm_trace_rss_stat (mm, member); }static inline void inc_mm_counter (struct mm_struct *mm,int member) { -percpu_counter_inc (&mm->rss_stat[member]); +percpu_counter_tree_add (&mm->rss_stat[member],1 );mm_trace_rss_stat (mm, member); }static inline void dec_mm_counter (struct mm_struct *mm,int member) { -percpu_counter_dec (&mm->rss_stat[member]); +percpu_counter_tree_add (&mm->rss_stat[member],-1 );mm_trace_rss_stat (mm, member); }
include/linux/mm_types.h +2 -1
@@ -19,6 +19,7 @@#include <linux/workqueue.h> #include <linux/seqlock.h> #include <linux/percpu_counter.h> +#include <linux/percpu_counter_tree.h> #include <linux/types.h> #include <linux/futex_types.h> #include <linux/rseq_types.h>
@@ -1350,7 +1351,7 @@ struct mm_struct {unsigned long saved_e_flags;#endif -struct percpu_counter rss_stat[NR_MM_COUNTERS ]; +struct percpu_counter_tree rss_stat[NR_MM_COUNTERS ];struct linux_binfmt *binfmt;
include/trace/events/kmem.h +1 -2
@@ -448,8 +448,7 @@ TRACE_EVENT(rss_stat,*/ __entry->curr = current->mm == mm && !(current->flags &PF_KTHREAD ); __entry->member = member; - __entry->size = (percpu_counter_sum_positive (&mm->rss_stat[member]) - <<PAGE_SHIFT ); + __entry->size =get_mm_counter_sum (mm, member) <<PAGE_SHIFT ; ), TP_printk("mm_id=%u curr=%d type=%s size=%ldB",
kernel/fork.c +9 -4
@@ -137,6 +137,11 @@*/ #define MAX_THREADS FUTEX_TID_MASK +/* +* Batch size of rss stat approximation +*/ +#define RSS_STAT_BATCH_SIZE 32 +/* * Protected counters by write_lock_irq(&tasklist_lock) */
@@ -633,7 +638,7 @@ static void check_mm(struct mm_struct *mm) "Please make sure 'struct resident_page_types[]' is updated as well");for (i =0 ; i <NR_MM_COUNTERS ; i++) { -long x =percpu_counter_sum (&mm->rss_stat[i]); +long x =percpu_counter_tree_precise_sum (&mm->rss_stat[i]);if (unlikely (x)) { pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld Comm:%s Pid:%d\n",
@@ -738,7 +743,7 @@ void __mmdrop(struct mm_struct *mm)check_mm (mm);mm_pasid_drop (mm);mm_destroy_cid (mm); -percpu_counter_destroy_many (mm->rss_stat,NR_MM_COUNTERS ); +percpu_counter_tree_destroy_many (mm->rss_stat,NR_MM_COUNTERS );free_mm (mm); }
@@ -1143,8 +1148,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)if (mm_alloc_sched (mm)) goto fail_sched; -if (percpu_counter_init_many (mm->rss_stat,0 ,GFP_KERNEL_ACCOUNT , -NR_MM_COUNTERS )) +if (percpu_counter_tree_init_many (mm->rss_stat,NR_MM_COUNTERS ,RSS_STAT_BATCH_SIZE , +GFP_KERNEL_ACCOUNT )) goto fail_pcpu;lru_gen_init_mm (mm);
Signature · 1 line
2.47.3
Mathieu Desnoyers
reviewerRFC PATCH v23 4/10 tools/testing/hpcc: add userspace test of hierarchical per-CPU counters
In reply to Mathieu DesnoyersThe KUnit test only covers the topology of the machine it runs on. Add a userspace test which builds lib/percpu_counter_tree.c with a simulated topology, which allows testing every supported tree configuration on a single machine, following the approach of tools/testing/vma and tools/testing/radix-tree.
A small shim header provides the kernel environment needed by the library, on top of tools/include: per-CPU data is an array indexed by the simulated current CPU, and atomic operations are plain operations, because the test is single-threaded. It therefore only covers settled counters.
Validate the configuration table, then, for each simulated number of CPUs (1 to 65536, up to 1048576 with -a) and batch size (2 to 128):
- Saturate the counter upward and downward: leave every tree item with its largest positive (or negative) residual which has not been propagated yet, check that the distance between the precise and approximate sums matches the residual capacity of the tree, computed from the configuration table, which is exactly the accuracy range, the same in both directions, and check the accuracy range contract and the comparison functions in these worst-case states. A lost carry fails the saturation rather than hanging it.
- Random walk: apply random increments from random CPUs to two counters, and check the accuracy range contract and the comparison functions against shadow sums after each increment.
Along with the accuracy range contract, check the residuals of the tree items, and that no carry was lost or duplicated.
The comparison checks include the edges of the possible precise range:
the approximate comparison against a value must be definitive exactly outside of [approx - over, approx + under].
Usage:
make -C tools/testing/hpcc
tools/testing/hpcc/hpcc [-a]
--- MAINTAINERS | 1 + tools/testing/hpcc/.gitignore | 2 + tools/testing/hpcc/Makefile | 18 + tools/testing/hpcc/include/hpcc_shim.h | 177 ++++++++ tools/testing/hpcc/include/kunit/visibility.h | 2 + tools/testing/hpcc/include/linux/args.h | 4 + tools/testing/hpcc/include/linux/atomic.h | 2 + tools/testing/hpcc/include/linux/bug.h | 2 + tools/testing/hpcc/include/linux/cache.h | 2 + tools/testing/hpcc/include/linux/cpumask.h | 2 + tools/testing/hpcc/include/linux/errno.h | 3 + tools/testing/hpcc/include/linux/limits.h | 2 + tools/testing/hpcc/include/linux/math.h | 2 + tools/testing/hpcc/include/linux/percpu.h | 2 + .../hpcc/include/linux/percpu_counter_tree.h | 2 + tools/testing/hpcc/include/linux/preempt.h | 2 + tools/testing/hpcc/include/linux/slab.h | 2 + tools/testing/hpcc/main.c | 413 ++++++++++++++++++ 18 files changed, 640 insertions(+) create mode 100644 tools/testing/hpcc/.gitignore create mode 100644 tools/testing/hpcc/Makefile create mode 100644 tools/testing/hpcc/include/hpcc_shim.h create mode 100644 tools/testing/hpcc/include/kunit/visibility.h create mode 100644 tools/testing/hpcc/include/linux/args.h create mode 100644 tools/testing/hpcc/include/linux/atomic.h create mode 100644 tools/testing/hpcc/include/linux/bug.h create mode 100644 tools/testing/hpcc/include/linux/cache.h create mode 100644 tools/testing/hpcc/include/linux/cpumask.h create mode 100644 tools/testing/hpcc/include/linux/errno.h create mode 100644 tools/testing/hpcc/include/linux/limits.h create mode 100644 tools/testing/hpcc/include/linux/math.h create mode 100644 tools/testing/hpcc/include/linux/percpu.h create mode 100644 tools/testing/hpcc/include/linux/percpu_counter_tree.h create mode 100644 tools/testing/hpcc/include/linux/preempt.h create mode 100644 tools/testing/hpcc/include/linux/slab.h create mode 100644 tools/testing/hpcc/main.c
MAINTAINERS +1 -0
@@ -11956,6 +11956,7 @@ F: Documentation/core-api/percpu-counter-tree.rst F: include/linux/percpu_counter_tree.h F: lib/percpu_counter_tree.c F: lib/tests/percpu_counter_tree_kunit.c +F: tools/testing/hpcc/ HIGH-RESOLUTION TIMERS, TIMER WHEEL, CLOCKEVENTS M: Anna-Maria Behnsen <[email protected]>
tools/testing/hpcc/.gitignore new file +2 -0
@@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0+ OR MIT +hpcc
tools/testing/hpcc/Makefile new file +18 -0
@@ -0,0 +1,18 @@ +# SPDX-License-Identifier: GPL-2.0+ OR MIT + +.PHONY: default clean + +default: hpcc + +CFLAGS += -O2 -g -Wall -I./include -I../../include -fno-strict-overflow -fno-strict-aliasing +CFLAGS += -DCONFIG_SMP -DCONFIG_KUNIT=1 -D'NR_CPUS=(1U << 20)' + +DEPS = ../../../lib/percpu_counter_tree.c \ + ../../../include/linux/percpu_counter_tree.h \ + include/hpcc_shim.h + +hpcc: main.c $(DEPS) + $(CC) $(CFLAGS) -o $@ main.c $(LDLIBS) + +clean: + $(RM) hpcc
tools/testing/hpcc/include/hpcc_shim.h new file +177 -0
177 lines not on this page · raw message
tools/testing/hpcc/include/kunit/visibility.h new file +2 -0
@@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +#define EXPORT_SYMBOL_IF_KUNIT (symbol)
tools/testing/hpcc/include/linux/args.h new file +4 -0
@@ -0,0 +1,4 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +/* Also defined by the C library <sys/cdefs.h>. */ +#undef __CONCAT +#include "../../../../../include/linux/args.h"
tools/testing/hpcc/include/linux/atomic.h new file +2 -0
@@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +#include "../hpcc_shim.h"
tools/testing/hpcc/include/linux/bug.h new file +2 -0
@@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +#include "../hpcc_shim.h"
tools/testing/hpcc/include/linux/cache.h new file +2 -0
@@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +#include "../hpcc_shim.h"
tools/testing/hpcc/include/linux/cpumask.h new file +2 -0
@@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +#include "../hpcc_shim.h"
tools/testing/hpcc/include/linux/errno.h new file +3 -0
@@ -0,0 +1,3 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +/* Also included by the C library <errno.h>: only provide the error numbers. */ +#include <asm/errno.h>
tools/testing/hpcc/include/linux/limits.h new file +2 -0
@@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +#include "../hpcc_shim.h"
tools/testing/hpcc/include/linux/math.h new file +2 -0
@@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +#include "../hpcc_shim.h"
tools/testing/hpcc/include/linux/percpu.h new file +2 -0
@@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +#include "../hpcc_shim.h"
tools/testing/hpcc/include/linux/percpu_counter_tree.h new file +2 -0
@@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +#include "../../../../../include/linux/percpu_counter_tree.h"
tools/testing/hpcc/include/linux/preempt.h new file +2 -0
@@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +#include "../hpcc_shim.h"
tools/testing/hpcc/include/linux/slab.h new file +2 -0
@@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +#include "../hpcc_shim.h"
tools/testing/hpcc/main.c new file +413 -0
413 lines not on this page · raw message
Signature · 1 line
2.47.3
[email protected]
AI reviewRe: [RFC PATCH v23 04/10] tools/testing/hpcc: add userspace test of hierarchical per-CPU counters
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] tools/testing/hpcc: counter leak in test_saturated() and test_random_walk() --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] tools/testing/hpcc: counter leak in test_saturated() and test_random_walk() --
commit 503d613a781c32158eec55568b3137245d93fe15
Author: Mathieu Desnoyers <[email protected]>
tools/testing/hpcc: add userspace test of hierarchical per-CPU counters
This commit adds a single-threaded userspace test harness under tools/testing/hpcc that builds lib/percpu_counter_tree.c with a simulated topology to verify counter saturation, random walks, accuracy range contracts, and comparison functions across all supported CPU counts and batch sizes.
Quoted 6 lines
tools/testing/hpcc/main.c new file +0
-0@@ -0,0 +1,413 @@
[ ... ]
Quoted 11 lines
tools/testing/hpcc/main.c new file +11
-0+static void test_saturated (unsigned long batch_size,long inc) +{ +struct test_counter c, probe, exact; +unsigned long accuracy; +long approx, delta; + +if (test_counter_init (&c, batch_size) ||test_counter_init (&probe, batch_size) || +test_counter_init (&exact, batch_size)) { +CHECK (0 ); +return ; + }
[ ... ]
tools/testing/hpcc/main.c line 311
Quoted 9 lines
tools/testing/hpcc/main.c new file +9
-0+static void test_random_walk (unsigned long batch_size,unsigned long nr_steps) +{ +struct test_counter c[2 ]; +unsigned long step; + +if (test_counter_init (&c[0 ], batch_size) ||test_counter_init (&c[1 ], batch_size)) { +CHECK (0 ); +return ; + }
[Severity: Medium]
Can these error paths in test_saturated() and test_random_walk() leak the already-initialized counters when a subsequent test_counter_init() call fails?
Each test_counter_init() call allocates per-CPU level0 storage and tree items via percpu_counter_tree_init(). In test_saturated(), if test_counter_init(&c, batch_size) succeeds and test_counter_init(&probe, batch_size) or test_counter_init(&exact, batch_size) fails, the function records CHECK(0) and returns to run_case() without calling test_counter_destroy() on c (or probe). Similarly, in test_random_walk(), if test_counter_init(&c[0], batch_size) succeeds and test_counter_init(&c[1], batch_size) fails, c[0] is not destroyed before returning.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Mathieu Desnoyers
reviewerRFC PATCH v23 5/10 mm: report the approximate RSS in the rss_stat tracepoint
In reply to Mathieu DesnoyersThe rss_stat tracepoint is emitted on every RSS update, including each page fault. When it is enabled, it reports the precise sum of the RSS counter, which iterates over all possible CPUs for each event: 384 per page fault on a machine with 384 logical CPUs.
Report the approximate sum of the hierarchical per-cpu counter instead, through get_mm_counter(), which reads the root of the counter tree. The size field of the event, the RSS counter after the update, is therefore within the accuracy bound of the counter from the precise value: within 158.5MB on a machine with 384 logical CPUs and 4kB pages.
The tracepoint reports the precise sum since commit f1a7941243c1 ("mm:
convert mm's rss stats into percpu_counter"). Before that, it reported an atomic counter which only received the per-task cached updates in batches.
---
Changes since v22:
- Split from "mm: improve RSS counter approximation accuracy for proc interfaces", which now keeps reporting the precise sum.
--- include/trace/events/kmem.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/trace/events/kmem.h +1 -1
@@ -448,7 +448,7 @@ TRACE_EVENT(rss_stat,*/ __entry->curr = current->mm == mm && !(current->flags &PF_KTHREAD ); __entry->member = member; - __entry->size =get_mm_counter_sum (mm, member) <<PAGE_SHIFT ; + __entry->size =get_mm_counter (mm, member) <<PAGE_SHIFT ; ), TP_printk("mm_id=%u curr=%d type=%s size=%ldB",
Signature · 1 line
2.47.3
Mathieu Desnoyers
reviewerRFC PATCH v23 6/10 lib: percpu_counter_tree: add lazy tree activation
In reply to Mathieu DesnoyersAdd counters which start without their tree, for counters which may never need its scalability, such as the RSS counters of a single-threaded process.
percpu_counter_tree_init_many_lazy() initializes counters without allocating their per-cpu counters and tree items. Until percpu_counter_tree_activate_many() allocates and publishes them, the counters are exact:
- percpu_counter_tree_add_owner() adds to the bias of the counter, without atomic instruction. Its calls for a given counter must not run concurrently with each other: they come from a single context, the owner chosen by the user of the counters, such as a task.
- percpu_counter_tree_add() adds atomically to a new atomic bias of the counter, for any other updater.
The precise and approximate sums read both biases, in O(1). The activation does not move these values: the sums keep reading them along with the tree, so updates racing with the activation are not lost, whichever side of it they land on. The bias only has the owner as writer, and the atomic bias only receives atomic additions, before and after the activation. After the activation, both update functions add to the tree, and percpu_counter_tree_active() reports it, so a user can skip selecting between them. The accuracy range of a lazily initialized counter is the range of its tree, even before activation, which keeps the comparison functions correct across an activation.
The activations of the same counters must be serialized by the caller, for instance with a mutex: a concurrent activation would publish a second tree. The activation publishes the root, then the per-cpu counters, of each counter in order, with release ordering. Once the last counter is active, the activation is complete, so a caller which sees it active can skip the activation and its serialization; it may not observe the other counters active yet, which only delays their updates going to the tree. The update functions only reach the per-cpu counters through their published pointer, which orders their initialization before their use by address dependency, as for rcu_dereference(), without an acquire on the fast path. The approximate sum reaches the root the same way. The carry path reads the root pointer after the per-cpu counters pointer with a read barrier between them, and the precise sum reads the per-cpu counters pointer with an acquire.
struct percpu_counter_tree grows from 32 to 40 bytes with the atomic bias, the size of struct percpu_counter.
Test the lazy counters:
- KUnit: exact sums before activation with owner and atomic updates, set(), idempotent activation, destruction without activation, and an activation racing with an owner and remote updaters on other CPUs, which also activate the tree when released, serialized by a mutex, checking that the counter is exact while they pause, that the tree is active once an activation returns, and that no update is lost across the activations.
- Userspace test: random owner and atomic updates from random CPUs, exact before the tree is activated halfway, with the accuracy and comparison checks after each update. The test checks that the activation allocates one tree, that activating again allocates nothing, and that destroying the counter frees the tree. The shim provides the ordering primitives as plain accesses.
--- .../core-api/percpu-counter-tree.rst | 28 ++ include/linux/percpu_counter_tree.h | 81 ++++- lib/percpu_counter_tree.c | 315 ++++++++++++++++-- lib/tests/percpu_counter_tree_kunit.c | 218 ++++++++++++ tools/testing/hpcc/include/hpcc_shim.h | 28 +- tools/testing/hpcc/main.c | 47 +++ 6 files changed, 674 insertions(+), 43 deletions(-)
Documentation/core-api/percpu-counter-tree.rst +28 -0
@@ -74,6 +74,34 @@ takes no lock: an update which carries from the per-cpu counter to the upper levels can be counted in part, the residual it left below read and its carry missed on the way up. +Lazy Tree Activation +==================== + +Counters initialized with percpu_counter_tree_init_many_lazy() start +without their per-cpu counters and tree items, for counters which may +never need the scalability of the tree. Until +percpu_counter_tree_activate_many() allocates and publishes the tree, +the counters are exact: + + * percpu_counter_tree_add_owner() adds to the bias of the counter, + written only by its owner, a single context chosen by the user of the + counters, such as a task, without atomic instruction. + + * percpu_counter_tree_add() adds to the atomic bias of the counter, for + any other updater. + +The precise and approximate sums read both, in O(1), and count each +update whole. The activation does not move these values: the sums keep +reading them along with the tree, so updates racing with the activation +are not lost, whichever side of it they land on. The bias only has the +owner as writer, and the atomic bias only receives atomic additions, +before and after the activation. After activation, both update functions +add to the tree, and the approximation has the accuracy of the tree. + +The activations of the same counters must be serialized by the caller: +a concurrent activation would publish a second tree and lose the +updates made to the first. + Use Cases =========
include/linux/percpu_counter_tree.h +77 -4
124 lines not on this page · raw message
lib/percpu_counter_tree.c +279 -36
443 lines not on this page · raw message
lib/tests/percpu_counter_tree_kunit.c +218 -0
236 lines not on this page · raw message
tools/testing/hpcc/include/hpcc_shim.h +25 -3
@@ -68,6 +68,9 @@ static inline int get_count_order(unsigned int count)#define atomic_long_add_return_relaxed (i, v) ((v)->counter += (i))#define atomic_long_add_return_release (i, v) ((v)->counter += (i))#define smp_rmb () barrier() +/* The harness runs on one thread: ordering primitives are plain accesses. */ +#define smp_load_acquire (p) READ_ONCE(*(p)) +#define smp_store_release (p, v) WRITE_ONCE(*(p), (v))static inline bool atomic_long_try_cmpxchg_relaxed (atomic_long_t *v,long *old,long new) {
@@ -92,15 +95,32 @@ extern unsigned int hpcc_current_cpu;*/ #define HPCC_PCPU_UNIT 64 +/* Outstanding allocations, to check for leaks. */ +extern long hpcc_nr_allocs; + +static inline void *hpcc_alloc_account (void *p) +{ +if (p) + hpcc_nr_allocs++; +return p; +} + +static inline void hpcc_free_account (void *p) +{ +if (p) + hpcc_nr_allocs--; +} +static inline void *__alloc_percpu_gfp (size_t size,size_t align,gfp_t gfp) {if (size >HPCC_PCPU_UNIT || align >HPCC_PCPU_UNIT )return NULL ; -return calloc (nr_cpu_ids,HPCC_PCPU_UNIT ); +return hpcc_alloc_account (calloc (nr_cpu_ids,HPCC_PCPU_UNIT )); }static inline void free_percpu (void *ptr) { +hpcc_free_account (ptr);free (ptr); }
@@ -136,21 +156,23 @@ static inline void *kmem_cache_zalloc(struct kmem_cache *s, gfp_t gfp)if (p)memset (p,0 , s->size); -return p; +return hpcc_alloc_account (p); }static inline void kmem_cache_free (struct kmem_cache *s,void *p) { +hpcc_free_account (p);free (p); }static inline void *kzalloc (size_t size,gfp_t gfp) { -return calloc (1 , size); +return hpcc_alloc_account (calloc (1 , size)); }static inline void kfree (void *p) { +hpcc_free_account (p);free (p); }
tools/testing/hpcc/main.c +47 -0
71 lines not on this page · raw message
Signature · 1 line
2.47.3
Mathieu Desnoyers
reviewerRFC PATCH v23 7/10 mm: keep the RSS counters of single-threaded processes exact
In reply to Mathieu DesnoyersUntil a second thread shares the mm, initialize the RSS counters lazily, without their counter tree: the task owning the mm adds to a plain counter, and other tasks, such as reclaim, khugepaged, the OOM reaper or remote GUP, add atomically. The RSS counters of single-threaded processes are therefore exact, their precise sums for /proc/PID/status, /proc/PID/statm and the OOM killer take O(1), and they use no per-cpu memory. The readers of the approximation, such as /proc/PID/stat, the RSS high watermark and the rss_stat tracepoint, get the exact value for these processes.
The owner, mm->rss_owner, is the task running on the mm: the child in dup_mm(), or the task in exec_mmap(), which also clears it from the mm it leaves, as exit_mm() does. It stays NULL, which matches no task, in the mm of exec() until exec_mmap() and in those of the other users of mm_alloc(): all their updates are atomic. It is only compared with current, never dereferenced, so a single task writes the plain counter, including while a vfork() child shares the mm. mm->owner (CONFIG_MEMCG) is not used, because mm_update_next_owner() hands it over to another task while a vfork() child may still update the counters.
copy_mm() activates the counter tree when a thread is created with CLONE_VM, except for vfork(), whose parent waits while the child shares the mm. mm->rss_stat_mutex serializes the activations, as mm->mm_cid.mutex serializes the mode changes of mm_cid at fork and exit, without relying on which tasks can create threads. Once the tree is active, thread creation no longer takes the mutex. This moves the allocation of the per-cpu counters and tree items of the RSS counters from fork() to the creation of the second thread of a process, which returns -ENOMEM if it fails, as fork() could before. The tasks the kernel creates with CLONE_VM for a process, such as the io_uring workers and the vhost tasks, also share its mm: a single-threaded process using them gets the counter tree. The updates made before the activation stay in the single counters, which the sums keep reading, so updates racing with the activation are not lost. Once the tree is active, __add_mm_counter() no longer reads rss_owner, so the updates of the threads of a process do not pull in its cache line, which other mm_struct fields share.
The tree stays active when the process returns to a single thread:
going back to the single counters would need a grace period for the remote updaters, and a process alternating between one and two threads would pay for each switch.
Before the activation, the updates of tasks other than the owner are atomic additions to one word per counter: concurrent remote updates of the same single-threaded process from many CPUs, such as direct reclaim unmapping its pages, contend on it.
* Testing results:
Test hardware: 2 sockets AMD EPYC 9654 96-Core Processor (384 logical CPUs).
With the test program and the instrumentation described in the patch switching the RSS counters to hierarchical counters, in a KVM guest with 384 vCPUs, each pinned to its own CPU of the host, with the NUMA topology of the host, the approximation of the RSS counters of the single-threaded program is exact, compared to an 841MB delta for the upstream counters. With an idle second thread, which activates the counter tree, the hierarchical counters reach a 96MB delta, compared to 845MB for the upstream counters.
On bare metal, with 1000 to 16000 single-threaded processes alive, a read of /proc/PID/statm, which computes 3 precise sums, takes 5.3 to 6.5us, compared to 16.5 to 19.0us with the upstream counters, and a read of /proc/PID/status, which computes 4 precise sums, takes 11.4 to 13.1us, compared to 26.6 to 28.6us. A fork + exit of a child touching no pages takes 116us, compared to 139us (219us compared to 247us when it touches 64 pages): its counters are destroyed without 4 precise sums over 384 logical CPUs in check_mm(), and neither per-cpu memory nor tree items are allocated or freed.
On bare metal with SMT disabled at runtime, where the upstream batch size is 384, will-it-scale page_fault2 with one process is 5.1% faster than with the upstream counters, and page_fault3 1.3% faster (one boot each, the upstream one on the previous day; the boot-to-boot spread reaches 3%, so neither is claimed as a gain).
* Memory Use
Per mm, on a 64-bit arch with 64-byte cache lines, including the rss_owner pointer and the 24-byte activation mutex, for a single-threaded and a multi-threaded process, compared to the upstream percpu counters:
Bytes per mm
nr_cpus fan-out 1 thread threads upstream
2 2 192 264 192
16 16 192 320 416
64 16,4 192 768 1184
384 8,8,8 192 6400 6304
1024 8,8,16 192 13568 16544A single-threaded process uses 192 bytes for its RSS counters whatever the number of CPUs: the 4 counter structures, rss_owner and the mutex in mm_struct, compared to the 4 upstream counter structures (160 bytes) and their per-cpu memory. On x86_64 with 16, 384 and 1024 possible CPUs, the mm_struct slab object keeps its upstream size (1752, 1872 and 2112 bytes in /proc/slabinfo). A multi-threaded process adds nr_cpus bytes of per-cpu memory per counter (1536 per mm at 384 logical CPUs), compared to 4 * nr_cpus bytes per counter (6144 per mm) for the upstream counters, and the tree items, without the rounding of their slab cache. fork() no longer allocates memory for the RSS counters.
--- .../core-api/percpu-counter-tree.rst | 7 ++++ fs/exec.c | 5 +++ include/linux/mm.h | 24 +++++++++++-- include/linux/mm_types.h | 16 +++++++++ kernel/exit.c | 3 ++ kernel/fork.c | 35 +++++++++++++++++-- 6 files changed, 85 insertions(+), 5 deletions(-)
Documentation/core-api/percpu-counter-tree.rst +7 -0
@@ -102,6 +102,13 @@ The activations of the same counters must be serialized by the caller: a concurrent activation would publish a second tree and lose the updates made to the first. +The RSS counters of an mm use this: the task owning the mm updates the +plain counter, and other tasks, such as reclaim, update the atomic +counter. The tree is activated when a second thread shares the mm, +under mm->rss_stat_mutex, which serializes the activations. +A single-threaded process therefore has exact RSS counters, cheap +precise sums, and neither per-cpu memory nor tree items for them. + Use Cases =========
fs/exec.c +5 -0
@@ -875,6 +875,8 @@ static int exec_mmap(struct linux_binprm *bprm) } } +/* This task owns the RSS counters of the new mm, see __add_mm_counter(). */ +WRITE_ONCE (mm->rss_owner, tsk);task_lock (tsk);membarrier_exec_mmap (mm);
@@ -903,6 +905,9 @@ static int exec_mmap(struct linux_binprm *bprm)if (old_mm) {mmap_read_unlock (old_mm);BUG_ON (active_mm != old_mm); +/* Updates of the old mm by this task are now atomic. */ +if (READ_ONCE (old_mm->rss_owner) == tsk) +WRITE_ONCE (old_mm->rss_owner,NULL );/* Defer teardown to setup_new_exec(), outside the exec locks. */ bprm->old_mm = old_mm; }else {
include/linux/mm.h +21 -3
@@ -3617,23 +3617,41 @@ static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)void mm_trace_rss_stat (struct mm_struct *mm,int member); +/* +* Until a second thread shares the mm, the RSS counters are exact single +* counters: the task which owns the mm updates a plain counter, and other +* tasks (reclaim, khugepaged, the OOM reaper, remote GUP, ...) an atomic +* one. See copy_mm(). Once the tree is active, all updates go to it: +* rss_owner is not read, so the updates of the threads of a process do not +* touch its cache line. +*/ +static inline void __add_mm_counter (struct mm_struct *mm,int member,long value) +{ +struct percpu_counter_tree *counter = &mm->rss_stat[member]; + +if (!percpu_counter_tree_active (counter) && current ==READ_ONCE (mm->rss_owner)) +percpu_counter_tree_add_owner (counter, value); +else +percpu_counter_tree_add (counter, value); +} +static inline void add_mm_counter (struct mm_struct *mm,int member,long value) { -percpu_counter_tree_add (&mm->rss_stat[member], value); +__add_mm_counter (mm, member, value);mm_trace_rss_stat (mm, member); }static inline void inc_mm_counter (struct mm_struct *mm,int member) { -percpu_counter_tree_add (&mm->rss_stat[member],1 ); +__add_mm_counter (mm, member,1 );mm_trace_rss_stat (mm, member); }static inline void dec_mm_counter (struct mm_struct *mm,int member) { -percpu_counter_tree_add (&mm->rss_stat[member],-1 ); +__add_mm_counter (mm, member,-1 );mm_trace_rss_stat (mm, member); }
include/linux/mm_types.h +16 -0
@@ -1352,6 +1352,22 @@ struct mm_struct {#endif struct percpu_counter_tree rss_stat[NR_MM_COUNTERS ]; +/* +* Task updating rss_stat with percpu_counter_tree_add_owner() +* until the counter tree is activated by a second thread: the +* task running on this mm, set by dup_mm() and exec_mmap(), +* cleared when that task exits or execs, NULL for the other +* users of mm_alloc(). Only compared with current, never +* dereferenced. Not mm->owner, which depends on CONFIG_MEMCG +* and which mm_update_next_owner() hands over to another task +* while a vfork() child may still update the counters. +*/ +struct task_struct *rss_owner; +/* +* Serializes the activations of the rss_stat counter tree, +* until the tree is active. See mm_activate_rss_stat(). +*/ +struct mutex rss_stat_mutex;struct linux_binfmt *binfmt;
kernel/exit.c +3 -0
@@ -582,6 +582,9 @@ static void exit_mm(void)local_irq_enable ();task_unlock (current);mmap_read_unlock (mm); +/* From now on, this task updates the RSS counters of @mm atomically. */ +if (READ_ONCE (mm->rss_owner) == current) +WRITE_ONCE (mm->rss_owner,NULL );mm_update_next_owner (mm);mmput (mm);if (test_thread_flag (TIF_MEMDIE ))
kernel/fork.c +33 -2
@@ -1108,6 +1108,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)memset (&mm->rss_stat,0 ,sizeof (mm->rss_stat));spin_lock_init (&mm->page_table_lock);spin_lock_init (&mm->arg_lock); +mutex_init (&mm->rss_stat_mutex);mm_init_cpumask (mm);mm_init_aio (mm);mm_init_owner (mm, p);
@@ -1148,8 +1149,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)if (mm_alloc_sched (mm)) goto fail_sched; -if (percpu_counter_tree_init_many (mm->rss_stat,NR_MM_COUNTERS ,RSS_STAT_BATCH_SIZE , -GFP_KERNEL_ACCOUNT )) +if (percpu_counter_tree_init_many_lazy (mm->rss_stat,NR_MM_COUNTERS ,RSS_STAT_BATCH_SIZE )) goto fail_pcpu;lru_gen_init_mm (mm);
@@ -1542,6 +1542,8 @@ static struct mm_struct *dup_mm(struct task_struct *tsk,if (!mm_init (mm, tsk)) goto fail_nomem; +/* The child owns its RSS counters, see __add_mm_counter(). */ + mm->rss_owner = tsk;uprobe_start_dup_mmap (); err =dup_mmap (mm, oldmm);
@@ -1569,6 +1571,23 @@ static struct mm_struct *dup_mm(struct task_struct *tsk,return NULL ; } +/* +* Serialize the activations of the RSS counter tree, whichever tasks create +* threads. Once the last counter is active, the activation is complete and +* thread creation skips the mutex. That check needs no acquire: the updates +* and the sums of the counters order their own accesses to the tree. Not +* observing the other counters active yet only delays their updates going +* to the tree. +*/ +static int mm_activate_rss_stat (struct mm_struct *mm) +{ +if (percpu_counter_tree_active (&mm->rss_stat[NR_MM_COUNTERS -1 ])) +return 0 ; +guard (mutex)(&mm->rss_stat_mutex); +return percpu_counter_tree_activate_many (mm->rss_stat,NR_MM_COUNTERS , +GFP_KERNEL_ACCOUNT ); +} +static int copy_mm (u64 clone_flags,struct task_struct *tsk) {struct mm_struct *mm, *oldmm;
@@ -1593,6 +1612,18 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk)return 0 ;if (clone_flags &CLONE_VM ) { +int err =0 ; + +/* +* A second thread shares the mm: activate the RSS counter +* tree. A vfork child shares the mm while the parent waits, +* so the single counters still fit. The tree stays active if +* the clone fails later, which is harmless. +*/ +if (!(clone_flags &CLONE_VFORK )) + err =mm_activate_rss_stat (oldmm); +if (err) +return err;mmget (oldmm); mm = oldmm; }else {
Signature · 1 line
2.47.3
Mathieu Desnoyers
reviewerRFC PATCH v23 8/10 lib: percpu_counter_tree: add the arity boot parameter
In reply to Mathieu DesnoyersThe fan-out of each tree level is selected at boot from nr_cpu_ids, the highest possible CPU number plus one, using a table tuned for the RSS counters on one machine, a 2-socket AMD EPYC 9654 with 192 cores and 384 logical CPUs. The best shape depends on the cost of cache line transfers on a given machine, which varies with the number of sockets, the cache topology and SMT, so it may differ on other hardware.
Add the percpu_counter_tree.arity= boot parameter, which overrides the tree shape with the fan-out of each level, from the per-cpu counters to the root, so the shapes can be compared on other machines, to improve the default table. For example, when nr_cpu_ids is 16, "16" is a flat tree and "4,4" a tree with 4 intermediate counters. Each fan-out is a power of 2, and their product must be nr_cpu_ids rounded up to a power of 2. An invalid shape is ignored with a warning.
The shape in use is printed at boot, and can be read from /sys/module/percpu_counter_tree/parameters/arity.
Cover the parsing of the parameter in the userspace test, with valid and invalid shapes, and run the counter tests on the valid ones.
--- .../admin-guide/kernel-parameters.txt | 14 +++ .../core-api/percpu-counter-tree.rst | 3 +- lib/percpu_counter_tree.c | 96 ++++++++++++++++++- tools/testing/hpcc/include/hpcc_shim.h | 4 + .../testing/hpcc/include/linux/moduleparam.h | 12 +++ tools/testing/hpcc/main.c | 78 ++++++++++++++- 6 files changed, 201 insertions(+), 6 deletions(-) create mode 100644 tools/testing/hpcc/include/linux/moduleparam.h
Documentation/admin-guide/kernel-parameters.txt +14 -0
@@ -5342,6 +5342,20 @@ Kernel parameters allocator. This parameter is primarily for debugging and performance comparison. + percpu_counter_tree.arity= [KNL,SMP] + Format: <fan-out>[,<fan-out>...] + Override the shape of the hierarchical per-cpu + counter trees (used for the RSS counters), listing + the fan-out of each tree level from the per-cpu + counters to the root. Each fan-out is a power of 2 + and their product must be nr_cpu_ids (the highest + possible CPU number plus one) rounded up to a power + of 2, e.g. "16" (flat) or "4,4" when nr_cpu_ids + is 16. An invalid shape is ignored with a warning. + The shape in use is shown in + /sys/module/percpu_counter_tree/parameters/arity. + This parameter is meant for performance comparison. + pirq= [SMP,APIC] Manual mp-table setup See Documentation/arch/x86/i386/IO-APIC.rst.
Documentation/core-api/percpu-counter-tree.rst +2 -1
@@ -63,7 +63,8 @@ possible CPU, and the tree items, whose number depends on nr_cpu_ids, so struct percpu_counter_tree has a fixed size. The fan-out of each tree level is selected at boot from nr_cpu_ids, the -highest possible CPU number plus one. +highest possible CPU number plus one. It can be overridden with the +percpu_counter_tree.arity= boot parameter for performance comparison. The bounded accuracy is a property of the settled tree. While updates are in flight the transient error can exceed it, bounded by the number of
lib/percpu_counter_tree.c +92 -4
146 lines not on this page · raw message
tools/testing/hpcc/include/hpcc_shim.h +4 -0
@@ -34,8 +34,12 @@#define abs (x) ({ typeof(x) __x = (x); __x < 0 ? -__x : __x; })#define pr_err (...) fprintf(stderr, __VA_ARGS__) +#define pr_warn (...) fprintf(stderr, __VA_ARGS__)#define pr_info (...) do { if (0) printf(__VA_ARGS__); } while (0) +/* Implemented in main.c. */ +char *get_options (const char *str,int nints,int *ints); +extern unsigned long hpcc_nr_warnings;#define WARN (cond, ...) ({ \
tools/testing/hpcc/include/linux/moduleparam.h new file +12 -0
@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +#include "../hpcc_shim.h" + +struct kernel_param ; + +struct kernel_param_ops { +int (*set)(const char *val,const struct kernel_param *kp); +int (*get)(char *buffer,const struct kernel_param *kp); +}; + +#define module_param_cb (name, ops, arg, perm) +#define MODULE_PARM_DESC (parm, desc)
tools/testing/hpcc/main.c +77 -1
101 lines not on this page · raw message
Signature · 1 line
2.47.3
Mathieu Desnoyers
reviewerRFC PATCH v23 9/10 mm: add the rss_stat_batch boot parameter
In reply to Mathieu DesnoyersThe RSS counters use a batch size of 32: each CPU accumulates up to 31 pages in either direction before propagating a carry into the counter tree. A smaller batch improves the accuracy of the approximate RSS, and a larger batch reduces the cost of updates.
Add the rss_stat_batch= boot parameter to compare batch sizes on a given machine, along with the percpu_counter_tree.arity= boot parameter for the tree shape. It accepts a power of 2 between 2 and 128, and is ignored with a warning otherwise.
--- .../admin-guide/kernel-parameters.txt | 10 ++++++++ kernel/fork.c | 23 ++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +10 -0
@@ -6792,6 +6792,16 @@ Kernel parameters debug mode. Defaults to CONFIG_RSEQ_DEBUG_DEFAULT_ENABLE. Format: <bool> + rss_stat_batch= [KNL,SMP] + Format: <integer> + Batch size of the per-mm RSS counters: the number of + pages a CPU accumulates before propagating them into + the counter tree. A power of 2 between 2 and 128. + A smaller batch improves the accuracy of the + approximate RSS, a larger batch reduces the update + cost. Default: 32. + This parameter is meant for performance comparison. + rt_group_sched= [KNL] Enable or disable SCHED_RR/FIFO group scheduling when CONFIG_RT_GROUP_SCHED=y. Defaults to !CONFIG_RT_GROUP_SCHED_DEFAULT_DISABLED.
kernel/fork.c +20 -3
@@ -138,9 +138,26 @@#define MAX_THREADS FUTEX_TID_MASK/* -* Batch size of rss stat approximation +* Batch size of rss stat approximation, tunable with the rss_stat_batch= +* boot parameter to compare batch sizes on a given machine. */ -#define RSS_STAT_BATCH_SIZE 32 +static unsigned int rss_stat_batch_size__ro_after_init =32 ; + +static int __init setup_rss_stat_batch (char *str) +{ +unsigned int batch; + +if (kstrtouint (str,0 , &batch) || batch <2 || !is_power_of_2 (batch) || + batch >PERCPU_COUNTER_TREE_MAX_BATCH ) { +pr_warn ("rss_stat_batch: expecting a power of 2 between 2 and %u\n" , +PERCPU_COUNTER_TREE_MAX_BATCH ); +return 1 ; + } + rss_stat_batch_size = batch; +pr_info ("rss_stat_batch: %u\n" , batch); +return 1 ; +} +__setup ("rss_stat_batch=" , setup_rss_stat_batch);/* * Protected counters by write_lock_irq(&tasklist_lock)
@@ -1149,7 +1166,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)if (mm_alloc_sched (mm)) goto fail_sched; -if (percpu_counter_tree_init_many_lazy (mm->rss_stat,NR_MM_COUNTERS ,RSS_STAT_BATCH_SIZE )) +if (percpu_counter_tree_init_many_lazy (mm->rss_stat,NR_MM_COUNTERS , rss_stat_batch_size)) goto fail_pcpu;lru_gen_init_mm (mm);
Signature · 1 line
2.47.3
Mathieu Desnoyers
reviewerRFC PATCH v23 10/10 lib: add a benchmark of hierarchical per-cpu counters
In reply to Mathieu DesnoyersAdd a KUnit benchmark comparing the hierarchical per-cpu counters with struct percpu_counter, so their performance can be compared on various machines:
- update: cost of an update with a kthread on each online CPU, on one shared counter, on one counter per thread, on groups of 4 counters initialized together like the RSS counters of an mm, and on one lazily initialized counter per thread updated by its owner, like a single-threaded process, with increments of +1, alternating +1/-1, and random increments up to 32 and 512.
- lifecycle: cost of initializing and destroying 4 counters together, like fork and exit of an mm, eagerly or lazily, from one thread or from all online CPUs.
- churn: like lifecycle, with a few updates, keeping some of the counters alive until the end, like long-lived processes.
The batch size of the hierarchical counters is selected with the percpu_counter_tree_bench.batch= parameter (32 by default, like the RSS counters), and the tree shape with the percpu_counter_tree.arity= boot parameter. The struct percpu_counter variants use their default batch size, like the RSS counters before the hierarchical counters.
The benchmark reports the accuracy of both counter implementations along with the timings. It runs for a few minutes on large machines, so it has its own config option, CONFIG_PERCPU_COUNTER_TREE_BENCH.
For example:
./tools/testing/kunit/kunit.py run --arch=x86_64 \ --kconfig_add CONFIG_PERCPU_COUNTER_TREE_BENCH=y \ --kconfig_add CONFIG_NR_CPUS=256 --qemu_args="-smp 64 -m 8192" \ --kernel_args percpu_counter_tree.arity=8,8 \ --kernel_args percpu_counter_tree_bench.batch=16 \ --timeout 3000 percpu_counter_tree_bench
or, on a running kernel with CONFIG_PERCPU_COUNTER_TREE_BENCH=m:
modprobe percpu_counter_tree_bench batch=16
--- MAINTAINERS | 1 + lib/Kconfig | 15 + lib/tests/Makefile | 1 + lib/tests/percpu_counter_tree_bench.c | 604 ++++++++++++++++++++++++++ 4 files changed, 621 insertions(+) create mode 100644 lib/tests/percpu_counter_tree_bench.c
MAINTAINERS +1 -0
@@ -11955,6 +11955,7 @@ S: Supported F: Documentation/core-api/percpu-counter-tree.rst F: include/linux/percpu_counter_tree.h F: lib/percpu_counter_tree.c +F: lib/tests/percpu_counter_tree_bench.c F: lib/tests/percpu_counter_tree_kunit.c F: tools/testing/hpcc/
lib/Kconfig +15 -0
@@ -53,6 +53,21 @@ config PERCPU_COUNTER_TREE_KUNIT_TEST When in doubt, say N. +config PERCPU_COUNTER_TREE_BENCH + tristate "Benchmark of hierarchical per-cpu counters" + depends on KUNIT && SMP + help + This builds a KUnit benchmark comparing the hierarchical per-cpu + counters with struct percpu_counter: update cost with all online + CPUs updating shared and private counters, and init/destroy cost. + It runs for a few minutes on large machines. + + The batch size and the tree shape can be changed with the + percpu_counter_tree_bench.batch= and percpu_counter_tree.arity= + parameters, see lib/tests/percpu_counter_tree_bench.c. + + When in doubt, say N. + config BITREVERSE tristate select GENERIC_BITREVERSE if !HAVE_ARCH_BITREVERSE
lib/tests/Makefile +1 -0
@@ -60,5 +60,6 @@ obj-$(CONFIG_RATELIMIT_KUNIT_TEST) += test_ratelimit.o obj-$(CONFIG_UUID_KUNIT_TEST) += uuid_kunit.o obj-$(CONFIG_PERCPU_COUNTER_TREE_KUNIT_TEST) += percpu_counter_tree_kunit.o +obj-$(CONFIG_PERCPU_COUNTER_TREE_BENCH) += percpu_counter_tree_bench.o obj-$(CONFIG_TEST_RUNTIME_MODULE) += module/
lib/tests/percpu_counter_tree_bench.c new file +604 -0
604 lines not on this page · raw message
Signature · 1 line
2.47.3
[email protected]
AI reviewRe: [RFC PATCH v23 10/10] lib: add a benchmark of hierarchical per-cpu counters
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] lib: stack-allocated percpu_counter in bench triggers debugobjects WARN_ON - [Low] lib: missing…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] lib: stack-allocated percpu_counter in bench triggers debugobjects WARN_ON
- [Low] lib: missing kernel-parameters.txt entry for percpu_counter_tree_bench.batch --
commit a1a6611650e4e4f4d5279cc3d46fb8ab12ea614c
Author: Mathieu Desnoyers <[email protected]>
lib: add a benchmark of hierarchical per-cpu counters
This commit adds a KUnit benchmark comparing hierarchical per-CPU counters with struct percpu_counter across update, lifecycle, and churn workloads. It introduces CONFIG_PERCPU_COUNTER_TREE_BENCH and the batch module parameter to configure the hierarchical counter batch size.
Quoted 5 lines
lib/tests/percpu_counter_tree_bench.c new file +0
-0
[ ... ]
lib/tests/percpu_counter_tree_bench.c line 55
Quoted 4 lines
lib/tests/percpu_counter_tree_bench.c new file +4
-0+/* Like the RSS counters by default. */ +static unsigned int batch =32 ; +module_param (batch, uint,0444 ); +MODULE_PARM_DESC (batch,"Batch size of the hierarchical counters (power of 2, 2 to 128)" );
[Severity: Low]
Should percpu_counter_tree_bench.batch= also be documented in Documentation/admin-guide/kernel-parameters.txt?
Both the commit message and the CONFIG_PERCPU_COUNTER_TREE_BENCH help text in lib/Kconfig advertise passing percpu_counter_tree_bench.batch= on the kernel command line alongside percpu_counter_tree.arity=, which is documented there.
[ ... ]
Quoted 19 lines
lib/tests/percpu_counter_tree_bench.c new file +19
-0+static int bench_lifecycle_fn (void *data) +{ +struct bench_thread *t = data; +struct percpu_counter_tree c[LIFECYCLE_NR ]; +struct percpu_counter pc[LIFECYCLE_NR ]; +u64 t0; +int i; + +if (bench_wait_go (t->sh)) +return 0 ; + t0 =ktime_get_ns (); +for (i =0 ; t->sh->impl ==IMPL_PERCPU_COUNTER && i <LIFECYCLE_ITER ; i++) { +if (percpu_counter_init_many (pc,0 ,GFP_KERNEL ,LIFECYCLE_NR )) +break ; +percpu_counter_add (&pc[0 ],1 ); +percpu_counter_add (&pc[0 ],-1 ); +percpu_counter_destroy_many (pc,LIFECYCLE_NR ); +cond_resched (); + }
[ ... ]
Quoted 8 lines
lib/tests/percpu_counter_tree_bench.c new file +8
-0+static int bench_churn_fn (void *data) +{ +struct bench_thread *t = data; +unsigned int pin_period = t->sh->pin_period; +struct percpu_counter_tree tmp[LIFECYCLE_NR ], *c; +struct percpu_counter tmp_pc[LIFECYCLE_NR ], *pc; +u64 t0; +int i;
lib/tests/percpu_counter_tree_bench.c line 226
Quoted 10 lines
lib/tests/percpu_counter_tree_bench.c new file +10
-0+ +if (bench_wait_go (t->sh)) +return 0 ; + t0 =ktime_get_ns (); +for (i =0 ; t->sh->impl ==IMPL_PERCPU_COUNTER && i <CHURN_ITER ; i++) { +bool pin = pin_period && !(i % pin_period); + + pc = pin ? &t->pinned_pc[t->nr_pinned *LIFECYCLE_NR ] : tmp_pc; +if (percpu_counter_init_many (pc,0 ,GFP_KERNEL ,LIFECYCLE_NR )) +break ;
[Severity: Medium]
Will passing stack-allocated struct percpu_counter arrays (pc in bench_lifecycle_fn() and tmp_pc in bench_churn_fn()) to percpu_counter_init_many() trigger a debugobjects WARN_ON(1) when CONFIG_DEBUG_OBJECTS_PERCPU_COUNTER is enabled?
The call chain is:
percpu_counter_init_many()
__percpu_counter_init_many()
debug_percpu_counter_activate()
debug_object_init()
__debug_object_init(..., onstack = 0)
lookup_object_or_alloc()
debug_object_is_on_stack()Because debug_object_init() is called with onstack = 0 while pc and tmp_pc reside on the kthread stack, debug_object_is_on_stack() will warn that a non-onstack object is on the stack.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10